Demystifying MITRE’s ATT&CK Evaluation
Imagine if each alert you received was actually usable and could tell a complete, linked, relevant story – that’s exactly what SentinelOne did in the MITRE ATT&CK simulation. We built our product to be bold and different. In MITRE terms, each SentinelOne alert curated 37 commands. Alert fatigue ends here.
Don’t be fooled: Missed real-time alerts yield attacker dwell time which increases risk and damage. Minimize real-time misses by mastering detection with a performant product versus a delayed service.
Missed real-time alerts yield attacker dwell time which increases risk and damage
It was 1996; Gary Kasparov, the world’s greatest chess grandmaster lost to IBM’s Deep Blue computer. In the EDR space, a similar watershed moment happened a few weeks ago: for the first time, in a third party test, SentinelOne proved that autonomous technology is independently and efficiently capable of the most nuanced EDR in real-time.
In MITRE’s recent ATT&CK Evaluation, the SentinelOne agent detected, tracked, and linked the context of all APT attacks with ONE autonomous agent in real-time without delayed cloud or human powered resources. Other EDR vendors have “store and analyze” dependencies, making them reliant on skilled people, cloud services, and required service modules. SentinelOne’s patented prevention capabilities stop attacks before and during execution at the source; in addition, all analysis is done on-agent, by the agent, in real-time, providing maximum EDR detection, visibility, and control.
- Technology that does more inherently helps people do more. The SentinelOne autonomous agent was able to do in real-time what required competitors’ MDR teams hours and days to do (their products alone cannot perform). It’s the first time this claim has been proven in a 3rd party EDR test.
- Alert fatigue is real: “crying wolf” alerts don’t solve problems. The MITRE ATT&CK evaluation simulated an end-to-end nation state campaign style attack. It wasn’t a standalone alert test of 78 different MITRE techniques or a standalone alert test of the 190 actions taken to execute those techniques. See which products represent the simulation as curated attacks, not a barrage of isolated events/alerts: the product with the most alerts doesn’t win in the real world – operational alerts and in-context telemetry enables your SOC to be effective, efficient, and win.
- Dependency on connectivity is a very dangerous assumption. Some of the tested products require endpoints to be online and responsive in order for a SOC analyst to query for results. SentinelOne’s autonomous agent detected, mapped, and streamed the indexed attack data in real-time to the console for presentation. Expecting an endpoint to be online when you need to gather data about an APT-level attack isn’t something you can rely on in the real world.
- The ATT&CK simulation was solely about detection, not about response and protection capabilities. The MITRE evaluation was performed in what we call a Legacy EDR or “Detect-Only” policy: SentinelOne’s default policy is “Protect.” In this policy, the autonomous agent contains, stops, and immunizes the enterprise environment from the MITRE attacks in real-time without human interaction or cloud reliance. Online and offline, SentinelOne’s performance is equivalent: no reliance on prior knowledge, “threat intel,” or non-product sources. SentinelOne believes this to be the most compelling finding in the attack test.
The MITRE Simulation: Two Attacks. Three Devices. Four SentinelOne Behavioral Alerts.
The MITRE test consisted of two multi-staged attacks using the Cobalt Strike and PowerShell Empire frameworks. The attacks used a total of three devices. SentinelOne showed four behavioral alerts in the console: this is by design. Each alert represents the complete story of an attack in real-time on the device it was detected on, with all of the information mapped in the alert details. Each alert was generated during the very first phase of each attack. Below is the SentinelOne console used for the test:
Under the alerts, the autonomous agent automatically captured and correctly mapped over 205 observed actions in real-time. If performed as standalone actions, SentinelOne alerts on each action – the brilliance is how SentinelOne recognized and autonomously mapped the orchestrated attack.
In the case of the test, SentinelOne doesn’t spawn multiple (unnecessary) alerts for the same attack even if it observes other alertable or enriching behavior in the attack as it progresses. Instead, SentinelOne maps the additional observed actions into the attack story, making a complete story which is much easier for SOC analysts to understand. In total, the autonomous agent mapped over 34,000 observed actions including C2 to their correct attack story lines.
Check out the real screenshots showing the attack story lines from the attack commands:
Understanding the Tests’ Implications
According to MITRE, this is a negative alert modifier for detections.
Delayed detections are detections that happened after commands had completed and required an MDR service to find them – after the fact. The time between when the command was completed and the alert showed in a console is added (and unnecessary) dwell time for the attacker to exploit an environment. According to MITRE testing guidelines, delayed detections were permitted to appear up to 24 hours following the completion of a command. In the real world, delayed detections can appear in hours, days, or sometimes even longer.
A real-life example: A person breaks into your house and steals a bunch of stuff. This is recorded on a video surveillance system which is reviewed hours later when the break in is discovered. The delayed detection of the break in lets you know what happened on camera, hours later, but not what happened off camera. Furthermore, the damage is done before it was even detected. And you must ask yourself: is the criminal still running around your house?
A valid way of looking at MITRE ATT&CK evaluation results is to focus on real-time data. Below is an example of the data when viewed from combining delayed and missed alerts and telemetry as reported by MITRE.
A performant EDR solution is one that doesn’t increase attacker dwell time with missed or delayed alerts.
According to MITRE, “tainted” is a positive modifier for telemetry:
“Tainted” detections links other commands or actions to a previous alert. This linking is something that the SOC analyst doesn’t have to spend time performing. Some vendors had to manually perform queries after the fact to the infected endpoints to obtain this data. In the case of SentinelOne, the “tainted” telemetry was mapped in real-time to the correct context of each attack.
A real-life example: A person is trying to enter your house. If you detected the person picking your door lock, you would know that they are not on your property with permission and they are not yet in your house. You also know that anything they do after picking the lock is certainly unwanted and/or criminal.
Alert Fatigue is a Real Problem: Data for the Sake of Data Doesn’t Solve Problems
A common set of problems plaguing CISOs’ experiences with EDR solutions is alert fatigue, a growing skills gap, and the expense of dealing with ever growing amounts of data. These observations are noted as key findings in Gartner’s “Market Guide for Endpoint Detection and Response Solutions.” CISOs tell us they don’t want more data to bog down their SOC unless it is in-context, actionable, and not a duplicate of something else.
The Target breach is a historical example of this. The breach was detected by the security tools in the environment, however, the alert was just one of thousands being handled by the SOC. Simply put, it was lost in the noise.
In the MITRE simulation, we saw two attacks on three machines at twenty stages with 190 total actions. One product in the evaluation didn’t raise any alerts – that’s pretty bad. On the other end of the spectrum, another product raised over 200 alerts. Assuming it takes a SOC analyst several minutes to triage an alert, how long will it take to string the standalone alerts into an attack story?
What’s Preferred for Your SOC?
- Standalone: A single alert per command x 100+ commands each with little to no context or relationship to the overall attack?
- Linked: A single alert with the telemetry and details of the next 100+ commands automatically mapped in the context of the first alert and attack story.
SentinelOne has chosen the later and designed our product for this purpose. It is up to CISOs to determine which is best suited for their security programs.
Conclusion…You’ve Mastered MITRE!
As prospects and customers explore the MITRE ATT&CK Evaluation and subsequent vendor discourse, our guidance is to test products to find the optimal solution for your environment with real-world scenarios. Just like the IBM Deep Blue computer which beat the first human at chess, our technology offers a revolutionary and differentiated EDR experience: instead of conscripting SOC analysts or a required MDR service to work for their EDR product, SentinelOne’s autonomous agent delivers automated and nuanced EDR for you.
We eagerly look forward to continuing our participation in MITRE’s APT testing as well as helping our customers leverage the ATT&CK framework through our in-product MITRE integration.