Is Zero Day Java Exploit Detection Possible?

Java is an expansive, ubiquitous, platform agnostic infrastructure for delivering remote code to endpoint computers. That makes it a perfect malware distribution pipeline. It enables malware authors to take advantage of Java exploits to infect computers on a global scale. To make matters worse, security updates are not installed automatically when the Java updater launches, so the natural response for most users is to ignore it.

How common are Java exploits, really? Well, according to Kaspersky’s 2013 security bulletin, Java vulnerabilities accounted for 90.52% of all attacks.

Since our product uses predictive and dynamic detection methods, we wanted to test the effectiveness of an anti-exploitation component for Java in the SentinelOne agent to determine whether they could detect and prevent zero-day exploits in real time.

I also researched a few exploits from the Metasploit framework, and identified several behavioral chokepoints common to these exploits as well. The sum total of these chokepoints include what I believe to be the fundamental methods that can be used to break or circumvent the secure Java sandbox. Barring horrible memory corruption bugs stemming from poor implementation, or game breaking logical bugs, adding these chokepoints to the SentinelOne agent should allow us to automatically detect any Java zero-day.

At this point, most security-conscious readers are probably thinking that my goal — (near) perfect zero-day coverage — is ludicrous. So let me clarify. This approach is not “absolutely” perfect, but it is extremely effective, well beyond what is capable with traditional detection methods.  Here’s why.

For every JVM instance, we perform deep instrumentation that establishes safeguards to detect common sandbox escape methods; such as setting the security manager to null, a very common exploit behavior. When one of these safeguards is triggered, like when an exploit attempts to execute a privileged “System.setSecurityManager(null)” statement, we recognize this as malicious and stop it before it can perform any privileged operation.

We then pitted this anti-exploitation component against a set of previously unseen Java exploits. It successfully detected a whopping 91% of them, with 0% false positives (for varying Java applets found on the internet – from simple calculators, through applets running on Oracle’s own site and ending with Java-based chatrooms). In addition, most of the exploits that weren’t detected were based on Java type-confusions to access a Classloader, which can load a specified class with whatever permissions and source code designated.

For example, the main exploit code for CVE-2012-0507 is a vulnerability based on a type-confusion in AtomicReferenceArray. It exhibits a familiar chokepoint when it inherits and illegally accesses the protected methodsof an existing Classloader object using a Helper class to point to it.  A simple tweak to the anti-exploitation component allowed us to detect and block this type of unprivileged Classloader access and increase our success rate to nearly 100%.

In a previous blog post we explained why anti-exploitation is only part of the solution. That’s why we designed the SentinelOne agent to go beyond simple anti-exploitation and provide predictive executive analysis that can monitor, detect and block all application processes that are malicious.