Breaking EvilQuest | Reversing A Custom macOS Ransomware File Encryption Routine

Executive Summary

  • A new macOS ransomware threat uses a custom file encryption routine
  • The routine appears to be partly based on RC2 rather than public key encryption
  • SentinelLabs has released a public decryptor for use with “EvilQuest” encrypted files


Researchers recently uncovered a new macOS malware threat[1], initially dubbed ‘EvilQuest’ and later ‘ThiefQuest'[2]. The malware exhibits multiple behaviors, including file encryption, data exfiltration and keylogging[3].

Of particular interest from a research perspective is the custom encryption routine. A cursory inspection of the malware code suggests that it is not related to public key encryption. At least part of it uses a table normally associated with RC2. The possible usage of RC2 and time-based seeds for file encryption led me to look deeper at the code, which allowed me to understand how to break the malware’s encryption routine. As a result, our team created a decryptor for public use.

Uncarving the Encryption Routine

As mentioned in other reports[4], the function responsible for file encryption is labelled internally as carve_target.

Before encrypting the file, the function checks whether the file is already encrypted by comparing the last 4 bytes of the file to a hardcoded DWORD value.

If the test fails, then file encryption begins by generating a 128 byte key and calling the tpcrypt function, which basically ends up calling generate_xkey. This function is the key expansion portion followed by tp_encrypt, which takes the expanded key and uses it to encrypt the data.

Following this, the key will then be encoded, using time as a seed. A DWORD value will be generated and utilized.

The encoding routine is simply a ROL-based XOR loop:

At this point, we can see that something interesting happens, and I am unsure if it is intentional by the developer or not. The key generated is 128 bytes, as we previously mentioned.

The calculations then used for encoding the key end up performing the loop 4 extra times, producing 132 bytes.

This means that the clear text key used for encoding the file encryption key ends up being appended to the encoded file encryption key. Taking a look at a completely encrypted file shows that a block of data has been appended to it.

Reversing the File Encryption

Fortunately, we don’t have to reverse that much as the actor has left the decryption function, uncarve_target, in the code. This function takes two parameters: a file location and a seed value that will be used to decode the onboard file key.

After checking if the file is an encrypted file by examining the last 4 bytes, the function begins reading a structure of data from the end of the file.

Following the code execution, we can statically rebuild a version of what this structure might look like:

struct data
enc blob[size+12]
long long size
int marker

struct enc
long long val
int val2                                      // 3rd param to eip_key
long long val3                                // 1st param to eip_key
char encoded_blob[4 - val % 4 + val]          // for 0x80 this is 132

The encoded file key will then be decrypted and checked using the two values from the structure and the other seed value passed to uncarve_target. The file key will be decrypted by eip_decrypt, which is the encrypt-in-place decrypt routine.

The function eip_key will take the two DWORD values and the seed argument to generate the XOR key to decode the filekey.

Next, the file is set to the beginning and then a temporary file is opened for writing.

The file is then read into an allocated buffer and the key and encoded file data are passed to tpdcrypt.

As before, we have a key expansion followed this time by a call to tp_decrypt.

A glance inside the key expansion function shows a reference to a hardcoded table which matches RC2 code that can be found online.

So now we have enough information to recover the file key:

import struct
import sys

rol = lambda val, r_bits, max_bits=32: 
(val << r_bits%max_bits) & (2**max_bits-1) |  ((val & (2**max_bits-1)) >> (max_bits-(r_bits%max_bits)))

data = open(sys.argv[1], 'rb').read()

test = data[-4:]
if test != 'xbexbaxbexdd':
  print("Unknown version")

append_length = struct.unpack_from('<I', data[-12:])[0]

append_struct = data[-(append_length+12):]

keySize = struct.unpack_from('<I', append_struct)[0]

if keySize != 0x80:
  print("Weird key?")

encoded_data = append_struct[20:20+132]

xorkey = struct.unpack_from('<I', encoded_data[-4:])[0]

def decode(blob, key):
  out = ""
  for i in range(len(blob)/4):
	temp = struct.unpack_from('<I', blob[i*4:])[0]
	temp ^= key
	key = rol(key, 1)
	out += struct.pack('<I', temp)
  return out[:0x80]

temp = decode(encoded_data, xorkey)

Attempting to RC2 decrypt the data, however, only seems to work partially at this time using RC2 routines in both Python and Golang libraries. Further analysis will be needed to verify what is different.

However, for the purpose of decrypting victim files, we need only take the file key and call the tp_decrypt function that is located inside the malware itself instead. Dumping the assembly for this function and building it into a shared object to be executed using the recovered file key appears to work correctly.

Using this method, SentinelLabs created a public decryptor which is available here (this tool is released under the MIT software license).


SHA-1: 178b29ba691eea7f366a40771635dd57d8e8f7e8
SHA-256: f409b059205d9a7700d45022dad179f889f18c58c7a284673975271f6af41794