Stub mechanics and the userland exec

First, it is necessary to understand that a software protector is actually made up of two programs:

• Protection phase code: The program that applies the protection to the target binary

• Runtime engine or stub: The program that is merged with the target binary that is responsible for deobfuscation and anti-debugging at runtime

The protector program can vary greatly depending on the types of protection that are being applied to the target binary. Whatever type of protection is being applied to the target binary must be understood by the runtime code. The runtime code (or stub) must know how to decrypt or deobfuscate the binary that it is merged with. In most cases of software protection, there is a relatively simple runtime engine merged with the protected binary; its sole purpose is to decrypt the binary and pass control to the decrypted binary in memory.

This type of runtime engine is not so much an engine—really—and we call it a stub. The stub is generally...