Obfuscation methods
A binary can be obfuscated or encrypted in many creative ways. Most binary protectors simply protect the entire binary with one or more layers of protection. At runtime, the binary is decrypted and can be dumped from the memory to acquire a copy of the unpacked binary. In more advanced protectors, such as Maya, every single function is encrypted individually, and allows only a single function to be decrypted at any given time.
Once a binary is encrypted, it must, of course, store the encryption keys somewhere. In the case of Maya (discussed earlier), a custom heap implementation that itself uses encryption to store encryption keys was designed. At some point, it would seem that a key has to be exposed (such as the key used to decrypt another key), but special techniques such as white-box cryptography can be used to make these final keys extremely obfuscated. If assistance from the kernel is used in a protector, then it is possible to store the key outside of the binary...
