Protecting control flow integrity

A protected binary should aim to protect the program during runtime (the process itself) just as much as—if not more than—the binary at rest on the disk. Runtime attacks can generally be classified into two types:

• Attacks based on ptrace

• Vulnerability-based attacks

Attacks based on ptrace

The first variety, ptrace based attacks, also falls under the category of debugging a process. As already discussed, a binary protector wants to make ptrace based debugging very difficult for a reverse engineer. Aside from debugging, however, there are many other attacks that could potentially help break a protected binary, and it is important to know and understand what some of these are in order to give further clarification as to why a binary protector wants to protect a running process from ptrace.

If a protector has gone so far that it is able to detect breakpoint instructions (and therefore make debugging more difficult) but is not able to protect itself from being traced...