Detecting software breakpoints (INT3)

This type of breakpoint is the easiest to use, as well the easiest to detect. As we stated in Chapter 1, A Crash Course in CISC/RISC and Programming Basics, this breakpoint modifies the instruction bytes by replacing the first byte with 0xCC (the INT3 instruction), which creates an exception (an error) that gets delivered to the debugger to handle.

Since it modifies the code in memory, it's easy to scan the code section in memory for the INT3 byte. A simple scan will look like this:

Figure 2: Simple INT3 scan

The only drawback of this approach is that some C++ compilers write INT3 instructions after the end of each function as filler bytes. An INT3 byte (0xCC) can also be found inside some instructions as part of an address or a value, so searching for this byte through the code may not be an effective solution, and could return lots of false positives.

There are two other techniques that are commonly used by malware to scan for an INT3 breakpoint...