Antivirus – the basics

Antivirus software is intended to detect and prevent the spread of malicious files and processes within the operating system, thus protecting the endpoint from running them.

Over time, antivirus engines have improved and become smarter and more sophisticated; however, the foundation is identical in most products.

The majority of antivirus products today are based on just a few engines, with each engine having a different goal, as follows:

• Static engine
• Dynamic engine (includes the sandbox engine)
• Heuristic engine
• Unpacking engine

Of course, most of these engines have their own drawbacks. For example, the drawback of a static engine is that it is extremely basic, as its name implies. Its goal is to identify threats using static signatures, for instance, the YARA signature (YARA, Welcome to YARA's documentation, https://yara.readthedocs.io/en/stable/). These signatures are written from time to time and updated by antivirus security analysts on an almost daily basis.

During a scan, the static engine of the antivirus software conducts comparisons of existing files within the operating system to a database of signatures, and in this way can identify malware. However, in practice, it is impossible to identify all malware that exists using static signatures because any change to a particular malware file may bypass a particular static signature, and perhaps even completely bypass the static engine.

The following diagram demonstrates the static engine scanning flow:

Figure 1.1 – Antivirus static engine illustration

Using a dynamic engine, antivirus software becomes a little more advanced. This type of engine can detect malware dynamically (when the malware is executed in the system).

The dynamic engine is a little more advanced than the static engine, and its role is to check the file at runtime, through several methods.

The first method is API monitoring – the goal of API monitoring is to intercept API calls in the operating system and to detect the malicious ones. The API monitoring is done by system hooks.

The second method is sandboxing. A sandbox is a virtual environment that is separated from the memory of the physical host computer. This allows the detection and analysis of malicious software by executing it within a virtual environment, and not directly on the memory of the physical computer itself.

Running malware inside a sandboxed environment will be effective against it especially when not signed and detected by the static engine of the antivirus software.

One of the big drawbacks of such a sandbox engine is that malware is executed only for a limited time. Security researchers and threat actors can learn what period of time the malware is executing in a sandbox for, suspend the malicious activity for this limited period of time, and only then run its designated malicious functionality.

The following diagram demonstrates the dynamic engine scanning flow:

Figure 1.2 – Antivirus dynamic engine illustration

Using a heuristic engine, antivirus software becomes even more advanced. This type of engine determines a score for each file by conducting a statistical analysis that combines the static and dynamic engine methodologies.

Heuristic-based detection is a method, that based on pre-defined behavioral rules, can detect potentially malicious behavior of running processes. Examples of such rules can be the following:

• If a process tries to interact with the LSASS.exe process that contains users' NTLM hashes, Kerberos tickets, and more
• If a process that is not signed by a reputable vendor tries to write itself into a persistent location
• If a process opens a listening port and waits to receive commands from a Command and Control (C2) server

The main drawback of the heuristic engine is that it can lead to a large number of false positive detections, and through several simple tests using trial and error, it is also possible to learn how the engine works and bypass it.

The following diagram demonstrates the heuristic engine scanning flow:

Figure 1.3 – Antivirus heuristic engine illustration

Another type of engine that is widely used by antivirus software is called the unpacker engine. In Chapter 5, Bypassing the Static Engine, we will discuss what a packer is, how the unpacking process works, and how to bypass antivirus software using packing.

One of the major drawbacks of today's advanced antivirus software centers on their use of unpackers, tools used by antivirus engines to reveal malicious software payloads that have undergone "packing," or compression, to hide a malicious pattern and thus thwart signature-based detection.

The problem is that there are lots of packers today that antivirus software does not have unpackers for. In order to create automated unpacker software, security researchers from the antivirus software vendor must first perform manual unpacking – and only then can they create an automated process to unpack it and add it to one of their antivirus engines.

Now that we understand the basic engines that exist in almost every antivirus software, we can move on to recognize practical ways to bypass them to ultimately reach the point where we are running malware that lets us remotely control the endpoint even while the antivirus software is up and running.