Software and IT are everywhere, and their adoption is increasing at an ever-increasing speed. Software programming is prevalent in the fields of entertainment, health, education, food, travel, auto, communication, media, and every other field we can think of. As the number of software programs and their features increase, so does the number of software bugs and flaws. A security flaw, glitch, or weakness found in software code that could be exploited by an attacker (threat source) is called a software vulnerability. The number of such vulnerabilities has been increasing drastically year by year, as seen in the following figure.

Figure 1.1 – Vulnerabilities trend over the past decades

Threat actors take advantage of such vulnerabilities and cause disruption to the confidentiality, integrity, or availability of the protected system. In certain vulnerabilities, the threat actor makes use of various exploits to deliver, install, and/or execute a malicious program on the system. Such malicious code is known as malware.

Malware comes in a variety of forms – viruses, worms, backdoors, trojans, adware, spyware, ransomware, and so on – each with its own characteristics. This malware aims to steal, damage, and/or destroy vulnerable systems – exfiltrating sensitive data or encrypting files and/or disks to make them unusable.

The damage caused by ransomware alone is shown in the following chart:

Figure 1.2 – Increasing cost of ransomware-related damage

Typical cyberattacks consist of a set of common phases or stages. Lockheed Martin has created a model called the Cyber Kill Chain to encapsulate these stages (https://www.lockheedmartin.com/en-us/capabilities/cyber/cyber-kill-chain.html). The stages are as follows:

1. Reconnaissance: This is the phase in which the adversary identifies the target’s possible vulnerabilities and weak points. This may involve active scanning of the target network, passive information gathering, social engineering, gathering information from the internet and/or social networks, and so on. This step provides the adversary with sufficient information to proceed with the attack – such as which IP addresses are accessible, what ports are open, what applications are running, and details of the vulnerabilities on each.
2. Weaponization: In this stage, the attacker creates a payload (weapon) that exploits the discovered vulnerability and plants malware on the victim’s machine.
3. Delivery: This is the stage when the attacker delivers the prepared payload, for example, an infected document to the target. A typical delivery mechanism is a phishing email containing a malicious link or an infected PDF document.
4. Exploitation: In this stage, the target machine is compromised by the exploit delivered in the previous stage. When the exploit code is executed, the attacker accomplishes their objective, such as remote control of the target machine. Subsequently, having gained a foothold on the victim’s machine, the attacker proceeds to the next phases, such as maintaining persistence and exfiltrating data.
5. Installation: In the installation phase, various types of malwares are installed on the target machine – ransomware, backdoors, or trojans – based on the plan of the attacker for their purposes.
6. Command and control: Once the malware is installed on the target machine, it typically contacts a command and control server. This may be to get additional instructions or commands to be executed on the target machine.
7. Action: In this stage, the malware acts on the target as per the commands or instructions from the attacker. This may involve installing additional malware, exfiltrating sensitive data and system information back to the attacker-controlled server, or even performing denial-of-service attacks on any specified targets.

These are the typical stages of a cyberattack. From a security point of view, the earlier the attack is detected, the better. If the defense mechanisms in place can detect and stop an attack at the delivery stage, any compromise can be prevented.

In the next section, let us look at a strategy that aims to ensure the highest chance of a successful defense against attack attempts.

Defense in depth is a strategy for protecting a system against any attack using several independent defense methods. This approach was originally conceived by the National Security Agency. The system that needs to be protected consists of a set of resources and assets, including the network itself. A typical scenario would include web servers, mail servers, DNS infrastructure, WAN and LAN routers, authentication servers, database servers, laptops, and desktops.

As mentioned earlier, a defense-in-depth strategy uses independent and mutually exclusive mechanisms to protect and defend the assets; thus, the chances of detecting an attack are higher than using a single mechanism. It is sufficient for any one of the layers to detect the attack, in order to prevent and thwart it. The several layers of the defense-in-depth strategy are depicted in Figure 1.3.

Figure 1.3 – Defense in depth

The defense-in-depth strategy would include security technology, processes, and/or policies at several layers, including network, perimeter, endpoint, application, and data security.

Some of the various layers of the defense-in-depth approach in a typical scenario are discussed in the following subsections.

Firewalls (network and host layers)

Network firewalls filter the network by inspecting traffic that enters or leaves through network boundaries/zones. They enforce user-defined security policies across single or multiple network segments, comparing policies, adding threat modules, and assessing the data packets to prevent unauthorized access. Firewall deployments are precisely placed within the network to inspect and manage traffic flow.

Network firewalls are analogous to doorkeepers. When deployed in the network perimeter, they are typically the outermost layer in the defense-in-depth strategy. However, network firewalls are also deployed within a segregated network to separate various sections and/or departments. Network firewalls perform basic protocol decoding and analysis in order to be able to allow or deny packets and/or connections in or out of the network.

Host-based firewalls are like network firewalls except that they are concerned only with a single host as opposed to a set of hosts in a network.

Network- and host-based firewalls can create logs for every inbound and outbound connection that traverses through them. This can be immensely valuable from a detection point of view.

Intrusion detection and prevention systems (network and host layers)

IDS are analogous to security cameras. They are devices or programs that detect malicious activity against the concerned network or host (network-based or host-based IDS).

For a network-based IDS, the system inspects and analyzes the network traffic and tries to detect malicious activity based on signatures (for known attacks) or anomalous behavior or deviation from standard. The deviation from the standard can either be a statistical deviation (statistical anomaly-based IDS) or a deviation from protocol specifications (protocol anomaly-based IDS).

A host-based IDS will monitor all host artifacts in order to detect malicious activity, including network traffic to or from that host, process details, host-based logs, and files on the host.

IPS are IDS with the additional capability to enforce actions that prevent an attack. For example, upon detection of an attack, the IPS may drop the concerned packet or block the entire connection.

Endpoint detection and response (host layer)

Endpoint detection and response (EDR) comprises tools and technology that monitor activity on endpoint hosts and servers in order to detect malicious activity. The activity that is monitored by EDR includes processes, connections (to and from) the host, files created/modified, and registry changes.

Web application firewalls (network and host layers)

Web application firewalls (WAF) are firewalls specifically for web traffic. WAF inspect and analyze web traffic comprehensively. They can analyze both HTTP and HTTPS protocols. In the case of HTTPS, WAF often terminate the SSL sessions to decrypt the traffic, which often involves playing a man-in-the-middle role between the web client and the web server.

Traditional firewalls allow or deny traffic based on OSI layer 3 and 4 headers. Network-based IPS can perform limited application-level analysis. Compared to these, WAF are capable of comprehensive web (HTTP/HTTPS) traffic analysis in order to make the allow versus deny decision.

Some of the commercial companies that offer WAF are Fortinet, Barracuda, and Imperva. ModSecurity is also a widely available option for an open source WAF.

Mail security gateway (network)

A mail security gateway or firewall is another application-level firewall but for email-related protocols. A significant percentage of threats involve emails. In the first half of 2021, 75% of threats were delivered using email. Emails are often used as bait to trap unsuspecting users – by prompting them to open a malicious attachment, or by tempting them to click a malicious link.

Mail security gateways protect users from threats related to email by analyzing and filtering the malicious artifacts from an email. Mail firewalls perform deep inspection of the protocols related to mail, namely SMTP, POP, IMAP, and their encrypted counterparts.

Log management and monitoring (network and host)

Log management and monitoring solutions collect, inspect, and archive log messages and files from a variety of devices in the network. They also enable capabilities such as indexing and searching across the collected logs.

In the next section, let us specifically look at network IDS and IPS and the role that they play in the defense-in-depth strategy.

Network-based IDS and IPS play a significant role in the defense-in-depth strategy for information security. This role is unique when compared with other pieces of the defense-in-depth approach. As the name suggests, the primary role of IDS is detection, whereas IPS adds the extra capability of blocking the attack that it has detected.

The network IDS processes network traffic – analyzes the various protocols that are involved – with the goal of detecting malicious activity in a real-time fashion. The network IDS typically also has the capability to analyze packet captures offline; however, the most common case is to perform the analysis live so as to detect the attack in real time.

In general, the network IDS functionality would include the following:

• Configuration management: IDS configuration essentially determines what exact functionality is performed by the IDS, how much memory needs to be allotted, the various parameters for learning for anomaly-based IDS, and the signatures to be analyzed.
• Packet acquisition module: This module is responsible for getting the network traffic data (packet data) from the source to the IDS. IDS often use packet capture libraries such as libpcap in order to attain this functionality.
• Decoder module: Irrespective of the type of IDS (signature-based or anomaly-based), there needs to be a module that can decode the various network protocols, maintain some state, and make the data available for the rest of the IDS to perform its detection operation.
• Detection module: This is the module that performs the detection functionality – whether it is signature matching or detecting an anomaly.
• Alert and log module: This module performs the task of generating an alert in the event of attack detection, as well as logging critical log messages regarding the IDS operation.

In the event of detecting an attack, the IDS/IPS generates an alert; these alerts are brought to the attention of a security operator for further action or sent to a central system such as Security Incident and Event Management (SIEM) for collection, correlation, and analysis. Figure 1.4 shows a typical IDS and IPS deployment scenario. It can be noted that the IPS is deployed in an inline fashion, whereas the IDS is deployed in an offline manner.

Figure 1.4 – Typical IDS and IPS deployment diagram

Due to the difference in their objectives, the IDS is typically deployed in a passive manner, often analyzing a copy of the network traffic (collected via a SPAN port on a router or firewall). IPS devices, on the other hand, operate in an inline mode – very similar to a firewall – so that they can block the offending packet or connection.

This difference – passive/offline versus inline – in the deployment leads to a key distinction. When the traffic rate increases to a level that the IDS cannot keep up with, it leads to packet drops; it does not affect the operation as it is a copy of the packet that was dropped. However, in the case of an inline operation, when the IPS cannot keep up with the rate of traffic leading to packet drops, it affects the network throughput and becomes a performance bottleneck. Therefore, there is increased demand on the IPS to have faster packet processing than for an IDS.

There is yet another key difference between the IDS and IPS, namely the consequence of a false positive. A false positive is when the IDS or IPS detects a benign packet or connection as malicious. For an IDS, this will result in a false positive alert being generated. This will result in an unnecessary alert and analysis. However, for an IPS that blocks packets and connections when an alert is generated, this will result in the interruption of a normal or benign connection, resulting in user dissatisfaction.

Due to these key differences, IDS and IPS devices are often configured very differently – one giving priority to detection (IDS) and the other giving priority to performance as well as detection (IPS).

In the next section, we will discuss how the IDS and IPS are categorized based on how the detection is done.

Intrusion detection approaches are classified into the following based on how malicious activity is detected. The most common approaches are signature-based, anomaly-based, and hybrid. Let us discuss each of these approaches.

Signature-based intrusion detection

The signature-based approach uses predefined signatures in order to detect known threats. When an attack is initiated that matches one of these signatures, a predefined action (for example, generate an alert) is taken.

This is the most common approach for intrusion detection, especially in commercial solutions. Open source IDS/IPS – such as Snort and Suricata – are essentially signature-based. Signature-based systems are very good and proven to detect known attacks with very good accuracy and efficiency. As opposed to anomaly detection techniques, the signature-based IDS does not require any training or learning phase. The most important disadvantage of this approach is the inability to detect unknown attacks. Due to this reason, this approach requires constant (almost daily) updates to the signature set so that it can detect new threats that appear daily.

A simplified block diagram of a signature-based IDS is shown in Figure 1.5.

Figure 1.5 – Block diagram of a typical signature-based IDS

The input from the monitored environment (for example, packets from a monitored network) is processed and matched against a set of signatures; if there is a match, the system generates an alert. The quality of the system clearly depends on the quality of the signatures, and therefore maintaining and keeping the signatures updated is one of the main challenges of the system. The race between the attacker, who tries to create an exploit for a newly known vulnerability, and the defender (security operator), who attempts to create a signature that detects attacks against that vulnerability, is often a race against time.

Here is an example of an IDS (Snort) signature:

alert tcp any any -> $HOME_NET [80,8080] (msg:"SQL Injection Detected"; flow:established,to_server; http_uri; content:"/wordpress/wp-content/plugins/demo_vul/endpoint.php"; content:"union",distance 0; content:"select",distance 0,nocase; content:"from", distance 0; sid:123;)

This is a rule written to detect and alert on a SQL injection attempt to a web server operating on port 80 or 8080. An example would be the following:

http://acunetix.php.example/wordpress/wp-content/plugins/demo_vul/endpoint.php?user=-1+union+select+1,2,3,4,5,6,7,8,9,(SELECT+user_pass+FROM+wp_users+WHERE+ID=1)

The rule starts with the rule action, namely alert, which indicates the action that results if this rule matches. The subsequent terms indicate the protocol (tcp) that needs to be matched. The rule specifies the TCP destination ports of 80 and 8080. Typically, these will be HTTP traffic.

The msg keyword specifies the message to be included in the generated alert. The flow keyword specifies that this rule needs to be applied only to those TCP sessions that are in an ESTABLISHED state. Subsequently, the rule goes on to specify that the URI needs to contain certain specific strings.

This gives an idea and example of an IDS/IPS signature. The detailed understanding of such a signature is beyond the scope of this chapter and will be discussed in Chapter 14.

Anomaly-based intrusion detection

Anomaly-based intrusion detection detects malicious activity by how it differs from normal behavior. This often requires the system to define and/or learn normal behavior. Since the normal for one environment is often different than the normal for another environment, this approach typically requires a learning phase where the system learns the appropriate normal for a particular environment. During the learning phase, a baseline for normal activity is recorded; subsequently, in the running phase, the activity is compared against the baseline to detect anomalies.

One of the main advantages of this approach is that the anomaly-based approach does not require signatures, and the race against time for security coverage is not an issue. In other words, the anomaly-based approach can detect novel attacks that the IDS/IPS has not encountered before.

On the other hand, the main challenge for anomaly-based systems is that of false positives. Anomaly detection assumes that the outlier case is malicious. However, all outliers are not malicious, and this is the underlying reason for the high false positive rates associated with this approach. Subsequently, significant effort would be required to tune the system – to balance the false positives and false negatives.

Additionally, since the anomaly-based IDS generates alerts when there is a deviation from normal, the alert will not be specific; the system only knows that it is not normal. This results in non-specific or vague alerts being generated.

There are several sub-types of anomaly-based intrusion detection, namely the following:

• Statistical anomaly-based: In the statistical anomaly-based approach, the IDS analyzes a set of predetermined values or variables (for example, packet sizes, login session variables, packet header values, and amount of data transferred) and maintains a baseline learned during the learning phase. Subsequently, the system analyzes the set of variables at runtime for deviation from the expected baseline. The system typically has a threshold setting that can be configured, and when the deviation from the predicted baseline is greater than the threshold, it detects the activity as malicious.
• Machine learning-based: Machine learning has made significant advances, and this approach is often used to detect outliers. Therefore, the technique is very good for anomaly detection-based IDS/IPS. This is a vast topic, but various techniques under machine learning can be used to detect unknown attacks.
• Protocol anomaly-based: This approach applies mainly to network-based IDS. Network traffic typically follows various network protocols. For example, email communication typically follows a set of protocols such as SMTP, IMAP, and POP. These protocols are clearly defined by specifications described in documents called RFC. Protocol anomaly-based IDS detect a deviation of network traffic from the concerned protocol’s RFC specification.

Anomaly detection can be a very powerful technique for detecting intrusions since it can detect new and unknown attacks, provided we can overcome the challenges, including high false-positive rates and tuning difficulties. One such technique combines anomaly detection with signature-based detection to create a hybrid solution.

Hybrid intrusion detection

As the name suggests, hybrid IDS combine signature-based and anomaly-based approaches to detect malicious activity. In the simplest design, the network traffic is processed by a signature-based component as well as an anomaly-based component, and the findings of each component are fed into a decision module that makes a final judgment on whether there is an attack or not.

In a more practical sense, typical IDS/IPS will be signature-based but may have some detection modules that work using an anomaly-based approach.

In the next section, let us discuss the state of the art in IDS/IPS. The section will discuss the important features present in the latest IDS/IPS.

The intrusion detection and prevention field has been evolving for a few decades. During this period, several commercial and open source IDS/IPS have been developed. As the nature of the internet and its protocols, as well as the complexity of threats, evolved, the IDS/IPS also had to evolve in order to keep up with the threats. Snort is an open source IDS/IPS that was created in 1998, and over the past 20+ years, it has evolved into one of the leading IDS/IPS software. Bro is another open source project, which started in 1994 and was mainly used in an academic setting for several years. Recently, it was renamed Zeek, and a community has formed around the open source project. Suricata is a relatively late player in the game and was created in 2009. It is a signature-based IDS/IPS similar to Snort. The rule syntax for Suricata is very similar to that of Snort. In addition to the rules, Suricata has many other similarities to Snort in functionality – although the design and implementation are completely different.

These three open source IDS/IPS have kept up with the challenges that they faced and stood the test of time. It may be said that the current state of these three IDS/IPS represents the state of the art in IDS/IPS. In this section, let us describe some of the challenges that these systems have faced and what features solved them.

Stateful analysis

Stateful analysis of the various network protocols is a necessary feature in any IDS/IPS. Snort was completely stateless and basically a packet analysis IDS in the initial years. Even when stateful analysis was introduced in the subsequent years, it was incomplete and insufficient. Ideally, the IDS/IPS must analyze the network traffic exactly as the end hosts would analyze it. This means that the IDS would need to maintain a very similar state to the end hosts. This is not a trivial task. This is the reason why it took decades for Snort to improve its stateful analysis functionality. Currently, one could say that Snort is a stateful IDS/IPS device, even though there are still limitations.

Fast packet acquisition

Historically, IDS/IPS devices used the packet capture library called libpcap. This is a library used by the tcpdump project and was available as open source. libpcap worked great, but as the internet speed increased, this library started becoming a performance bottleneck. In the case of libpcap, the packet data (network traffic) had to be copied several times before reaching the IDS for processing, and this was one of the reasons for the performance issue. Currently, the state of the art uses zero-copy mechanisms in order to improve performance. Although Snort still supports and offers libpcap-based packet acquisition, it offers all the latest packet acquisition mechanisms to be used.

Parallel processing

The state of the art is for the IDS/IPS to perform network traffic analysis using parallel processing – this could be a multi-process-based or multi-thread-based design. Snort started as a single-threaded, single-process IDS, and then evolved into a multi-process design. Currently, Snort uses a multi-threaded design.

In a multi-process and multi-threaded design, an incoming session would be processed by one of the processes or threads. Once a session is analyzed by a process or thread, then all the subsequent network packets for that session will be analyzed by that process or thread. This is called session pinning. Typically, such pinning is based on a hashing approach, where the hash will be based on the source and destination IP addresses, port, and protocol. However, in this approach, two related sessions that hash to two separate processes or threads will result in a lesser-grade analysis.

Pattern matching

Pattern matching has been and is still one of the most important features of IDS/IPS. A single signature may contain several pattern matches. Originally, these were evaluated one rule at a time, one pattern at a time. With time, multi-pattern search algorithms were used in order to speed up the rule processing.

In addition, as opposed to the crude pattern matching of the past, current IDS/IPS devices perform the pattern matching with context. For example, when a pattern is specified, it can also be specified what data to match against – HTTP URI, HTTP header, and so on. This improves the performance since the pattern search can be limited to specific data, and it also improves detection accuracy.

Extending rule language

Most IDS/IPS have a rich rule language. However, there will always be cases that cannot be covered by the limited capability offered by the rule language. Each system – Snort, Suricata, and Zeek – has its own approach to this challenge. Zeek from the Bro days had a full-fledged language to write detections in. So, the challenge really did not apply to Zeek. Snort came up with shared object (SO) rules, whereby custom C code could be written for a particular functionality and released as .so files in a release. Suricata integrated Lua scripting as part of the rule language extension.

App and protocol identification

Historically, Snort rules were based on protocol and port. For example, the rule would specify that it applies to TCP and on ports 80, 8080, and 3128. The list of ports could be more extensive to cover the usual HTTP ports. However, if there is an HTTP session on port 1000, the rule will not be applied against that session. This challenge was solved by introducing the app and protocol identification feature, which is a state-of-the-art feature. All leading IDS/IPS detect the various protocols on any random port to perform the analysis correctly.

File analysis

In certain cases, the IDS/IPS must analyze the data not as a stream of bytes but as a file. This feature is also the state of the art and is part of all leading IDS/IPS.

These standard features represent the state of the art in IDS/IPS. The intent of this discussion was not to present a comprehensive set of features but to give an idea of the various features.

The next section discusses the various metrics that are used to evaluate IDS/IPS. These metrics try to measure how effective the IDS/IPS are from an accuracy perspective, as well as how efficiently they do their tasks.

It is essential to be familiar with a few key metrics that are often used to describe how capable an IDS/IPS is. An IDS/IPS has two main metric classes: detection accuracy and performance metrics. These metrics are mainly used to compare IDS/IPS, which are also known as IDS/IPS evaluations. We will look at these topics in this section.

Detection accuracy

Every packet or connection analyzed has two possibilities – benign or malicious. Also, there are two possibilities for IDS analysis results – an alert is generated or no alert is generated. So, we end up with four possibilities, as described in the following table. This table is called a confusion matrix and is a valuable way to measure the performance of an IDS in classifying the connections or sessions as benign or malicious.

Table 1.1 – Intrusion detection confusion matrix

Let’s look at each of these cases:

• True positive (TP): TP is the case when the connection is malicious and the IDS correctly alerts.
• True negative (TN): TN is the case when the connection is benign and the IDS correctly avoids generating an alert. Ideally, the TN rate should be 100%; this means that 100% of benign connections will result in an absence of an alert.
• False positive (FP): FP is the case when the connection is benign and the IDS incorrectly generates an alert. Ideally, the FP rate should be 0%, meaning that the IDS does not generate any alerts for benign connections.
• False negative (FN): FN is the case when the connection is malicious and the IDS incorrectly fails to generate an alert. Ideally, the FN rate should be 0%, meaning that the IDS does not fail to generate an alert for malicious connections.

Now, the metrics used for detection accuracy are as follows:

• True positive rate (TPR): The TPR is calculated as the ratio of accurately detected attacks (TP) to the total number of attacks (TP + FN). Note that the total number of attacks is equal to the number of attacks detected (TP) plus the number of attacks missed (FN).

Ideally, the TPR should be equal to 1, which means FN should be 0; in other words, the IDS alerts on all the attacks.

• False positive rate (FPR): The FPR is calculated as the ratio of wrongly detected attacks (FP) to the total number of benign connections (FP + TN).

Ideally, the FPR should be equal to 0, which means FP should be 0; in other words, the IDS does not alert on any of the benign connections.

• Precision rate (PR): The PR is calculated as the ratio of accurately detected attacks (TP) to the total number of alerts generated (TP + FP).

Ideally, the PR should be equal to 1, which means TP should be non-zero and FP should be 0; in other words, all the alerts generated by the IDS should be for attacks.

The preceding metrics are useful in measuring the detection accuracy of IDS or IPS. The values of these metrics are useful for comparison purposes (to choose one system over another) or for benchmarking purposes (to measure the improvement of a system over time).

Generic versus specific signatures – a discussion

For a signature-based IDS/IPS, these metrics mostly depend on how specific or generic the signatures are. If a signature is too generic, it tends to have a good TPR, but at the same time, the FPR also increases. On the other hand, if a signature is too specific, it will result in low FPRs. However, it will also result in a miss when there is a slight modification to the attack; that is, FN increases.

Performance-related IDS/IPS metrics

The traffic volume on the internet, as well as the traffic volume within any network, has been increasing year by year. Figure 1.6 shows how the traffic volume has increased across the internet over the past several decades.

Figure 1.6 – The traffic volume on the internet

In addition, the network IDS/IPS is typically deployed in key points in the network where it must monitor the traffic to and from the entire network. IDS and IPS must perform at an efficient pace to keep up with the increasing traffic loads.

The complexity of the analysis performed by the IDS affects this rating. More complex analysis (for example, a higher number of signatures to check, or more complex signatures to check) leads to an increase in the IDS processing time for a packet. Typical IDS and IPS have configurations, which include various parameter settings that control their behavior, as well as the total database of signatures to match against. By controlling the configuration, we can control the packet processing time of the IDS/IPS, thereby affecting the throughput that can be sustained.

The following metrics are often used to measure the performance of an IDS/IPS:

• Throughput: This is the maximum amount of network data that can be analyzed by the IDS without packet drops. This is measured in bits per second (or megabits per second or gigabits per second).
• Latency: This metric is only applicable to IPS devices since it works in an inline (not offline, passive) fashion. The network traffic traverses the IPS, and packets are forwarded only after the IPS has evaluated it. This introduces a delay in the network traffic, which is measured by this metric. The higher the latency, the worse the performance of the system. This is typically measured in nanoseconds or microseconds.
• Packets per second: This is the maximum number of packets per second that can be analyzed by the IDS without packet drops. This is measured in the number of packets per second. Not all packets are the same; some packets take more time to be analyzed than others. So, this number has to be measured while maintaining the traffic profile as normal as the IDS would typically analyze.
• Packet drop rate: This is the rate that indicates the number of packets that are dropped by the IDS. This is usually specified in the number of packets per second.
• TCP connections per second: This is the rate of TCP connections that can be analyzed by the IDS. This is measured in connections per second.
• Simultaneous TCP connections: This metric indicates the number of TCP connections that the IDS can analyze simultaneously. To analyze a TCP connection, the IDS needs to maintain the TCP state and other data structures, which consume memory. Subsequently, this metric indirectly measures how much memory capacity the IDS has.

The preceding IDS/IPS metrics are useful for the performance evaluation of the system. In order to enable businesses to operate well as well as to provide protection, IDS and IPS devices must be highly efficient.

IDS/IPS evaluation and comparison

IDS/IPS evaluation is a process that involves a series of tests and/or experiments in order to measure the detection accuracy as well as the performance of the system. DARPA evaluations concentrated on detection accuracy during the early years of IDS evolution. Organizations such as ICSA Labs and NSS Labs conduct a series of tests that measure the detection accuracy as well as performance ratings of IDS.

These evaluations have to be taken with a grain of salt since the results will depend very much on the selection of attacks as well as the selection of traffic profiles. However, these results are still highly beneficial and help companies narrow down the IDS/IPS solutions to be evaluated in their environment with their particular traffic and test conditions.

Next, let us look at one of the challenges faced by IDS/IPS – IDS evasions. This is the scenario where the attacker is able to conduct an attack through the IDS without getting detected.

An advanced adversary may attack the monitoring infrastructure itself so that their actions are not detected. These approaches may involve the use of evasive techniques to trick the IDS/IPS and avoid detection. Alternatively, they may target the IDS and IPS device itself to render them less effective and thereby avoid detection.

IDS/IPS evasions

IDS/IPS evasion is a technique used by an adversary to trick the IDS or IPS to conclude that there is no attack occurring when there in fact is (evasion), or to conclude that there is an attack occurring when there in fact isn’t (insertion).

IDS and IPS are separate entities from end hosts, and there are inherent differences in what network traffic they see and how they process the traffic. Due to these differences, it is possible to craft the traffic in such a way as to trick the IDS or IPS device.

An example of an evasion case is as follows:

Figure 1.7 – An IDS evasion example

Figure 1.7 shows a typical scenario for an IDS evasion. The box marked R is an internet protocol router. According to the IP protocol, as the IP datagram is being routed from the source to the destination, the IP datagram’s time-to-live (TTL) value is decremented by 1 at each router.

In this case, the attacker manipulates the packet TTL values such that all of them are seen by the IDS, but only some of them reach the end host. Thereafter, the attacker sends the following:

1. Packet 1: Data: “ATT”.
2. Packet 2: Data: “END” (TTL 1).
3. Packet 2 (Retransmitted): Data: “ACK” (TTL 10).

The attacker sends the attack in two separate packets. Let us imagine that the IDS favors older data when reassembling the segments. So, the IDS reassembles the data (concatenates Packet 1 and Packet 2) as ATTEND. On the other hand, due to the TTL manipulation, the second packet does not reach the end host. So, the end host reassembles the data as ATTACK, which means a successful attack. The IDS, however, fails to generate an alert because it concluded the data as ATTEND.

If the attacker knows that the IDS does not validate packet checksums, the same test as the previous can be repeated, and instead of manipulating the TTL values, they can send the second packet with an invalid checksum, which will be processed by the IDS but discarded by the end host.

The technique for evasion can be adjusted based on the type of difference between the IDS processing and endpoint processing.

Attacks against the IDS/IPS

We saw that the IDS and IPS play a key role in the security posture of an organization. We noted earlier that IDS and IPS processing is complex and involved. In the previous section, we discussed how the adversary tries to go unnoticed during an attack by evasion (and insertion) techniques. In another tactic, the adversary could attack the IDS/IPS itself to render it partially or completely useless for a short or long duration. Such attacks against the IDS/IPS can be classified into two main types:

• Crash attack: This is a class of attack that tries to send some network traffic that causes the IDS to crash when processing it. For example, the IDS could have a buffer overflow vulnerability in its decoder or detector module, which is triggered when processing a certain type of traffic. A very early version of Snort (version 1.8) has a similar vulnerability when processing RPC traffic, causing a buffer overflow. An attacker knowing such a vulnerability could target the IDS and cause a crash, which would lead to degraded operation and missed attacks during that time window.
• Denial-of-service attack: In this approach, the attacker sends traffic that causes the IDS to spend a large amount of time processing it. A lot of such traffic can cause the IDS to go to a state where it cannot keep up and will start dropping packets. This can lead to degraded operations and missed attacks.

A robust IDS design would consider such attacks against itself and have mechanisms to defend against such attacks.

This chapter provided a brief introduction to IDS and IPS, including the need for them and the role these systems play in the defense-in-depth strategy. The chapter then discussed the different types of IDS, the current state of the art in the field, and some of the key metrics used to evaluate IDS and IPS.

In the following chapter, we will look at Snort, one of the most popular open source IDS/IPS and discuss the evolution of Snort from its early stage to where it is now.