The Console is the brain of QRadar and is the single indispensable component of QRadar. It can collect and process data and throw alerts based on the rules. This is the primary job of the Console. Other components (described later) are mostly used to scale these functionalities in one form or another. Now, let us look at the three major services running on the Console and understand them.

Tomcat

The primary utility of this service is for displaying the User Interface (UI) of QRadar. The QRadar UI can be accessed by typing the IP address or the hostname (if it can be resolved) in the browser.

If the tomcat service is down on QRadar, you will not be able to load the QRadar UI. It maintains the user sessions, active sessions, and current users – all those who have logged in to QRadar UI. It also plays a part in authenticating users, whether it is local authentication, Lightweight Directory Access Protocol (LDAP) authentication, or any other type of authentication. It is a multithreaded service that also deals with the export of data from the QRadar UI. The status of the tomcat service can be checked using a simple command:

 systemctl status tomcat

We will cover troubleshooting tricks and tips for tomcat in the final chapter.

Important note

The Tomcat service is only available on the QRadar Console.

Hostcontext

When the hostcontext service is started, it triggers many other services with it. All the functionalities of QRadar are dependent on the hostcontext service. This service is part of all the QRadar managed hosts, unlike the tomcat service. The hostcontext service is responsible for replicating the deployment changes from the Console to other managed hosts.

The following is the list of services triggered because of hostcontext:

• ecs-ec-ingress: The specialty of this service from the following services is that even if the hostcontext service is stopped from the command line or if the hostcontext service crashes, ecs-ec-ingress keeps running and collecting events from Log Sources. If the ecs-ec-ingress service is stopped, there are two ways of starting:
  1. Restarting the hostcontext service
  2. Starting the ecs-ec-ingress service separately
• ecs-ec: The primary function of this service is to parse (map) the incoming events/flows. This service converts the events into a form that QRadar understands. The event is mapped to its event name by this service. For example, Linux OS has sent an authentication event to QRadar that there is an invalid user named testdev trying to log in via SSH.

The payload of such an event would look like this:

"Apr  10 18:26:40 servername sshd[26388]: input_userauth_request: invalid user testdev
"

QRadar needs to make sense of this payload. This is called parsing. Then, QRadar needs to map this event to an appropriate event name, which is called event mapping. The event will be parsed as follows:

• Time: 6:26:40 p.m.
• Date: 10th April
• Event Name: Authentication Failure
• Server Details: servername

So, the two important functions of ecs-ec are as follows:

• Event parsing
• Event mapping

• ecs-ep: Once the events are parsed and mapped, they need to be processed. Rules are provided as a part of the initial installation. These rules can further be customized as per the security use cases of the organization. ecs-ep is responsible for matching each incoming event against all the enabled rules. If the rule conditions are fulfilled based on the incoming event/events, offenses (security alerts) can be triggered based on the rule action and rule response (defined).

For example, we could have a rule to trigger an offense if we receive an event called Authentication Failure from Linux OS after 6 p.m. In such a case, looking at the previous event in the example, an offense will be generated.

ecs-ep is also responsible for offense management in terms of the following:

• Offense creation
• Renaming offenses
• Attaching events to triggered offenses
• Offenses in a dormant condition
• Offense closure
• Offense deletion

• QFlow: This service is responsible for collecting flows in QRadar. Flows are network packet information collected in a specific format over a period of time.
• Accumulator: This service is responsible for creating global views in QRadar, which can be used for dashboards, reports, and so on.
• Ariel proxy: This service is responsible for relaying the search queries to the appropriate managed hosts.
• Ariel query: This service queries ariel databases across all managed hosts based on the query run on the Console.

There are many other services, and we will discuss them in detail when introducing the concepts related to them.

Hostservices

This service is also part of the Console and other managed hosts. Typically, QRadar has two types of databases:

• Ariel, which is not managed by hostservices (it stores event/flow data)
• Postgres, which is managed by hostservices (it stores configuration)

An ariel database has security data that is collected, and processed by hostcontext. Postgres has the configuration details of QRadar. This is managed by the hostservices service. Postgres is a relational database management system (RDBMS) that has multiple tables containing information on QRadar deployment, configuration, and settings. The Postgres database is replicated on different managed hosts using the hostcontext service. The Postgres database can be queried, if required, using the psql command line. We will discuss this in detail when we talk about optimization and tuning QRadar.

The brain behind QRadar is the Console, and the other components act as auxiliary parts of the system helping the Console perform the functions in a better way. Before jumping on to other QRadar components, let's first discuss and understand event data and flow data.

Every organization that plans on building a SOC has hundreds and thousands of applications, servers, and endpoints that it would like to monitor. Each of these applications or servers has security and audit logs. These security and audit logs are what we call event data. QRadar supports multiple protocols such as Syslog, JDBC, and UDP multiline. It also supports product-specific protocols such as the Akamai Kona REST API protocol and the CISCO NSEL protocol. Using these protocols, QRadar can either pull data from Log Sources or we can configure Log Sources to send data to QRadar.

The data sent is in the form of events that are parsed and mapped to certain event names. The events can be viewed from the QRadar UI from the Log Activity tab. There are multiple query options that can be used.

The following figure shows a screenshot of the Log Activity tab with filters applied:

Figure 1.1 – Log Activity tab

Here, we refer to Data Source as Log Source, and they can be used interchangeably.

Event Processor

We previously learned that the Console is the brain of QRadar. However, there is a limit to the amount of data that can be collected and processed by the Console. This is where the Event Processor comes into the picture. The Console can delegate the work of collecting and processing event data to the Event Processor.

The main services running on the Event Processor are as follows:

• hostcontext: There are multiple services that are triggered when the hostcontext service is started. On each managed host (Event Processor), there is a configuration file named /opt/qradar/conf/nva.hostcontext.conf. It has a parameter called 'COMPONENT_PROCESSES'. Based on the values, the services are started when hostcontext is started. The main services that are part of the hostcontext service on the Event Processor are as follows:
  • ecs-ec-ingress
  • ecs-ec
  • ecs-ep
  • Ariel query server
• hostservices: This is like what we have seen on the Console.

You will see that there is no ariel proxy service on the Event Processor. This is because the ariel proxy service is only on the Console. When we search for data on the Console, the ariel proxy service sends the query request to the Event Processor. This request is accepted and worked on by the ariel query server service.

The ariel query server queries the ariel database, which is on the Event Processor, and sends the resultant data back to the Console, where it is shown in Log Activity.

Important note

We will be using the term managed host for any QRadar component that can be managed from the Console – for example, Event Processor, Flow Processor, Event Collector, and so on. There are very few QRadar components that are not managed by the Console. We will discuss them later.

An Event Processor can collect data using ecs-ec-ingress, parse the data using ecs-ec, and process the data using ecs-ep. You will notice that there is an ariel database in each Event Processor, which means that the event data is stored locally. Only when the data is searched is the resultant data sent to the Console for display. Over a period, the resultant data on the Console is removed as per configured policies

As the collection, parsing, and processing of data are done separately, whenever we add an Event Processor to the Console, we say that is a distributed environment.

Along with the Event Processor, the Event Collector also plays an important role in QRadar deployments. Let's discuss this further.

Event Collector

The Event Collector is the component that collects the event data and then either sends it to the Event Processor (if there is one) or to the Console for storage.

In some deployments, the Log Sources (which are configured to send data to QRadar) are in different time zones or different networks, and it is not feasible to send data directly to the Event Processor or Console. In such scenarios, the Event Collector can be used to collect local (to the network) event data, parse it using DSM, and send it to the Event Processor or Console for correlation and storage.

If the connection between the Event Collector and Event Processor is lost, event data is stored locally on the Event Collector, and when the connection is restored, the data is sent to the configured Event Processor.

Important services running on the Event Collector are as follows:

• hostcontext: These are the subservices of the hostcontext service:
  • ecs-ec-ingress – To collect event data
  • ecs-ec – To parse collected event data
• hostservices: Same as in the Console

We have discussed in detail the log data from the applications, servers, and endpoints. The other most striking feature of QRadar is its ability to process flow data. Let's discuss that in the next section.

Flows are different from events. Flow data is the information of the session between two hosts. For example, if an employee logs in at 9 a.m. and starts using social media, QRadar can capture these session details between the employee’s machine and the social media site. This is done by capturing the network traffic from the span port of a switch. There are different types of flows, which we will discuss in detail later in Chapter 4. Flow data can be viewed in the Network Activity tab on the QRadar UI, as shown in the following screenshot:

Figure 1.2 – Network Activity tab

Similar to an Event Processor, we have a Flow Processor and Flow Collector for flow data. Let’s discuss them in detail next.

Flow Processor

Like the Event Processor, the Flow Processor is another managed host that collects and processes flow data. It has an ariel database where the flow data is stored, and it can be queried using the same mechanism as discussed for the Event Processor.

For the Event Processor, we had ecs-ec-ingress, which collected event data. But for the Flow Processor, we have the qflow service, which collects flows and then sends them to ecs-ec and ecs-ep for further processing and storage.

Important services running on the Flow Processor are as follows:

• hostcontext: For the Flow Processor, the 'COMPONENT_PROCESSES' parameter in the /opt/qradar/conf/nva_hostcontext.conf file has different values than in the Event Processor.
• qflow: This service is responsible for collecting the flows. The Flow Processor does NOT have the ecs-ec-ingress service
• hostservices: Same as in the Console.

Important note

A single managed host can act as both an Event Processor and Flow Processor. For this to happen, you need to choose the correct option while installing. Usually, on enterprise-level deployments, Event Processors and Flow Processors are kept separate.

Flow Collector

Like an Event Collector, a Flow Collector is used to collect flow data, analyze it, and send it to the Flow Processor or Console for processing.

The Flow Collector has a special service like Flow Processor called qflow, which collects flows. Flow sources are defined on the QRadar Console UI and then the configuration is pushed to managed hosts, thus the Flow Collector understands which flows need to be collected.

Important services running on the Flow Collector are as follows:

• hostcontext: The subservices of the hostcontext service are as follows:
  • qflow – This service is responsible for collecting flows
  • ecs-ec – This service is responsible for aggregating and analyzing flows
• hostservices: Same as in the Console

Another important component usually used in huge deployments is the Data Node. Let's see why in the next section.

Event and flow data are required for security purposes as well as for compliance. The amount of storage available on the Console and processors might not be enough for compliance.

For example, it may be mandated by Central Banks to keep event and flow data for 2 years. The available storage on processors can store data only for 6 months. In such a scenario, multiple Data Nodes can be added to a processor so that the processed data can be stored.

Adding a Data Node to deployment has two advantages:

• Increases the storage space for event and flow data
• Searches are more efficient when Data Nodes are used

Multiple Data Nodes can be attached to a single processor. One Data Node cannot be attached to multiple processors. What this means is that one Data Node will share data with just one processor.

When Data Nodes are added to the deployment, there is a process called data rebalancing that happens. The incoming data in the processor is distributed amongst the Data Nodes that are attached.

If a Data Node goes down (or crashes), the incoming data is not written to the Data Node. Once the Data Node is up, data is again rebalanced between the processor and Data Node. We will touch more on Data Nodes while discussing searches in Chapter 6.

Along with the collectors, processors, and Console, we have a few more feature-rich components that we will look at next. Some of the components that we will discuss are managed hosts, while the QRadar Packet Capture component is an unmanaged host.

Let’s get started and discuss each component in detail.

QRadar Network Insights

QRadar Network Insights (QNI) is another QRadar component that is used for capturing flow data. Now, it might be interesting to know why we have QNI when we already had a Flow Collector and Flow Processor, which were also meant to capture flow data.

The Flow Collector and Flow Processor have a service called qflow, which is responsible for collecting flow data. qflow captures the first 64 bytes of the packet and not the complete payload. This is done to extract important information from packet data. If complete packet data is saved on the Flow Processor, the amount of network packet data would be so huge that it would fill up the Flow Processor in days, if not hours. So, it is practical to capture the first 64 bytes of packet information and store it.

QNI as an appliance (we will learn what an appliance is and how it is different from software installs in Chapter 2,) has special hardware to capture huge chunks of data, process that data, extract metadata, and send it to the processor for storage. So, if the same packet capture is sent to QNI, it can capture more data than qflow will capture with the first 64 bytes of information. For example, if the payload contains information that a user tried to access Facebook or tried to upload a file on Dropbox, QNI be able to capture the hash of the file that is uploaded (if available in the payload). This hash value can then be used by QRadar to determine whether it is a known threat by comparing the hash value of the file to the threat intelligence feeds it has.

In short, QNI provides us with more detailed information about the same packet capture compared to the information if Flow Collector/Processor is used using qflow.

In a single deployment, there can be multiple QNIs installed. QNI captures the packets and sends the information/insights to the processor/Console in IPFIX format. As the insights from QNI are more detailed, granular rules can be created, providing better security coverage.

QRadar Incident Forensics

Till now, we have talked about managed hosts, which are deployed in the QRadar environment for collecting and processing data in terms of events and flows. When it comes to forensics and deep diving into a security incident, QRadar Incident Forensics (QRIF) comes into the picture.

A security incident, such as financial fraud, would require an in-depth investigation of all activities that took place over a period of time. QRIF helps in reconstructing network sessions from the packet capture available.

For example, last Friday night, someone in a bank environment downloaded a script on the server and ran it. After a while, some of the accounts were compromised and a transfer of funds took place. From events and flows, a preliminary analysis can be done. But if there was complete packet capture available from the span port on the switch where the server was connected, QRIF could recover and reconstruct the complete session based on the source and destination IP address (or any other query), and a web session showing the exact actions of the threat actor can be seen.

QRadar Packet Capture

If QRIF can reconstruct the complete sessions, it will require complete packet captures to be stored from the time of the incident. This is where QRadar Packet Capture (QPCAP) comes into the picture. QPCAP is not a managed host. This can be placed in the network where the capture port of QPCAP can collect the network packets and store them. Network packets can fill up QPCAP very quickly. So, it has large storage space and special hardware for direct I/O disk access.

Typically, QPCAP is configured to capture and store network packet data for 7 days. If needed, multiple packet captures can be used as per the requirement.

QRadar Vulnerability Manager

Along with event data and flow data, another important aspect of information from the point of view of security is the vulnerability data of the assets in the environment. Administrators should have information on the assets available and how vulnerable these assets are. QRadar Vulnerability Manager (QVM) performs active and passive scans on the assets in the network.

For example, a QVM scan can be run periodically to understand whether the patching of all Microsoft servers is completed. It helps in determining the security posture of an organization.

QVM is a complete asset management tool right from discovering assets to running remediation reports for those assets. For this, QVM seamlessly integrates with other products such as BigFix and IBM SiteProtector.

Figure 1.3 – Assets tab

In the preceding figure, the assets database can be seen. Each asset will have a unique ID, domain information, and vulnerability information associated with it.

Figure 1.4 – Vulnerabilities tab

In the preceding figure, a scan is run twice over a period. There are eight assets that were scanned both times, but the number of vulnerabilities is different. This indicates that over a period of time, new vulnerabilities were found for the same assets.

QRadar Risk Manager

QRadar Risk Manager (QRM) is a managed host that collects configuration information from the network devices in the environment. This information is then pictorially depicted in the form of network topology.

For example, in a typical data center, there are multiple network devices such as routers, firewalls, intrusion prevention devices, and so on. QRM pulls the configuration details from all the network devices and creates network topology graphs. It can also simulate any changes in the policies by calculating the risk to the environment.

Figure 1.5 – Risks tab

QRM can integrate with QVM. Risk policies can be defined on QRM. To give an example of a risk policy in QRM, a policy can be defined to find out whenever there is web traffic, but it is not HTTPS traffic. That means, whenever there is web traffic on HTTP, the policy triggers, and the information related is collected by QRM. This information is useful to change the vulnerability risk scores of the assets involved. Whenever risk scores are changed in QVM, alerts can be generated via email or dashboard alerts to get higher visibility on the high-risk assets.

The Risk Manager tab shows information about the network devices in the environment. In the upcoming versions of QRadar, QRM and QVM will be part of a single appliance and will be sold as a single solution. But the fundamentals of both QRM and QVM remain the same, as discussed in this chapter.

QRadar App Host

On QRadar, just like on our Android or Apple phones, there are applications, or apps, that can be installed. These applications enhance the usability of QRadar and also provide additional features. All the QRadar apps are publicly available on the QRadar App Exchange portal. Here is the link to the portal: https://exchange.xforce.ibmcloud.com/hub.

In the Understanding the Console section, we covered so many services and tasks that are running on the Console. Installing multiple apps on the Console will add computational overhead in terms of CPU cycles, memory utilization, and the disk space required by the apps. For this reason, we have a component called App Host, where the Console offloads all the apps.

In the following figure, we can see how different components fit together. The QRadar Console is the most important component to which other components are connected for command and control (except QPCAP).

Figure 1.6 – Sample QRadar deployment showing multiple components

From Chapter 8 onward, we will discuss at length all the important QRadar apps that can be installed on QRadar.

This first chapter was all about understanding the different IBM QRadar components. Being a highly scalable product, it is imperative that we know which components are to be used to scale the solution. We also understood how the event and flow data is handled in QRadar by different QRadar components. With that, you now have an idea of the basic architecture of QRadar components and how they fit together.

In the next chapter, we will learn how these components work together.