Exploring event data

Every organization that plans on building a SOC has hundreds and thousands of applications, servers, and endpoints that it would like to monitor. Each of these applications or servers has security and audit logs. These security and audit logs are what we call event data. QRadar supports multiple protocols such as Syslog, JDBC, and UDP multiline. It also supports product-specific protocols such as the Akamai Kona REST API protocol and the CISCO NSEL protocol. Using these protocols, QRadar can either pull data from Log Sources or we can configure Log Sources to send data to QRadar.

The data sent is in the form of events that are parsed and mapped to certain event names. The events can be viewed from the QRadar UI from the Log Activity tab. There are multiple query options that can be used.

The following figure shows a screenshot of the Log Activity tab with filters applied:

Figure 1.1 – Log Activity tab

Here, we refer to Data Source as Log Source, and they can be used interchangeably.

Event Processor

We previously learned that the Console is the brain of QRadar. However, there is a limit to the amount of data that can be collected and processed by the Console. This is where the Event Processor comes into the picture. The Console can delegate the work of collecting and processing event data to the Event Processor.

The main services running on the Event Processor are as follows:

• hostcontext: There are multiple services that are triggered when the hostcontext service is started. On each managed host (Event Processor), there is a configuration file named /opt/qradar/conf/nva.hostcontext.conf. It has a parameter called 'COMPONENT_PROCESSES'. Based on the values, the services are started when hostcontext is started. The main services that are part of the hostcontext service on the Event Processor are as follows:
  • ecs-ec-ingress
  • ecs-ec
  • ecs-ep
  • Ariel query server
• hostservices: This is like what we have seen on the Console.

You will see that there is no ariel proxy service on the Event Processor. This is because the ariel proxy service is only on the Console. When we search for data on the Console, the ariel proxy service sends the query request to the Event Processor. This request is accepted and worked on by the ariel query server service.

The ariel query server queries the ariel database, which is on the Event Processor, and sends the resultant data back to the Console, where it is shown in Log Activity.

Important note

We will be using the term managed host for any QRadar component that can be managed from the Console – for example, Event Processor, Flow Processor, Event Collector, and so on. There are very few QRadar components that are not managed by the Console. We will discuss them later.

An Event Processor can collect data using ecs-ec-ingress, parse the data using ecs-ec, and process the data using ecs-ep. You will notice that there is an ariel database in each Event Processor, which means that the event data is stored locally. Only when the data is searched is the resultant data sent to the Console for display. Over a period, the resultant data on the Console is removed as per configured policies

As the collection, parsing, and processing of data are done separately, whenever we add an Event Processor to the Console, we say that is a distributed environment.

Along with the Event Processor, the Event Collector also plays an important role in QRadar deployments. Let's discuss this further.

Event Collector

The Event Collector is the component that collects the event data and then either sends it to the Event Processor (if there is one) or to the Console for storage.

In some deployments, the Log Sources (which are configured to send data to QRadar) are in different time zones or different networks, and it is not feasible to send data directly to the Event Processor or Console. In such scenarios, the Event Collector can be used to collect local (to the network) event data, parse it using DSM, and send it to the Event Processor or Console for correlation and storage.

If the connection between the Event Collector and Event Processor is lost, event data is stored locally on the Event Collector, and when the connection is restored, the data is sent to the configured Event Processor.

Important services running on the Event Collector are as follows:

• hostcontext: These are the subservices of the hostcontext service:
  • ecs-ec-ingress – To collect event data
  • ecs-ec – To parse collected event data
• hostservices: Same as in the Console

We have discussed in detail the log data from the applications, servers, and endpoints. The other most striking feature of QRadar is its ability to process flow data. Let's discuss that in the next section.