Understanding queues and buffers

We discussed in the previous chapter how if the number of events being ingested in QRadar is more than the license threshold, a system notification is sent by the console on the UI. Let us dig deeper to discover how events are managed in different scenarios.

Persistent queues

QRadar changed its design concept to introduce persistent queues. This was primarily done to avoid event loss. We understand that there are three basic services in the event pipeline:

• ecs-ec-ingress
• ecs-ec
• ecs-ep

Once the events hit the event pipeline, QRadar ensures that they are ingested successfully.

Imagine that the ecs-ec service has crashed. What will happen to the incoming events? ecs-ec-ingress will still be collecting events and trying to send them to ecs-ec for parsing. If the ecs-ec service is down, then the events coming in are stored temporarily in the ingress persistent queue. Once the ecs-ec service is up and running again, the events...