Data plane telemetry processing

Network activities such as capacity planning, billing, or distributed denial-of-service (DDoS) attack monitoring require insights into the traffic flowing through a network. One way we can offer such visibility is by deploying a packet sampling technology. The premise is that at a high-enough rate, it’s possible to capture only a randomly sampled subset of packets to build a good understanding of the overall network traffic patterns.

While it’s the hardware that samples the packets, it’s the software that aggregates them into flows and exports them. NetFlow, sFlow, and IP Flow Information Export (IPFIX) are the three main protocols we use for this, and they define the structure of the payload and what metadata to include with each sampled packet.

One of the first steps in any telemetry processing pipeline is information ingestion. In our context, this means receiving and parsing data plane telemetry packets to extract and process...