Spark Streaming - introduction and architecture

Spark Streaming is a very useful extension to the Spark core API that's being widely used to process incoming streaming data in real-time or close to real-time as in near real-time (NRT). This API extension has all the core Spark features in terms of highly distributed, scalable, fault tolerant, and high throughput, low latency processing.

The following diagram captures how Spark Streaming works in close conjunction with the Spark execution engine to process real-time data streams:

Spark Streaming works on microbatching based architecture --we can envision it as an extension to the core Spark architecture where the framework performs real-time processing by actually clubbing the incoming events from the stream into deterministic batches. Each batch is of the same size, and the live data is collected and stacked into these deterministically sized microbatches for processing.

Under the Spark framework, the size of each microbatch is determined using...