Stream processing engines
A stream processing engine is the most critical component of any real-time data analytics system. The role of the stream processing engine is to continuously process events from a streaming data source and ingest them into a streaming data sink. The stream processing engine can process events as they arrive in a real real-time fashion or group a subset of events into a small batch and process one micro-batch at a time in
a near real-time manner. The choice of the engine greatly depends on the type of use case and the processing latency requirements. Some examples of modern streaming engines include Apache Storm, Apache Spark, Apache Flink, and Kafka Streams.
Apache Spark comes with a stream processing engine called Structured Streaming, which is based on Spark's SQL engine and DataFrame APIs. Structured Streaming uses the micro-batch style of processing and treats each incoming micro-batch as a small Spark DataFrame. It applies DataFrame operations...
