Spark Streaming overview

Spark Streaming is an extension of the core Spark API that enables scalable and fault-tolerant, stream-oriented processing of data. Spark provides the ability to stream data from multiple sources, with a number of key sources being the following:

• Apache Kafka
• Amazon Kinesis and S3
• TCP
• HDFS

Spark offers two flavors of streaming:

• Spark Structured Streaming that is built on top of the Spark SQL engine
• Spark Discretized Stream (DStream), which uses a discretized stream—that is, a continuous stream of data

In this section, we will be exploring Spark DStreams and develop an understanding of how this could be leveraged to develop streaming solutions.

Let's start with a classic word count problem, where we are trying to count the frequency of each distinct word.

...