Streaming sources

We will not be able to cover all the stream types with practical examples in this section, but where this chapter is too small to include code, we will at least provide a description. In this chapter, we will cover the TCP and file streams and the Flume, Kafka, and Twitter streams. Apache Spark tends only to support this limited set out of the box, but this is not a problem since 3rd party developers provide connectors to other sources as well. We will start with a practical TCP-based example. This chapter examines stream processing architecture.

Note

For instance, what happens in cases where the stream data delivery rate exceeds the potential data processing rate? Systems such as Kafka provide the possibility of solving this issue by caching data until it is requested with the additional ability to use multiple data topics and consumers (publish-subscribe model).

TCP stream

There is a possibility of using the Spark Streaming Context method called socketTextStream to stream...