Streaming transactions to Kafka

In this section, we are going to write a program that produces a stream of the BTC/USD transactions that happen in real time. Our program will:

1. Subscribe to the WebSocket API of Bitstamp to get a stream of transactions in JSON format.

2. For each transaction coming in the stream, it will:

   • Deserialize it

   • Convert it to the same Transaction case class that we used in BatchProducer in the previous chapter

   • Serialize it

   • Send it to a Kafka topic

In the next section, we will use Zeppelin again with Spark Streaming to query the data streamed to the Kafka topic.

Subscribing with Pusher

Go to Bitstamp's WebSocket API for live transactions: https://www.bitstamp.net/websocket/.

You will see that this API uses a tool called Pusher channels for real-time WebSocket streaming. The API documentation provides a Pusher Key that we need to use to receive live transactions.

Pusher channels is a hosted solution for delivering a stream of messages using a publish/subscribe pattern. You can find...