Reading streaming data from Apache Kafka

In this recipe, you will learn how to use Apache Kafka connectors for structured streaming in Azure Databricks to read data from Apache Kafka and write the streaming data to Delta tables and to Parquet files in the default DBFS location.

Getting ready

Before you start working on this recipe, please ensure you have gone through all the following points.

• Creating an Azure HD Insight Apache Kafka cluster (version 3.6 or later) that has Kafka version Kafka 0.10+. Perform the steps mentioned in the following document:

  https://docs.microsoft.com/en-us/azure/hdinsight/kafka/apache-kafka-get-started#:~:text=Table%201%20%20%20%20Property%20%20,a%20region%20...%20%205%20more%20rows

• You can use the following Python script, https://github.com/PacktPublishing/Azure-Databricks-Cookbook/blob/main/Chapter04/PythonCode/Kafka_producer.py, which will push the data to the Apache Kafka cluster as a streaming data producer.
• To run the Python...