Chapter 4: Real-Time Data Analytics

In the modern big data world, data is being generated at a tremendous pace, that is, faster than any of the past decade's technologies can handle, such as batch processing ETL tools, data warehouses, or business analytics systems. It is essential to process data and draw insights in real time for businesses to make tactical decisions that help them to stay competitive. Therefore, there is a need for real-time analytics systems that can process data in real or near real-time and help end users get to the latest data as quickly as possible.

In this chapter, you will explore the architecture and components of a real-time big data analytics processing system, including message queues as data sources, Delta as the data sink, and Spark's Structured Streaming as the stream processing engine. You will learn techniques to handle late-arriving data using stateful processing Structured Streaming. The techniques for maintaining an exact replica...