What this book covers
Chapter 1, Introducing the Big Data Technology Landscape and Analytics Platform, sets the context by providing an overview of the Big Data technology landscape, the various kinds of data processing that are handled on Big Data platforms, and the various types of platforms available for performing analytics. It introduces the paradigm of distributed processing of large data in batch and real-time or near real-time. It also talks about the distributed databases to handle high velocity/frequency reads or writes.
Chapter 2, Getting Acquainted with Storm, introduces the concepts, architecture, and programming with Apache Storm as a real-time or near real-time data processing framework. It talks about the various concepts of Storm, such as spouts, bolts, Storm parallelism, and so on. It also explains the usage of Storm in the world of real-time Big Data analytics with sufficient use cases and examples.
Chapter 3, Processing Data with Storm, is focused on various internals and operations, such as filters, joins, and aggregators exposed by Apache Storm to process the streaming of data in real or near real-time. It showcases the integration of Storm with various input data sources, such as Apache Kafka, sockets, filesystems, and so on, and finally leverages the Storm JDBC framework for persisting the processed data. It also talks about the various enterprise concerns in stream processing, such as reliability, acknowledgement of messages, and so on, in Storm.
Chapter 4, Introduction to Trident and Optimizing Storm Performance, examines the processing of transactional data in real or near real-time. It introduces Trident as a real time processing framework which is used primarily for processing transactional data. It talks about the various constructs for handling transactional use cases using Trident. This chapter also talks about various concepts and parameters available and their applicability for monitoring, optimizing, and performance tuning the Storm framework and its jobs. It touches the internals of Storm such as LMAX, ring buffer, ZeroMQ, and more.
Chapter 5, Getting Acquainted with Kinesis, talks about the real-time data processing technology available on the cloud—the Kinesis service for real-time data processing from Amazon Web Services (AWS). It starts with the explanation of the architecture and components of Kinesis and then illustrates an end-to-end example of real-time alert generation using various client libraries, such as KCL, KPL, and so on.
Chapter 6, Getting Acquainted with Spark, introduces the fundamentals of Apache Spark along with the high-level architecture and the building blocks for a Spark program. It starts with the overview of Spark and talks about the applications and usage of Spark in varied batch and real-time use cases. Further, the chapter talks about high-level architecture and various components of Spark and finally towards the end, the chapter also discusses the installation and configuration of a Spark cluster and execution of the first Spark job.
Chapter 7, Programming with RDDs, provides a code-level walkthrough of Spark RDDs. It talks about various kinds of operations exposed by RDD APIs along with their usage and applicability to perform data transformation and persistence. It also showcases the integration of Spark with NoSQL databases, such as Apache Cassandra.
Chapter 8, SQL Query Engine for Spark – Spark SQL, introduces a SQL style programming interface called Spark SQL for working with Spark. It familiarizes the reader with how to work with varied datasets, such as Parquet or Hive and build queries using DataFrames or raw SQL; it also makes recommendations on best practices.
Chapter 9, Analysis of Streaming Data Using Spark Streaming, introduces another extension of Spark—Spark Streaming for capturing and processing streaming data in real or near real-time. It starts with the architecture of Spark and also briefly talks about the varied APIs and operations exposed by Spark Streaming for data loading, transformations, and persistence. Further, the chapter also talks about the integration of Spark SQL and Spark Streaming for querying data in real time. Finally, towards the end, it also discusses the deployment and monitoring aspects of Spark Streaming jobs.
Chapter 10, Introducing Lambda Architecture, walks the reader through the emerging Lambda Architecture, which provides a hybrid platform for Big Data processing by combining real-time and pre-computed batch data to provide a near real-time view of the data. It leverages Apache Spark and discusses the realization of Lambda Architecture with a real life use case.
