Summary

In this chapter, you learned about Enterprise DSS in the context of big data analytics and its components. You learned about various types of data sources such as RDBMS-based operational systems, message queues, and file sources, and data sinks, such as data warehouses and data lakes, and their relative merits.

Additionally, you explored different types of data storage formats such as unstructured, structured, and semistructured and learned about the benefits of using structured formats such as Apache Parquet with Spark. You were introduced to data ingestion in a batch and real-time manner and learned how to implement them using Spark DataFrame APIs. We also introduced Spark's Structured Streaming framework for real-time streams processing, and you learned how to use Structured Streaming to implement incremental data loads using minimal programming overheads. Finally, you explored the Lambda Architecture to unify batch and real-time data processing and its implementation...