In this chapter, we introduced deep learning in Spark. We discussed various types of deep neural networks and their application. We also explored a few code examples provided in the BigDL distribution. As this is a rapidly evolving area in Spark, presently, we expect these libraries to provide a lot more functionalities using Spark SQL and the DataFrame/Dataset APIs. Additionally, we also expect them to mature and become more stable over the coming months.
In the next chapter, we will shift our focus to tuning Spark SQL applications. We will cover key foundational aspects regarding serialization/deserialization using encoders and the logical and physical plans associated with query executions, and then present the details of the cost-based optimization (CBO) feature released in Spark 2.2. Additionally, we will present some tips and tricks that developers can use to improve...
