The Photon Engine
In this chapter, we will turn our attention back to SQL Warehouses. This time, however, we will focus on the query engine running on SQL Warehouses. The query engine is known as Photon Engine. We will begin by learning about Photon Engine and its place in the Apache Spark framework. Going ahead, we will understand the core engineering philosophy of Photon Engine. Finally, we will go through its limitations and the roadmap to overcome them.
I do want to highlight that you don’t need to learn the details of Photon Engine to work with Databricks SQL. This chapter is intended for those who are interested in how Databricks SQL achieves record-beating query performances on the Lakehouse setup with open source storage formats.
In this chapter, we will cover the following topics:
- Understanding Photon Engine
- Understanding vectorization
- Discussing the Photon product roadmap
