Understanding Photon Engine

In this section, we will learn about Photon Engine. We will begin by understanding its place in the Databricks SQL ecosystem. Then, we will get a bit more technical and see how Databricks SQL uses Photon Engine with Apache Spark to provide the speed and concurrency that it proclaims to have.

What is Photon?

Photon (https://databricks.com/product/photon) is a vectorized query engine that is written in C++ that can leverage the data and instruction-level parallelism available in modern-day CPUs.

Photon is 100% compatible with Apache Spark APIs. This means you do not have to learn any new language or programming paradigm or rewrite your code to be able to leverage Photon.

The mission of Photon is simple: eliminate the need for data warehouses by providing the same or better performance than data warehouses while querying data in open file formats over cheap, elastic cloud storage.

It is the compute complement to the Delta Lake file format...