Always have a plan
Due to the columnar nature of the Arrow format, along with the separation of the validity bitmap from the data buffers, it’s easy to see why it is a popular choice for interoperability between systems. Some query engines, such as InfluxDB, have adopted Arrow as their internal format due to its efficiency when it comes to performing computations. Others, such as DuckDB and Velox, have internal representations that are nearly identical to Arrow, providing zero-copy interactions with the rest of the ecosystem. We’ll touch more on those projects in a later chapter, but for now we’re going to examine Acero, a reference implementation of an execution engine using Arrow as its internal data representation.
One thing I want to stress here is that in most cases, while the compute functions are extremely useful, Acero is not intended to be used directly by data scientists. Typically, users would use some kind of frontend (i.e., pandas, Ibis, or a SQL...
