Understanding the Arrow format and specifications

The Apache Arrow documentation states the following [1]:

Apache Arrow is a development platform for in-memory analytics. It contains a set of technologies that enable big data systems to process and move data fast. It specifies a standardized language-independent columnar memory format for flat and hierarchical data, organized for efficient analytic operations on modern hardware.

Well, that’s a lot of technical jargon! Let’s start from the top. Apache Arrow (just Arrow for brevity) is an open source project from the Apache Software Foundation (https://apache.org) that is released under the Apache License, version 2.0 [2]. It was co-created by Jacques Nadeau and Wes McKinney, the creator of pandas, and first released in 2016. Simply put, Arrow is a collection of libraries and specifications that make it easy to build high-performance software utilities for processing and transporting large datasets. It consists of...