For the portion of this chapter that describes how to set up a development environment for working with various Arrow libraries, you’ll need the following:

• Your preferred integrated development environment (IDE) – for example, VS Code, Sublime, Emacs, or Vim
• Plugins for your desired language (optional but highly recommended)
• An interpreter or toolchain for your desired language(s):
  • Python 3.8+: pip and venv and/or pipenv
  • Go 1.21+
  • C++ Compiler (capable of compiling C++17 or newer)

The Apache Arrow documentation states the following [1]:

Apache Arrow is a development platform for in-memory analytics. It contains a set of technologies that enable big data systems to process and move data fast. It specifies a standardized language-independent columnar memory format for flat and hierarchical data, organized for efficient analytic operations on modern hardware.

Well, that’s a lot of technical jargon! Let’s start from the top. Apache Arrow (just Arrow for brevity) is an open source project from the Apache Software Foundation (https://apache.org) that is released under the Apache License, version 2.0 [2]. It was co-created by Jacques Nadeau and Wes McKinney, the creator of pandas, and first released in 2016. Simply put, Arrow is a collection of libraries and specifications that make it easy to build high-performance software utilities for processing and transporting large datasets. It consists of...

There is often a lot of debate surrounding whether a database should be row-oriented or column-oriented, but this primarily refers to the on-disk format of the underlying storage files. Arrow’s data format is different from most cases discussed so far since it uses a columnar organization of data structures in memory directly. If you’re not familiar with columnar as a term, let’s take a look at what it means. First, imagine the following table of data:

Figure 1.3 – Sample data table

Traditionally, if you were to read this table into memory, you’d likely have some structure to represent a row and then read the data in one row at a time – maybe something like struct { string archer; string location; int year }. The result is that you have the memory grouped closely together for each row, which is great if you always want to read all the columns for every row or are...

As mentioned previously, the Arrow columnar format specification includes definitions of the in-memory data structures, metadata serialization, and protocols for data transportation. The format itself has a few key promises:

• Data adjacency for sequential access
• O(1) (constant time) random access
• SIMD and vectorization-friendly
• Relocatable, allowing for zero-copy access in shared memory

To ensure we’re all on the same page, here’s a quick glossary of terms that are used throughout the format specification and the rest of this book:

• Array: A list of values with a known length of the same type.
• Slot: The value in an array identified by a specific index.
• Buffer/contiguous memory region: A single contiguous block of memory with a given length.
• Physical layout: The underlying memory layout for an array without accounting for the interpretation of the logical value. For example...

To ensure confidence that updating the version of the Arrow library in use won’t break applications and the long-term stability of the Arrow project, two versions are used to describe each release of the project: the format version and the library version. Different library implementations and releases can have different versions, but they will always implement a specific format version. From version 1.0.0 onward, semantic versioning is used with releases.

Provided the major version of the format is the same between two libraries, any new library is backward-compatible with any older library with regards to being able to read data and metadata produced by an older library. Increases in the minor version of the format, such as an increase from version 1.0.0 to version 1.1.0, indicate new features that were added. So long as these new features are not used (such as new data types or physical layouts), older libraries will be able to read...

As mentioned previously, the Arrow project contains a variety of libraries for multiple programming languages. These official libraries enable anyone to work with Arrow data without having to implement the Arrow format themselves, regardless of the platform and programming language they are utilizing. There are two primary types of libraries that exist so far: ones that are distinct implementations of the Arrow specification, and ones that are built on other implementations. At the time of writing this book, there are implementations for Arrow in C++ [3], C# [4], Go [5], Java [6], JavaScript [7], Julia [8], and Rust [9], all of which are distinct implementations.

On top of those, there are libraries for C (Glib) [10], MATLAB [11], Python [12], R [13], and Ruby [14], all of which are built on top of the C++ library, which happens to have the most active development. As you might expect, the various implementations all have different stages...

By now, you should have a pretty solid understanding of what Arrow is, the basics of how it’s laid out in memory, and the basic terminology. So, let’s set up a development environment where you can test out and play with Arrow. For this book, I’m going to primarily focus on the three libraries that I’m most familiar with: the C++ library, the Python library, and the Go library. While the basic concepts will apply to all of the implementations, the precise APIs may differ between them. So, armed with the knowledge you’ve gained so far, you should be able to make sense of the documentation for your preferred language, even without precise examples for that language being printed here.

For each of C++, Python, and Go, after the instructions for installing the Arrow library, I’ll go through a few exercises to get you acquainted with the basics of using the Arrow library in that language.

The full code for all...

The goal of this chapter was to explain what Apache Arrow is, get you acquainted with the format, and have you use it in some simple use cases. This knowledge forms the baseline of everything else for us to talk about in the rest of this book!

Just as a reminder, you can check out this book’s GitHub repository (https://github.com/PacktPublishing/In-Memory-Analytics-with-Apache-Arrow-Second-Edition) for the solutions to the exercises presented here and for the full code samples so that you can make sure you understand the concepts!

The examples and exercises that were provided in this chapter are all fairly trivial and are meant to help reinforce the concepts that were introduced about Arrow’s format and specification while helping you get familiar with using Arrow in code.

In Chapter 2, Working with Key Arrow Specifications, you’ll learn how to read your data into the Arrow format, whether it’s on your local disk, Hadoop Distributed File...

Here’s a list of the references that were provided in this chapter – there were quite a lot!

1. Apache Arrow documentation: https://arrow.apache.org/docs/
2. Apache License 2.0: https://apache.org/licenses/LICENSE-2.0
3. C++ Apache Arrow documentation: https://arrow.apache.org/docs/cpp/
4. C# documentation for Arrow: https://github.com/apache/arrow/blob/master/csharp/README.md
5. Golang documentation for Arrow: https://pkg.go.dev/github.com/apache/arrow/go/v7/arrow
6. Java documentation for Arrow: https://arrow.apache.org/docs/java/
7. JavaScript documentation for Arrow: https://arrow.apache.org/docs/js/
8. Julia documentation for Arrow: https://arrow.apache.org/julia/stable/
9. Rust documentation for Arrow: https://docs.rs/crate/arrow/
10. Glib documentation for Arrow: https://arrow.apache.org/docs/c_glib/
11. MATLAB documentation for Arrow: https://github.com/apache/arrow/blob/master/matlab/README.md
12. Python documentation for Arrow: https...