You're reading from In-Memory Analytics with Apache Arrow Perform fast and efficient data analytics on both flat and hierarchical structured data

Product type Paperback

Published in Jun 2022

Publisher Packt

ISBN-13 9781801071031

Length 392 pages

Edition 1st Edition

Languages

Tools

Apache arrow

Concepts

Big Data

Author (1):

Matthew Topol

View More author details

Table of Contents (16) Chapters

Preface

1. Section 1: Overview of What Arrow Is, its Capabilities, Benefits, and Goals

2. Chapter 1: Getting Started with Apache Arrow FREE CHAPTER

3. Chapter 2: Working with Key Arrow Specifications

4. Chapter 3: Data Science with Apache Arrow

5. Section 2: Interoperability with Arrow: pandas, Parquet, Flight, and Datasets

6. Chapter 4: Format and Memory Handling

7. Chapter 5: Crossing the Language Barrier with the Arrow C Data API

8. Chapter 6: Leveraging the Arrow Compute APIs

9. Chapter 7: Using the Arrow Datasets API

10. Chapter 8: Exploring Apache Arrow Flight RPC

11. Section 3: Real-World Examples, Use Cases, and Future Development

12. Chapter 9: Powered by Apache Arrow

13. Chapter 10: How to Leave Your Mark on Arrow

14. Chapter 11: Future Development and Plans

15. Other Books You May Enjoy

Arrow format versioning and stability

In order to ensure confidence that updating the version of the Arrow library in use won't break applications and the long-term stability of the Arrow project, there are two versions used to describe each release of the project: The format version and the library version. Different library implementations and releases can have different versions, but will always be implementing a specific format version. From version 1.0.0 onward, semantic versioning is used with releases.

Provided the major version of the format is the same between two libraries, any new library is backward-compatible with any older library with regards to being able to read data and metadata produced by an older library. Increases in the minor version of the format, such as an increase from version 1.0.0 to version 1.1.0, indicate new features that were added. As long as these new features are not used (such as new logical types or physical layouts), older libraries will be able to read data and metadata produced by newer versions of the libraries.

As far as the long-term stability of the format and libraries, only increases in the major version of the format would indicate any issue with the previous guarantees about compatibility. The Arrow project says that they do not expect this to be a frequent occurrence, rather it would be an exceptional event, in which case such a release would exercise caution for deployment. As a result of these compatibility guarantees, it ends up being safe and simple to ensure backward and forward compatibility when using the Arrow libraries and format.