Using file formats for data storage in data lakes

The file format you choose to store data in a data lake is key in determining the ease of data storage and retrieval, query performance, and storage space. So, it is vital that you choose the optimal data format that can balance these factors. Data storage formats can be broadly classified into structured, unstructured, and semi-structured formats. In this section, we will explore each of these types with the help of code examples.

Unstructured data storage formats

Unstructured data is any data that is not represented by a predefined data model and can be either human or machine-generated. For instance, unstructured data could be data stored in plain text documents, PDF documents, sensor data, log files, video files, images, audio files, social media feeds, and more.

Unstructured data might contain important patterns, and extracting these patterns could lead to valuable insights. However, storing data in unstructured format...