File formats
There are a number of file formats that are data structures by themselves. In the chapter on Hive, we saw ORC files, an optimized form of record columnar file storage. There are a few other popular file container formats supported by Hadoop. We will look at them in this section.
The Sequence file format
A Sequence file is a container format for binary key-value pairs. Each record in a Sequence file contains a key and its associated value. Sequence files are used to combine smaller files into a single large file to alleviate the small file problem in Hadoop. In this situation, the filename forms the key and the file contents are the values associated with the key. Sequence files have a broader applicability as they can be split into configurable blocks. They can be combined with fast compression methods such as LZO or Snappy and can provide speed as well as storage and bandwidth.
The next image shows the format of a Sequence file. The start of the file has a magic number, a binary...
