Commonly Supported File Formats
We've already seen the ease with which you can manipulate text files using Spark with the textFile() method on SparkContext. However, you'll be pleased to know that Apache Spark supports a large number of other formats, which are increasing with every release of Spark. With Apache Spark release 2.0, the following file formats are supported out of the box:
- TextFiles (already covered)
- JSON files
- CSV Files
- Sequence Files
- Object Files
Text Files
We've already seen various examples in Chapter 1, Architecture and Installation and Chapter 2, Transformations and Actions with Spark RDDs on how to read text files using the textFile() function. Each line in the text file is assumed to be a new record. We've also seen examples of wholeTextFiles(), which return a PairRDD, with the key being the identifier of the file. This is very useful in ETL jobs, where you might want to process data differently based on the key, or even pass that on to downstream processing.
An...
