Inferring schema using case classes

Case classes are special classes in Scala that provide you with the boiler plate implementation of the constructor, getters (accessors), equals and hashCode, and implement Serializable. Case classes work really well to encapsulate data as objects. Readers, familiar with Java, can relate it to plain old Java objects (POJOs) or Java bean.

The beauty of case classes is that all that grunt work, which is required in Java, can be done with case classes in a single line of code. Spark uses reflection on case classes to infer schema.

How to do it...

1. Start the Spark shell and give it some extra memory:

   $ spark-shell --driver-memory 1G
   

2. Import for the implicit conversions:

   scala> import sqlContext.implicits._
   

3. Create a Person case class:

   scala> case class Person(first_name:String,last_name:String,age:Int)
   

4. In another shell, create some sample data to be put in HDFS:

   $ mkdir person
   $ echo "Barack,Obama,53" >> person/person.txt
   $ echo "George,Bush,68" >...