Chapter 12 – Working with Big Data

Courses on Hadoop

Both Yahoo and Google have great tutorials on Hadoop, which go from beginner to quite advanced levels. They don't specifically address using Python, but learning the Hadoop concepts and then applying them in Pydoop or a similar library can yield great results.

Yahoo's tutorial: https://developer.yahoo.com/hadoop/tutorial/

Google's tutorial: https://cloud.google.com/hadoop/what-is-hadoop

Pydoop

Pydoop is a python library to run Hadoop jobs—it also has a great tutorial that can be found here: http://crs4.github.io/pydoop/tutorial/index.html.

Pydoop also works with HDFS, the Hadoop File System, although you can get that functionality in mrjob as well. Pydoop will give you a bit more control over running some jobs.

Recommendation engine

Building a large recommendation engine is a good test of your Big data skills. A great blog post by Mark Litwintschik covers an engine using Apache Spark, a big data technology: http...