Upcoming Hadoop changes

Before discussing alternative Hadoop distributions, let's look at some changes to Hadoop itself in the near future. We've already discussed the HDFS changes coming in Hadoop 2.0, particularly the high availability of NameNode enabled by the new BackupNameNode and CheckpointNameNode services. This is a significant capability for Hadoop as it will make HDFS much more robust, greatly enhancing its enterprise credentials and streamlining cluster operations. The impact of NameNode HA is hard to exaggerate; it will almost certainly become one of those capabilities that no one will be able to remember how we lived without in a few years' time.

MapReduce is not standing still while all this is going on, and in fact, the changes being introduced may not have as much immediate impact but are actually much more fundamental.

These changes were initially developed under the name MapReduce 2.0 or MRV2 . However, the name now being used is YARN (Yet Another Resource Negotiator)...