Hadoop is changing the way people think about data. We need to know what led to the origin of this magical innovation. Who developed Hadoop and why? What problems existed before Hadoop? How has it solved these problems? What challenges were encountered during development? How has Hadoop transformed from version 1 to version 3? Let's walk through the origins of Hadoop and its journey to version 3.

In 1997, Doug Cutting, a co-founder of Hadoop, started working on project Lucene, which is a full-text search library. It was completely written in Java and is a full-text search engine. It analyzes text and builds an index on it. An index is just a mapping of text to locations, so it...

The first alpha release of Hadoop version 3.0.0 was on 30 August 2016. It was called version 3.0.0-alpha1. This was the first alpha release in a series of planned alphas and betas that ultimately led to 3.0.0 GA. The intention behind this alpha release was to quickly gather and act on feedback from downstream users.

With any such releases, there are some key drivers that lead to its birth. These key drivers create benefits that will ultimately help in the better functioning of Hadoop-augmented enterprise applications. Before we discuss the features of Hadoop 3, you should understand these driving factors. Some driving factors behind the release of Hadoop 3 are as follows:

• A lot of bug fixes and performance improvements: Hadoop has a growing open source community of developers regularly adding major/minor changes or improvements...

The Hadoop Logical view can be divided into multiple sections. These sections can be viewed as a logical sequence, with steps starting from Ingress/Egress and ending at Data Storage Medium.

The following diagram shows the Hadoop platform logical view:

We will touch upon these sections as shown in the preceding diagram one by one, to understand them. However, when designing any Hadoop application, you should think in terms of those sections and make technological choices according to the use case problems you are trying to solve. Without wasting time, let's look at these sections one by one:

• Ingress/egress/processing: Any interaction with the Hadoop platform should be viewed in terms of the following:
  • Ingesting (ingress) data 
  • Reading (Egress) data 
  • Processing already ingested data

These actions can be automated via the use of...

Hadoop is an open-source project under the Apache Software Foundation, and most components in the Hadoop ecosystem are also open-sourced. Many companies have taken important components and bundled them together to form a complete distribution package that is easier to use and manage. A Hadoop distribution offers the following benefits:

• Installation: The distribution package provides an easy way to install any component or rpm-like package on clusters. It provides an easy interface too.
• Packaging: It comes with multiple open-source tools that are well configured to work together. Assume that you want to install and configure each component separately on a multi-node cluster and then test whether it's working properly or not. What if we forget some testing scenarios and the cluster behaves unexpectedly? The Hadoop distribution assures us that...

We provided a basic introduction to Hadoop and the following are a few points to remember:

• Doug Cutting, the founder of Hadoop, started the development of Hadoop at Nutch based on a Google research paper on Google File System and MapReduce.
• Apache Lucene is a full-text open-source search library initially written by Doug Cutting in Java.
• Hadoop consists of two important parts, one called the Hadoop Distributed File System and the other called MapReduce.
• YARN is a resource management framework used to schedule and run applications such as MapReduce and Spark.
• Hadoop distributions are a complete package of all open source big data tools integrated together to work with each other in an efficient way.

In this chapter, we covered Hadoop's origins and how Hadoop evolved over time with more performance-optimized features and tools. We also covered a logical view of the Hadoop platform in detail and understood its different layers. Hadoop distribution was also covered, to help you understand which distribution you should choose. We described the new features available in Hadoop version 3 and will discuss these in more detail in upcoming chapters.

In the next chapter, we will cover HDFS and will walk you through the HDFS architecture and its component in detail. We will go much deeper into the internals of HDFS and HDFS high availability features. We will then look into HDFS read-write operations and how the HDFS caching and federation service works.