Hadoop Essentials

Chapter 2. Hadoop Ecosystem

Now that we have discussed and understood big data and Hadoop, we can move on to understanding the Hadoop ecosystem. A Hadoop cluster may have hundreds or thousands of nodes which are difficult to design, configure, and manage manually. Due to this, there arises a need for tools and utilities to manage systems and data easily and effectively. Along with Hadoop, we have separate sub-projects which are contributed by some organizations and contributors, and are managed mostly by Apache. The sub-projects integrate very well with Hadoop and can help us concentrate more on design and development rather than maintenance and monitoring, and can also help in the development and data management.

Before we understand different tools and technologies, let's understand a use case and how it differs from traditional systems.

Filter reviews by

All

Amazon verified reviews

Amazon Customer Jan 13, 2017

This book covers a good number of Hadoop tools and api. A well-composed compilation of the important need to know knowledge in Big Data.

Amazon Verified review

David Jun 08, 2015

Excellent book to introduce Hadoop. I known the name for years without ever finding time to look deeper at it. The book introduces the major concepts behind and the modules available with, whenever available, alternatives. It's clearly a must read for people new to the concept of big data manipulation with framework like Hadoop. Each pieces from distributed filesystem to data parsing is covered by the book.

Francesco Corti Jun 17, 2015

I think this is a nice book for developers and software architect with no relevant experience in big data and Apache Hadoop. Big data is definitely an IT buzzword but this book tries to make order in the today’s scenario, with an practical an interesting look at the Apache Hadoop implementation. I enjoyed reading this book and I think I will fix in my mind some of the initial descriptions of the introductory part that I find very rational and clear. Nice job, Shiva.

PJG May 21, 2015

This is a very detailed introductory guide to Hadoop. The key question is: how does it rate against the millions of other Hadoop books in existence?At 194 pages, this is a slim volume when compared to "Hadoop: The Definitive Guide", but in terms of content it's suprisingly packed. Chapter 3, which covers HDFS, MapReduce, and YARN, is a good case in point: there's quite a lot of depth here, but the number of diagrams really helps clarify what is a fairly whistle-stop tour through fundamental Hadoop building blocks. Later chapters give a detailed overview of Hbase, Sqoop and Flume and these are useful. Spark and Storm are covered, with a brief note on Lambda architectures.Overall, this is quite a nice reference, although the small page count does reflect a rather terse style and I don't think it would be a particularly easy book to learn from for a newcomer to Hadoop. However, sometimes it's useful to have a rapid overview with just the distilled, essential facts and the book certainly achieves that.

Ian Stirk Jun 10, 2015

Hi,I have written a detailed chapter-by-chapter review of this book on www DOT i-programmer DOT info, the first and last parts of this review are given here. For my review of all chapters, search i-programmer DOT info for STIRK together with the book's title.This book aims to give you an understanding of Hadoop and some of its major components, explaining how and when to use them, and providing scenarios where they should be used.It is aimed at application and system developers that want to solve practical problems using the Hadoop framework. It is also intended for Hadoop professionals who want to find solutions to the different challenges they come across in their Hadoop projects.A prerequisite is a good understanding of Java programming, additionally, a basic understanding of distributed computing would be helpful.Below is a chapter-by-chapter exploration of the topics covered.Chapter 1 Introduction to Big Data and HadoopThe chapter opens with the emergence of big data systems as a response to the limitations of relational databases (RDBMS), which were unable to process big data in a timely and cost-effective manner.The next section looks at explaining the need for big data systems with reference to the 3 Vs of big data:*Volume (1.8 zettabytes of data created in 2011, 35 zettabytes expected by 2020)*Velocity (data arriving quickly)*Variety (structured and semi-structured data e.g. emails)The chapter continues with a look at the sources of big data, including: monitoring sensors, social media posts, videos/photos, logs etc. Some big data use case patterns are briefly described.Next, Hadoop is examined, being the most popular big data platform. Hadoop is open source, and offers large-scale massively parallel distributed processing. Hadoop has 2 major components: HDFS (Hadoop Distributed File System) - Hadoop’s storage system, and MapReduce – Hadoop’s batch processing model. The section continues with a look at Hadoop’s history, advantages, uses, and related components.The remainder of the chapter provides an overview of the other chapters of the book, namely:*Pillars of Hadoop (HDFS, MapReduce, YARN)*Data access components (Hive, Pig)*Data storage component (HBase)*Data ingestion in Hadoop (Sqoop, Flume)*Streaming and real-time analysis (Storm, Spark)This chapter provides a useful understanding of how big data processing arose, and how Hadoop fulfils this need. There’s a useful overview of the four main types of NoSQL database. There’s a helpful overview of Hadoop, its history, advantages, uses, and associated components. There are plenty of helpful diagrams to aid understanding (as there are in the rest of the book), and a useful introduction to what’s coming in the rest of the book.Sometimes, the English grammar is substandard; this occurs in various sections of the book. Some subsections seem disjointed (okay within themselves, but not part of a wider coherent section) – again this occurs in other parts of the book. There’s a small error relating to the amount of total data created in 2009, the value given is 800GB, the correct value is 800 exabytes or 0.8 zettabytes. All these problems should have been caught by the reviewers/editors....ConclusionThis book aims to give you an understanding of Hadoop and some of its major components, and largely succeeds. For a short book, it covers a wide area. I think only a little understanding of Java (or a comparable language) is needed to read this book. The extensive use of diagrams is helpful.The book should prove useful to developers wanting to know more about Hadoop and its major associated technologies. The book provides a helpful overview of Hadoop, HDFS, MapReduce, YARN, Hive, Pig, HBase, Sqoop, Flume, Storm and Spark. While not comprehensive (e.g. Impala and Hue are not discussed), it does cover many of the popular components.The English grammar in some sections is substandard, making the book awkward to read. An editor with a good understanding of English would improve the book’s readability. Some sentences are illogical e.g. “Hadoop is primarily designed for batch processing and for Lambda Architecture systems.” – But, Lambda Architecture includes batch and stream processing! Additionally, some sections seem muddled – probably amplified by the bad grammar and illogical thought.Overall, if you can bypass the problems, this is a useful book, wide in scope and quite detailed for a short book.

Hadoop Essentials: Delve into the key concepts of Hadoop and get a thorough understanding of the Hadoop ecosystem

What do you get with Print?

Hadoop Essentials

Chapter 2. Hadoop Ecosystem

Traditional systems

Database trend

The Hadoop use cases

Hadoop's basic data flow

Hadoop integration

The Hadoop ecosystem

Traditional systems

Database trend

The Hadoop use cases

Hadoop's basic data flow

Hadoop integration

The Hadoop ecosystem

Distributed filesystem

HDFS

Distributed programming

NoSQL databases

Apache HBase

Data ingestion

Page 1 of 15

Key benefits

Description

Who is this book for?

What you will learn

Product Details

What do you get with Print?

Product Details

Frequently bought together

Table of Contents

Recommendations for you

Customer reviews

Filter reviews by

People who bought this also bought

About the author

FAQs

Hadoop Essentials: Delve into the key concepts of Hadoop and get a thorough understanding of the Hadoop ecosystem

What do you get with Print?

Contact Details

Shipping Address

Billing Address

Database trend

Key benefits

Description

Who is this book for?

What you will learn

Product Details

What do you get with Print?

Contact Details

Shipping Address

Billing Address

Product Details

Packt Subscriptions

Frequently bought together

Table of Contents

Recommendations for you

Customer reviews

Filter reviews by

People who bought this also bought

About the author

FAQs