Search icon Close icon
Search icon CANCEL
Subscription
0
Cart icon
Cart
Close icon
You have no products in your basket yet
Save more on your purchases!
Savings automatically calculated. No voucher code required
Arrow left icon
All Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Newsletters
Free Learning
Arrow right icon
Arrow up icon
GO TO TOP
Modern Big Data Processing with Hadoop

You're reading from  Modern Big Data Processing with Hadoop

Product type Book
Published in Mar 2018
Publisher Packt
ISBN-13 9781787122765
Pages 394 pages
Edition 1st Edition
Languages
Concepts
Authors (3):
V Naresh Kumar V Naresh Kumar
Profile icon V Naresh Kumar
LinkedIn
Naresh has more than a decade of professional experience in designing, implementing and running very large scale Internet Applications in Fortune Top 500 Companies. He is a Full Stack Architect with hands-on experience in domains like E-commerce, Web-hosting, Healthcare, Bigdata & Analytics, Data Streaming, Advertising and Databases. He believes in Opensource and contributes to them actively. He keeps himself up-to-date with emerging technologies starting from Linux Systems Internals to Frontend technologies. He studied in BITS-Pilani, Rajasthan with Dual Degree in Computer Science & Economics.
Read more
See other products by V Naresh Kumar
Manoj R Patil Manoj R Patil
Profile icon Manoj R Patil
LinkedIn
Manoj R Patil is the Chief Architect in Big Data at Compassites Software Solutions Pvt. Ltd. where he overlooks the overall platform architecture related to Big Data solutions, and he also has a hands-on contribution to some assignments. He has been working in the IT industry for the last 15 years. He started as a programmer and, on the way, acquired skills in architecting and designing solutions, managing projects keeping each stakeholder's interest in mind, and deploying and maintaining the solution on a cloud infrastructure. He has been working on the Pentaho-related stack for the last 5 years, providing solutions while working with employers and as a freelancer as well. Manoj has extensive experience in JavaEE, MySQL, various frameworks, and Business Intelligence, and is keen to pursue his interest in predictive analysis. He was also associated with TalentBeat, Inc. and Persistent Systems, and implemented interesting solutions in logistics, data masking, and data-intensive life sciences.
Read more
See other products by Manoj R Patil
Prashant Shindgikar Prashant Shindgikar
Profile icon Prashant Shindgikar
LinkedIn
Prashant Shindgikar is an accomplished big data Architect with over 20 years of experience in data analytics. He specializes in data innovation and resolving data challenges for major retail brands. He is a hands-on architect having an innovative approach to solving data problems. He provides thought leadership and pursues strategies for engagements with the senior executives on innovation in data processing and analytics. He presently works for a large USA-based retail company.
Read more
See other products by Prashant Shindgikar
View More author details
Toc

Table of Contents (12) Chapters close

Preface 1. Enterprise Data Architecture Principles 2. Hadoop Life Cycle Management 3. Hadoop Design Consideration 4. Data Movement Techniques 5. Data Modeling in Hadoop 6. Designing Real-Time Streaming Data Pipelines 7. Large-Scale Data Processing Frameworks 8. Building Enterprise Search Platform 9. Designing Data Visualization Solutions 10. Developing Applications Using the Cloud 11. Production Hadoop Cluster Deployment

Summary

In this chapter, we have learned all the popular data ingestion tools used in production environments. Sqoop is mainly used to import and export data in and out of RDBMS data stores. Apache Flume is used in real-time systems to import data, mainly from files sources. It supports a wide variety of sources and sinks. Apache NiFi is a fairly new tool and getting very popular these days. It also supports GUI-based ETL development. Hortonworks has started supporting this tool since their HDP 2.4 release. Apache Kafka Connect is another popular tool in the market. It is also a part of the Confluent Data Platform. Kafka Connect can ingest entire databases or collect metrics from all your application servers into Kafka topics, making the data available for stream processing with low latency.

Since we so far know how to build Hadoop clusters and how to ingest data in them, we will...

lock icon The rest of the chapter is locked
arrow left Previous Section
Section 7 of 7
Next Section arrow right
Register for a free Packt account to unlock a world of extra content!
A free Packt account unlocks extra newsletters, articles, discounted offers, and much more. Start advancing your knowledge today.
Sign up now
Unlock this book and the full library FREE for 7 days
Get unlimited access to 7000+ expert-authored eBooks and videos courses covering every tech area you can think of
Start free trial
Renews at AU $19.99/month. Cancel anytime}

Authors (3)

Left arrow icon
Profile icon V Naresh Kumar
LinkedIn
Naresh has more than a decade of professional experience in designing, implementing and running very large scale Internet Applications in Fortune Top 500 Companies. He is a Full Stack Architect with hands-on experience in domains like E-commerce, Web-hosting, Healthcare, Bigdata & Analytics, Data Streaming, Advertising and Databases. He believes in Opensource and contributes to them actively. He keeps himself up-to-date with emerging technologies starting from Linux Systems Internals to Frontend technologies. He studied in BITS-Pilani, Rajasthan with Dual Degree in Computer Science & Economics.
Read more
See other products by V Naresh Kumar
Profile icon Manoj R Patil
LinkedIn
Manoj R Patil is the Chief Architect in Big Data at Compassites Software Solutions Pvt. Ltd. where he overlooks the overall platform architecture related to Big Data solutions, and he also has a hands-on contribution to some assignments. He has been working in the IT industry for the last 15 years. He started as a programmer and, on the way, acquired skills in architecting and designing solutions, managing projects keeping each stakeholder's interest in mind, and deploying and maintaining the solution on a cloud infrastructure. He has been working on the Pentaho-related stack for the last 5 years, providing solutions while working with employers and as a freelancer as well. Manoj has extensive experience in JavaEE, MySQL, various frameworks, and Business Intelligence, and is keen to pursue his interest in predictive analysis. He was also associated with TalentBeat, Inc. and Persistent Systems, and implemented interesting solutions in logistics, data masking, and data-intensive life sciences.
Read more
See other products by Manoj R Patil
Profile icon Prashant Shindgikar
LinkedIn
Prashant Shindgikar is an accomplished big data Architect with over 20 years of experience in data analytics. He specializes in data innovation and resolving data challenges for major retail brands. He is a hands-on architect having an innovative approach to solving data problems. He provides thought leadership and pursues strategies for engagements with the senior executives on innovation in data processing and analytics. He presently works for a large USA-based retail company.
Read more
See other products by Prashant Shindgikar
Right arrow icon

Other recommended products

Related to this chapter
Left arrow icon
Apache Kafka 1.0 Cookbook
Apache Kafka 1.0 Cookbook
Read more
Apache Kafka is an open source stream processing platform to handle real-time data feeds. This book is a highly practical guide to help you understand the fundamentals as well as the advanced applications of Apache Kafka as an enterprise messaging service. It begins with configuring the basic Kafka APIs, and then shows you how to set up Kafka clusters and basic Kafka operations. It covers the recently released Kafka version 1.0, the Confluent Platform and Kafka Streams. By the end of this book, you will have all the knowledge you need to take your understanding of Apache Kafka to the next level and tackle any problem you might encounter while working with it.
Read more
Dec 2017 8 hours 20 minutes
Mastering Apache Storm
Mastering Apache Storm
Read more
With real-world examples and clear explanations, this book will ensure you will have a thorough mastery Apache Storm.You’ll get an understanding of deploying Storm on clusters. Introduce yourself to topics such as trident topology, monitoring, Storm Parallelism, scheduler and log processing. Learn how to integrate Storm with other well-known Big Data technologies such as HBase, Redis, Kafka, and Hadoop to realize the full potential of Storm.You will be able to use the knowledge to develop efficient, distributed real-time applications to cater to your business needs.
Read more
Aug 2017 9 hours 28 minutes
Apache Kafka Quick Start Guide
Apache Kafka Quick Start Guide
Read more
Learn how to use Apache Kafka for efficient processing of distributed applications. This book focuses on programming rather than configuration management of Kafka clusters or Dev Ops. Each chapter focuses on a practical aspect and tries to avoid the tedious theoretical sections. By the end of this book, you will be familiar with solving everyday problems in fast data and processing pipelines.
Read more
Dec 2018 6 hours 12 minutes
Hadoop 2.x Administration Cookbook
Hadoop 2.x Administration Cookbook
Read more
A practical and use case driven approach to Hadoop administration with coverage on a vast array of topics including Hadoop cluster installation, performance tuning, cluster planning, security, and much more. This book covers Hadoop from the perspective of running clusters in critical and large environments with complex data and at scale.
Read more
May 2017 11 hours 36 minutes
Apache Hive Essentials
Apache Hive Essentials
Read more
Apache Hive helps you deal with data summarization, queries, and analysis for huge amounts of data. This book will give you a background in big data, and familiarize you with your Hive working environment. Next you will cover advanced topics like performance and security in Hive and how to work efficiently to find solutions to big data problems.
Read more
Jun 2018 7 hours 0 minutes
Data Lake for Enterprises
Data Lake for Enterprises
Read more
The term 'Data Lake' has recently emerged as a prominent term in the big data industry. Data scientists can make use of it in deriving meaningful insights which can be used by businesses to redefine or transform the way they operate. Lambda architecture is also emerging as one of the very eminent patterns in the big data landscape, as it helps to derive useful information from not only the historical data but also correlates real-time data to enable business for taking critical decisions. This book tries to bring these two important aspects into one, namely data lake and lambda architecture.
Read more
May 2017 19 hours 52 minutes
Mastering Hadoop 3
Mastering Hadoop 3
Read more
This is a comprehensive guide to understand advanced concepts of Hadoop ecosystem. You will learn how Hadoop works internally, and build solutions to some of real world use cases. Finally, you will have a solid understanding of how components in the Hadoop ecosystem are effectively integrated to implement a fast and reliable Big Data pipeline
Read more
Feb 2019 18 hours 8 minutes
Apache Hadoop 3 Quick Start Guide
Apache Hadoop 3 Quick Start Guide
Read more
Apache Hadoop is a widely used distributed data platform. It enables large datasets to be efficiently processed instead of using one large computer to store and process the data. This book will get you started with the Hadoop ecosystem, and introduce you to the main technical topics such as MapReduce, YARN and HDFS.
Read more
Oct 2018 7 hours 20 minutes
Building Data Streaming Applications with Apache Kafka
Building Data Streaming Applications with Apache Kafka
Read more
Apache Kafka is a popular distributed streaming platform which acts as a messaging queue or an enterprise messaging system. This book is a comprehensive guide on designing and architecting enterprise-grade streaming applications using Apache Kafka and other Big Data tools. Once you grasp the basics, we will take you through the more advance concepts in Apache Kafka such as capacity planning and security.
Read more
Aug 2017 9 hours 16 minutes
MySQL 8 for Big Data
MySQL 8 for Big Data
Read more
MySQL is one of the most popular relational databases in the world today, and has become a popular choice of tool to handle vast amounts of structured data - that is, structured Big Data. This book will demonstrate how you can dabble with large amounts of data using MySQL 8. It also highlights topics such as integrating MySQL 8 and a Big Data solution like Apache Hadoop using different tools like Apache Sqoop and MySQL Applier. With practical examples and use-cases, you will get a better clarity on how you can leverage the offerings of MySQL 8 to build a robust Big Data solution.
Read more
Oct 2017 9 hours 52 minutes
HBase High Performance Cookbook
HBase High Performance Cookbook
Read more
Read more
Jan 2017 11 hours 40 minutes
Practical Real-time Data Processing and Analytics
Practical Real-time Data Processing and Analytics
Read more
Real-time data processing involves continuous input, processing and output of data, with the condition that the time required for processing is as short as possible. This book covers the majority of the existing and evolving open source technology stack for real-time processing and analytics. You will get to know about all the real-time solution aspects, from the source to the presentation to persistence. Through this practical book, you’ll be equipped with a clear understanding of how to solve challenges on your own.
Read more
Sep 2017 12 hours 0 minutes
Right arrow icon

Personalised recommendations for you

Based on your interests and search pattern
Left arrow icon
Et al.
Et al.
Read more
Ever wonder why speech recognition systems don't understand the Scottish accent, or what would happen if an astronaut only ate mac 'n' cheese, or other spurious reflections you'd have at a bar? We did, then collated those deliberations into absurd research articles with fake figures and methodologies inspired by even more fictionally absurd studies.
Read more
Aug 2023 7 hours 40 minutes
Generative AI with LangChain
Generative AI with LangChain
Read more
This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.
Read more
Dec 2023 12 hours 0 minutes
Generative AI with LangChain
Generative AI with LangChain
Read more
This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.
Read more
Dec 2023 12 hours 0 minutes
Generative AI with LangChain
Generative AI with LangChain
Read more
This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.
Read more
Dec 2023 12 hours 0 minutes
Generative AI with LangChain
Generative AI with LangChain
Read more
This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.
Read more
Dec 2023 12 hours 0 minutes
Mastering Tableau 2023
Mastering Tableau 2023
Read more
This book is a comprehensive resource to mastering your Tableau skills and becoming a BI expert. As you progress, you will learn how to build advanced dashboards and improve your storytelling to derive key business insight, as well as make you well-versed with advanced functionalities of Tableau in the business intelligence domain.
Read more
Aug 2023 22 hours 48 minutes
Building AI Applications with ChatGPT APIs
Building AI Applications with ChatGPT APIs
Read more
This guide covers all ChatGPT API features for effortless creation of robust AI powered apps. With its help, you’ll be able to leverage ChatGPT’s cutting-edge NLP models to take your app development skills to the next level. You’ll also work on ten exciting projects that will give you the practical know-how that you can apply to your existing applications.
Read more
Sep 2023 8 hours 36 minutes
Building AI Applications with ChatGPT APIs
Building AI Applications with ChatGPT APIs
Read more
This guide covers all ChatGPT API features for effortless creation of robust AI powered apps. With its help, you’ll be able to leverage ChatGPT’s cutting-edge NLP models to take your app development skills to the next level. You’ll also work on ten exciting projects that will give you the practical know-how that you can apply to your existing applications.
Read more
Sep 2023 8 hours 36 minutes
Data Engineering with AWS
Data Engineering with AWS
Read more
Embark on a journey to master data engineering pipelines on AWS! Our book offers a hands-on experience of AWS services for ingesting, transforming, and consuming data. Whether you're an absolute beginner or someone with basic data engineering experience, this guide is an indispensable resource.
Read more
Oct 2023 21 hours 12 minutes
Modern Data Architecture on AWS
Modern Data Architecture on AWS
Read more
Every organization wants an agile, performant, and cost-effective data platform that meets all their current and future business needs. Purpose-built AWS analytics services and their features play a big part in building such a modern data platform. This book brings to you all the design and architectural patterns that’ll help you achieve this goal.
Read more
Aug 2023 14 hours 0 minutes
Practical Guide to Applied Conformal Prediction in Python
Practical Guide to Applied Conformal Prediction in Python
Read more
Discover the power of Conformal Prediction with the "Practical Guide to Applied Conformal Prediction in Python." Master the latest techniques to quantify uncertainty in machine learning and computer vision models, and seamlessly apply them to your industry applications.
Read more
Dec 2023 8 hours 0 minutes
TinyML Cookbook
TinyML Cookbook
Read more
With over 70 project-based recipes, the TinyML Cookbook is a practical guide that will help you to get the most out of your microcontrollers. It provides a comprehensive understanding of the theoretical foundations while giving you hands-on experience training ML models for deployment on Arduino Nano 33 BLE Sense, Raspberry Pi Pico, and SparkFun RedBoard Artemis Nano microcontrollers.
Read more
Nov 2023 22 hours 8 minutes
Right arrow icon