Twitter, Facebook, Amazon, Verizon, Macy's, and Whole Foods are all companies that run their business using data analytics and base many of the decisions on the analytics. Think about what kind of data they are collecting, how much data they might be collecting, and then how they might be using the data.

Let's look at the grocery store example seen earlier; what if the store starts expanding its business to set up hundreds of stores? Naturally, the sales transactions will have to be collected and stored at a scale hundreds of times more than the single store. But then, no business works independently any more. There is a lot of information out there, starting from local news, tweets, Yelp reviews, customer complaints, survey activities, competition from other stores, the changing demographics or economy of the local area, and so on. All such...

We are surrounded by devices such as the smart refrigerator, smart watch, phone, tablet, laptops, kiosks at the airport, ATMs dispensing cash to you, and many many more, with the help of which we are now able to do things that were unimaginable just a few years ago. We are so used to applications such as Instagram, Snapchat, Gmail, Facebook, Twitter, and Pinterest that it is next to impossible to go a day without access to such applications. Today, cloud computing has introduced us to the following concepts:

• Infrastructure as a Service
• Platform as a Service
• Software as a Service

Behind the scenes is the world of highly scalable distributed computing, which makes it possible to store and process Petabytes (PB) of data:

• 1 EB = 1024 PB (50 million Blu-ray movies)
• 1 PB = 1024 TB (50,000 Blu-ray movies)
• 1 TB = 1024 GB (50 Blu-ray movies...

MapReduce is a framework used to compute a large amount of data in a Hadoop cluster. MapReduce uses YARN to schedule the mappers and reducers as tasks, using the containers. 

An example of a MapReduce job to count frequencies of words is shown in the following diagram:

MapReduce works closely with YARN to plan the job and the various tasks in the job, requests computing resources from the cluster manager (resource manager), schedules the execution of the tasks on the compute resources of the cluster, and then executes the plan of execution. Using MapReduce, you can read write many different types of files of varying formats and perform very complex computations in a distributed manner. We will see more of this in the next chapter on MapReduce frameworks.

Hive provides a SQL layer abstraction over the MapReduce framework with several optimizations. This is needed because of the complexity of writing code using the MapReduce framework. For example, a simple count of the records in a specific file takes at least a few dozen lines of code, which is not productive to anyone. Hive abstracts the MapReduce code by encapsulating the logic from the SQL statement into a MapReduce framework code, which is automatically generated and executed on the backend. This saves incredible amounts of time for anyone who needs to spend more time on doing something useful with the data, rather than going through the boiler plate coding for every single task that needs to be executed and every single computation that's desired as part of your job:

Hive is not designed for online transaction processing and does not offer real-time queries and...

Apache Spark is a unified distributed computing engine across different workloads and platforms. Spark can connect to different platforms and process different data workloads using a variety of paradigms such as Spark Streaming, Spark ML, Spark SQL, and Spark Graphx.

Apache Spark is a fast in-memory data processing engine with elegant and expressive development APIs, which allow data workers to efficiently execute streaming machine learning or SQL workloads that require fast interactive access to data sets. 

Additional libraries built on top of the core allow the workloads for streaming, SQL, graph processing, and machine learning. SparkML, for instance, is designed for data science and its abstraction makes data science easier.

Spark provides real-time streaming, queries, machine learning, and graph processing. Before Apache Spark, we had to use different technologies...

Whichever method we use to perform distributed computing on big data, its very hard to comprehend the meaning of the data without the aid of tools such as Tableau, which can provide an easy to understand visualization of data.

We can do visualization using many tools such as Cognos, Tableau, Zoom data, KineticaDB, Python Matplotlib, R + Shiny, JavaScript, and so on. We will cover visualization in more detail in Chapter 10, Visualizing Big Data.

The following is a simple horizontal bar chart in Tableau:

The following is a geospatial view of data in Tableau: