Hadoop was designed to store and analyze large volumes of data. The ecosystem of tools for Hadoop analysis is large and complex. Depending on the type of analysis, many different tools can be used. The Apache Foundation set of tools has a number of standard options such as Hive, HBase, and Pig, but other open source and commercial solutions have been developed to meet different analysis requirements using Hadoop's HDFS and MapReduce features. For example, Cloudera's Impala database runs on Hadoop, but it is not part of the Apache Foundation suite of applications.

Understanding which data analysis tools are used in a Hadoop cluster is important for identifying and properly collecting data. Some data analysis tools store data in formatted files and may offer easier methods for data collection. Other tools may read data directly from files stored in HDFS, but the scripts used for the tool may serve as useful information when later analyzing the data. This...

Hadoop has its own file management concepts that come with many different mechanisms for data storage and retrieval. Hadoop is designed to manage large volumes of data distributed across many nodes built with commodity hardware. As such, Hadoop manages the distribution of large volumes of data using techniques designed to divide, compress, and share the data all while dealing with the possibilities of node failures and numerous processes accessing the same data simultaneously. Many of the filesystem concepts in Hadoop are exactly the same as in other systems, such as directory structures. However, other concepts, such as MapFiles and Hadoop Archive Files, are unique to Hadoop. This section covers many of the file management concepts that are unique to Hadoop.

Forensics is based on evidence. For digital investigations, evidence is data. For Hadoop, the evidence is the information stored on disk and in memory. Not all information stored in Hadoop is relevant; it depends on the nature of the investigation. Evidence that is relevant in one investigation may not be relevant in another. This section summarizes the various sources of evidence and the overall ecosystem of Hadoop forensic evidence.

Standard Hadoop processes or system-generated diagnostic information may not be relevant to a forensic investigation. For example, a Hadoop cluster installed without any customizations that only stores and analyses web log data may not require a collection of all files and process data. Instead, a targeted collection of the web log data can be performed without losing evidence. In other investigations, collecting the log and configuration files may be necessary.

Forensic data in Hadoop falls into three categories:

Hadoop can be run from a number of different platforms. Hadoop can be installed and run from a single desktop, from a distributed network of systems, or as a cloud-based service. Investigators should be aware of the differences and versed in the various architectures. Hadoop runs in the same manner on all three setups; however, collecting evidence may require different steps depending on how the data is stored. For instance, a cloud-based Hadoop server may require a different collection because of the lack of physical access to the servers.

This section details how to set up and run Hadoop using a free virtual machine instance (LightHadoop) and a cloud-based service (Amazon Web Services). Both LightHadoop and Amazon Web Services are used in the examples throughout this book. They serve as testbed environments to highlight how Big Data forensics is performed against different setups.