Summary
Hopefully, this chapter presented the topic of data life cycle management as something other than a dry abstract concept. We covered a lot, particularly:
The definition of data life cycle management and how it covers a number of issues and techniques that usually become important with large data volumes
The concept of building a data ingest pipeline along good data life cycle management principles that can then be utilized by higher-level analytic tools
Oozie as a Hadoop-focused workflow manager and how we can use it to compose a series of actions into a unified workflow
Various Oozie tools, such as subworkflows, parallel action execution, and global variables, that allow us to apply true design principles to our workflows
HCatalog and how it provides the means for tools other than Hive to read and write table-structured data; we showed its great promise and integration with tools such as Pig but also highlighted some current weaknesses
Avro as our tool of choice to handle schema evolution...
