Common big data design patterns

Design patterns provide a common vocabulary for data personas to understand and share design and architecture blueprints. So, given a set of requirements, everyone understands the likely design pattern to apply as a plausible solution. Traditional software engineering design patterns are object-oriented and are of three types, categorized under creational, structural, and behavioral patterns. Data engineering is not necessarily object-oriented (OO) related and is better articulated around concepts of data ingestion, transformation, storage, and analytics patterns. In the next few sections, we will look at reusable patterns in each of these areas.

Ingestion

Ingestion refers to all aspects of consolidating data into a target site for further processing and analysis from multiple sources using different languages, different file formats, and different sizes and frequencies. The number of such combinations is large, which is why we see a lot of data...