Data engineering patterns for IT operability
At this juncture, you will want to know, “How do I apply all this? I want to get started, and I don’t want to reinvent the wheel and I don’t want to end up driving into a pothole, or worse off a cliff.” I’ve heard all these sayings and suffered a few of these outcomes myself.
What do we mean by data engineering for IT operability? It is being able to right-size data, metadata, and data access service levels. To accomplish this, you must be pragmatic regarding today’s technology, speed, scale, and cost. In Chapter 1, the need for data zones was raised. Bronze, silver, and gold zones were minimally identified with the need for a raw landing zone and a consumption zone for analytics. Each zone should have clear interfaces for ingesting and egressing data and ensuring access patterns are documented and governed. What goes into this interface contract is key to how data underneath is organized.
...
