Other important challenges
There are a number of important areas of your engineering effort that cut orthogonally across the capabilities required by your architecture, as mentioned in Chapter 5 when the conceptual architecture was elaborated upon:
- Business realities
- Security and privacy
- Knowledge engineering
- Artificial intelligence
Business competition, intellectual property rights, and fair use of data are very real drags on data engineering efforts. It would be great if all data were free and open source, but that is not the case. The monetization of low-value data into high-value insights will require lots of your own data as well as other data. Some of that data has to be bought, integrated, and metered for fair use. That data, and its derivative data, needs to be secured and remain subject to privacy constraints that are auditable and maintained in perpetuity.
Knowledge is the outcome of information processing of raw data, and artificial intelligence...
