Typical process to build visualizations
Let's review the process for creating insights using traditional BI tools like Oracle OBIEE, SAP Business Objects, and IBM Cognos. At a high level, building a BI dashboard involves the following:
- Ingestion framework to collect data from source systems. These systems are typically files and relational databases.
- Standardize, clean, and build facts, dimensions, and aggregates based on key performance indicators requested by business.
- Build BI logical data models; typically star or snowflakes based on various dashboard needs.
- Build reports and dashboards on the web.
- Publish and share results with data analysts and business stakeholders.
The preceding data flow is shown in the following diagram and is primarily built by IT with regular consultation with data stewards and dashboard consumers:
Figure 1.2: Process flow for traditional BI tools
Key issues with traditional BI tools
The traditional BI tools have primarily following issues that organizations are facing:
- BI software is expensive. In a study done by Amazon, a three year Total Cost of Ownership (TCO) is between $150 to $250 per user per month (source: AWS Summit Series 2016, Chicago at https://aws.amazon.com/summits/chicago/).
- It requires a large IT team to acquire data, model data, build reports, publish and repeat the entire process. A typical BI initiative will require at least 6 months before a production rollout of the dashboard (source: AWS Summit Series 2016, Chicago at https://aws.amazon.com/summits/chicago/).
- They do not work well with unstructured, NoSQL, and streaming data sources. The old BI tools often require ETL teams to build aggregate data in relational form to report.
- They do not scale well as data grows, which is required for big data analytics.
- They do not work well with cloud-hosted data sources like Amazon S3, RDS, and other cloud sources.
