Addressing concurrency and latency requirements with Delta
Analytics queries are of two types:
- Ad-hoc data exploration by analysts as they proceed with data discovery activities. Data scientists and BI analysts have some tolerance for ad-hoc queries, meaning if it takes longer to retrieve the results, it is undesirable but tolerated.
- Known queries for well-defined consumption patterns. There is very little tolerance for known queries. Consumers expect these to be refreshed quickly as the end user may be a business executive or someone outside of the data organization who'll dislike the latency.
We should remember that a dashboard hosts several queries as widgets or sections and there are several consumers of that data. The time it takes to return the results is referred to as the latency of the query, and the maximum number of simultaneous users that it serves at the same point in time is referred to as the concurrency of the query. Latency and concurrency...
