Bringing statistics in to Hive
Statistics in terms of the number of records in a table or partitions or histograms of a column is important. Also, it could help in query optimization. Statistical data is required as an input to many functions so that it can compare different plans. Statistics also help users by storing answers to some of the most frequently queried data and prevent long-running execution plans each time a query is executed. Common examples include unique visitors to the site, top 10 stories read by the visitors, and so on.
How to do it…
There are the following different levels at which statistics can be derived:
Statistics at table level—These statistics can be used to derive the number of rows and files, size, and so on in a table.
Statistics at partition level—These statistics can be used to derive the number of rows and files, size, and so on in all partitions or specified partitions of a table.
Statistics at column level—These statistics can be used to derive the number of...
