Making use of sparse data
As datasets increase in dimension, some attributes are likely to be sparse, which means most observations do not share values of the attribute. This is a natural consequence of the curse of dimensionality in which this ever-increasing detail turns observations into outliers identified by their unique combination of attributes. It is very uncommon for sparse data to have any specific value, or perhaps even any value at all—as was the case in the sparse matrices for text data found in Chapter 4, Probabilistic Learning – Classification Using Naive Bayes, and the sparse matrices for shopping cart data in Chapter 8, Finding Patterns – Market Basket Analysis Using Association Rules.
This is not the same as missing data, where typically a relatively small portion of values are unknown. In sparse data, most values are known, but the number of interesting, meaningful values is dwarfed by an overwhelming number of values that add little value...
