What is data, and what are we doing with it?
A simple answer is that we attempt to place our data as points on paper, graph them, think, and look for simple explanations that approximate the data well. The simple geometric line of F=ma (force being proportional to acceleration) explained a lot of noisy data for hundreds of years. I tend to think of data science as data compression at times.
Sometimes, when a machine is given only win-lose outcomes (of winning games of checkers, for example) and trained, I think of artificial intelligence. It is never taught explicit directions on how to play to win in such a case.
This chapter deals with the pre-processing of data in scikit-learn. Some questions you can ask about your dataset are as follows:
- Are there missing values in your dataset?
- Are there outliers (points far away from the others) in your set?
- What are the variables...