Exploring the contents of a data file
Before writing code to process a dataset, you first need to know some information about the contents of the dataset. This is slightly different from exploratory data analysis, in which the goal is to draw insight from the data.
The details of your initial exploration generally depend on what you already know about a particular dataset and what you ultimately intend to do with the data. That being said, there are a few questions that are usually helpful to ask:
- How is the data structured?
- If the dataset is tabular, the answer to this question is rather simple. However, for a hierarchical dataset, there may be a relatively loose structure of the data.
- What are the data variables?
- For each available variable, what is the data type and what is the range of possible values?
- Are there any errors, missing values, or outliers in the data that can be corrected?
It is not always necessary to do this exploration programmatically. However, often files are too big or messy...
