Working with datasets

Datasets are the backbone of any data science project. With a good, well-structured dataset, we have the opportunity to explore, ideate, and discover important insights from the data. The terms good and well-structured are key. In the real world, this rarely happens by accident. I am the lead developer on a project that does data science every day. We ingest diagnostic, utilization, and performance data from various hardware platforms such as storage arrays, switches, virtualization nodes (such as VMware), backup devices, and more. We collect it for the entire enterprise; every device in every data center. Our software then turns that raw data into visualizations that provide insights, allowing organizations to effectively manage their IT estate through consolidating health monitoring, utilization and performance reporting, and capacity planning.

I’ve been at it for 10 years now and we’re always looking to support new devices and systems. Our...