PixieDust (https://github.com/ibm-watson-data-lab/pixiedust) is an open-source project composed primarily of three components designed to address the three goals stated at the beginning of this chapter:

Loading data into a Notebook is one of the most repetitive tasks a data scientist can do, yet depending on the framework or data source being used, writing the code can be difficult and time-consuming.

Let's take a concrete example of trying to load a CSV file from an open data site (say https://data.cityofnewyork.us) into both a pandas and Apache Spark DataFrame.

For pandas, the code is pretty straightforward as it provides an API to directly load from URL:

import pandas
data_url = "https://data.cityofnewyork.us/api/views/e98g-f8hy/rows.csv?accessType=DOWNLOAD"
building_df = pandas.read_csv(data_url)
building_df

The last statement, calling building_df, will print its contents in the output cell. This is possible without a print because Jupyter is interpreting the last statement of a cell calling a variable as a directive to print it...

Working in a controlled experiment is, most of the time, not the same as working in the real world. By this I mean that, during development, we usually pick (or I should say manufacture) a sample dataset that is designed to behave; it has the right format, it complies with the schema specification, no data is missing, and so on. The goal is to focus on verifying the hypotheses and build the algorithms, and not so much on data cleansing, which can be very painful and time-consuming. However, there is an undeniable benefit to get data that is as close to the real thing as early as possible in the development process. To help with this task, I worked with two IBM colleagues, Jamie Jennings and Terry Antony, who volunteered to build an extension to PixieDust called pixiedust_rosie.

This Python package implements a simple wrangle_data() method to automate the cleansing of raw data. The pixiedust_rosie package currently supports CSV and JSON, but more formats...

Data visualization is another very important data science task that is indispensable for exploring and forming hypotheses. Fortunately, the Python ecosystem has a lot of powerful libraries dedicated to data visualization, such as these popular examples:

However, similar to data loading and cleaning, using these libraries in a Notebook can be difficult and time-consuming. Each of these libraries come with their own programming model and APIs are not always easy to learn and use, especially if you are not an experienced developer. Another issue is that these libraries do not have a high-level interface to commonly used data processing frameworks such as pandas (except maybe Matplotlib) or Apache Spark and, as a result, a lot of data preparation is needed before plotting the data.

To help with...

To better explore data, PixieDust also provides a built-in, simple graphical interface that lets you quickly filter the data being visualized. You can quickly invoke the filter by clicking on the filter toggle button in the top-level menu. To keep things simple, the filter only supports building predicates based on one column only, which is sufficient in most cases to validate simple hypotheses (based on feedback, this feature may be enhanced in the future to support multiple predicates). The filter UI will automatically let you select the column to filter on and, based on its type, will show different options:

Solving hard data problems is only part of the mission given to data science teams. They also need to make sure that data science results get properly operationalized to deliver business value to the organization. Operationalizing data analytics is very much use case - dependent. It could mean, for example, creating a dashboard that synthesizes insights for decision makers or integrating a machine learning model, such as a recommendation engine, into a web application.

In most cases, this is where data science meets software engineering (or as some would say, where the rubber meets the road). Sustained collaboration between the teams—instead of a one-time handoff—is key to a successful completion of the task. More often than not, they also have to grapple with different languages and platforms, leading to significant code rewrites by the software engineering team.

We experienced it firsthand in our Sentiment...

In the previous section, we saw how PixieApps combined with the PixieDust display framework offer an easy way to build powerful dashboards that connect directly with your data analytics, allowing for rapid iterations between the algorithms and the user interface. This is great for rapid prototyping, but Notebooks are not suitable to be used in a production environment where the target persona is the line of business user. One obvious solution would be to rewrite the PixieApp using a traditional three tiers web application architecture, for example, as follows:

However, this would provide only a marginal improvement over the existing process, which would consist only, in this case, of the ability to do iterative implementation with the...