CRISP-DM overview
The CRISP-DM is considered to be the de facto standard for conducting a data mining project. Starting with the Business Understanding phase and ending with the Deployment phase, this six-phase process has a total of 24 tasks. It is important to not get by with just focusing on the highest level of the phases, since it is well worth the effort to familiarize yourself with all of the 24 tasks. The diagram shown next illustrates the six phases of the CRISP-DM process model and the following pages will discuss each of these phases:
Business Understanding
The Business Understanding phase is focused on good problem definition and ensuring that you are solving the business's problem. You must begin from a business perspective and business knowledge, and proceed by converting this knowledge into a data mining problem definition. You will not be performing the actual Business Understanding in Modeler, as such, but Modeler allows you to organize supporting material such as word documents and PowerPoint presentations as part of a Modeler project file. You don't need to organize this material in a project file, but you do need to remember to do a proper job at this phase. For more detailed information on each task within a phase, refer to the CRISP-DM document itself. It is free and readily available on the internet.
The four tasks in this phase are:
- Determine business objectives
- Assess situation
- Determine data mining goals
- Produce project plan
Data Understanding
Modeler has numerous resources for exploring your data in preparation for the other phases. We will demonstrate a number of these in Chapter 3, Importing Data into Modeler; Chapter 4, Data Quality and Exploration; and Chapter 8, Looking for Relationships Between Fields. The Data Understanding phase includes activities for getting familiar with the data as well as data collection and data quality. The four Data Understanding tasks are:
- Collect initial data
- Describe data
- Explore data
- Verify data quality
Data Preparation
The Data Preparation phase covers all activities to construct the final dataset (the data that will be fed into the modeling tool(s)) from the initial raw data. Data Preparation is often described as the most labor-intensive phase for the data analyst. It is terribly important that Data Preparation is done well, and a substantial amount of this book is dedicated to it. We cover cleaning, selecting, integrating, and constructing data, in Chapter 5, Cleaning and Selecting Data; Chapter 6,Combining Data Files; and Chapter 7, Deriving New Fields, respectively. However, a book dedicated to the basics of data mining can really only start you on your journey when it comes to Data Preparation, since there are so many ways in which you can improve and prepare data. When you are ready for a more advanced treatment of this topic, there are two resources that will go into Data Preparation in much more depth, and both have extensive Modeler software examples: The IBM SPSS Modeler Cookbook (Packt Publishing) and Effective Data Preparation (Cambridge University Press).
The five Data Preparation tasks are:
- Select data
- Clean data
- Construct data
- Integrate data
- Format data
Modeling
The Modeling phase is probably what you expect it to be—the phase where the modeling algorithms move to the forefront. In many ways, this is the easiest phase, as the algorithms do a lot of the work if you have done an excellent job on the prior phases and you've done a good job translating the business problem into a data mining problem. Despite the fact that the algorithms are doing the heavy lifting in this phase, it is generally considered the most intimidating; it is understandable why. There are an overwhelming number of algorithms to choose from. Even in a well-curated workbench such as Modeler, there are dozens of choices. Open source options such as R have hundreds of choices. While this book is not an algorithms guide, and even though it is impossible to offer a chapter on each algorithm, Chapter 9, Introduction to Modeling Options in IBM SPSS Modeler should be very helpful in understanding, at a high level, what options are available in Modeler. Also, in Chapter 10, Decision Tree Models we go through a thorough demonstration of one modeling technique, decision trees, to orient you to modeling in Modeler.
The four tasks in this phase are:
- Select modeling technique
- Generate test design
- Build model
- Assess model
Evaluation
At this stage in the project you have built a model (or models) that appears to be of high quality, from a data analysis perspective. Before proceeding to final deployment of the model, it is important to more thoroughly evaluate the model—to be certain it properly achieves the business objectives.
Evaluation is frequently confused with model assessment—the last task of the Modeling phase. Assess model is all about the data analysis perspective and includes metrics such as model accuracy. The authors of CRISP-DM considered calling this phase business evaluation because it has to be conducted in the language of the business and using the metrics of the business as indicators of success. Given the nature of this book, and its emphasis on the point and click operation of Modeler, there will be virtually no opportunity to practice this phase, but in real world projects it is a critical phase.
The three tasks in this phase are:
- Evaluate results
- Review process
- Determine next steps
Deployment
Creation of the model is generally not the end of the project. Depending on the requirements, the Deployment phase can be as simple as generating a report or as complex as implementing a repeatable data mining process. Given the software focus of this book and the spirit of sticking to the basics, we will really only cover using models for the scoring of new data. Real world deployment is much more complex and a complex deployment can more than double the length of a project. Modeler's capabilities in this area go far beyond what we will be able to show in this book. The final chapter of this book, Chapter 11, Model Assessment and Scoring, briefly talks about some of these issues.
However, it is not unusual for the deployment team to be different than the modeling team, and the responsibility may fall to team members with more of an IT focus. The IBM software stack offers dedicated tools for complex deployment scenarios. IBM Collaboration and Deployment Services has such advanced features.
The four tasks in the Deployment phase are:
- Plan deployment
- Plan monitoring and maintenance
- Produce final report
- Review project
Learning more about CRISP-DM
Here are five great resources to learn more about CRISP-DM:
- The CRISP-DM document itself, found in various forms: https://en.wikipedia.org/wiki/Cross_Industry_Standard_Process_for_Data_Mining
- Tom Khabaza's Nine Laws of Data Mining: http://khabaza.codimension.net/index_files/9laws.htm
- The IBM SPSS Modeler Cookbook (Packt Publishing), in particular there is a more extensive introduction and an appendix with essays on the
Business Understandingphase - Data Mining For Dummies: There is no software instruction in this book, but it has an excellent chapter on CRISP-DM and also a chapter on the nine laws
- https://www.lynda.com/SPSS-tutorials/Essential-Elements-Predictive-Analytics-Data-Mining: This web-based course does not cover software tasks but discusses predictive analytics strategy, CRISP-DM, and the Nine Laws of Data Mining
