IBM Watson DeepQA
One project that exemplifies the idea that data science is a team sport is the IBM DeepQA research project which originated as an IBM grand challenge to build an artificial intelligence system capable of answering natural language questions against predetermined domain knowledge. The Question Answering (QA) system should be good enough to be able to compete with human contestants at the Jeopardy! popular television game show.
As is widely known, this system dubbed IBM Watson went on to win the competition in 2011 against two of the most seasoned Jeopardy! champions: Ken Jennings and Brad Rutter. The following photo was taken from the actual game that aired on February 2011:
IBM Watson battling Ken Jennings and Brad Rutter at Jeopardy!
Source: https://upload.wikimedia.org/wikipedia/e
It was during the time that I was interacting with the research team that built the IBM Watson QA computer system that I got to take a closer look at the DeepQA project architecture and realized first-hand how many data science fields were actually put to use.
The following diagram depicts a high-level architecture of the DeepQA data pipeline:
Watson DeepQA architecture diagram
Source: https://researcher.watson.ibm.com/researcher/files/us-mi
As the preceding diagram shows, the data pipeline for answering a question is composed of the following high-level steps:
- Question & Topic Analysis (natural language processing): This step uses a deep parsing component which detects dependency and hierarchy between the words that compose the question. The goal is to have a deeper understanding of the question and extracts fundamental properties, such as the following:
- Focus: What is the question about?
- Lexical Answer Type (LAT): What is the type of the expected answer, for example, a person, a place, and so on. This information is very important during the scoring of candidate answers as it provides an early filter for answers that don't match the LAT.
- Named-entity resolution: This resolves an entity into a standardized name, for example, "Big Apple" to "New York".
- Anaphora resolution: This links pronouns to previous terms in the question, for example, in the sentence "On Sept. 1, 1715 Louis XIV died in this city, site of a fabulous palace he built," the pronoun "he" refers to Louis XIV.
- Relations detection: This detects relations within the question, for example, "She divorced Joe DiMaggio in 1954" where the relation is "Joe DiMaggio Married X." These type of relations (Subject->Predicate->Object) can be used to query triple stores and yield high-quality candidate answers.
- Question class: This maps the question to one of the predefined types used in Jeopardy!, for example, factoid, multiple-choice, puzzle, and so on.
- Primary search and Hypothesis Generation (information retrieval): This step relies heavily on the results of the question analysis step to assemble a set of queries adapted to the different answer sources available. Some example of answer sources include a variety of full-text search engines, such as Indri (https://www.lemurproject.org/indri.php) and Apache Lucene/Solr (http://lucene.apache.org/solr), document-oriented and title-oriented search (Wikipedia), triple stores, and so on. The search results are then used to generate candidate answers. For example, title-oriented results will be directly used as candidates while document searches will require more detailed analysis of the passages (again using NLP techniques) to extract possible candidate answers.
- Hypothesis and Evidence scoring (NLP and information retrieval): For each candidate answer, another round of search is performed to find additional supporting evidence using different scoring techniques. This step also acts as a prescreening test where some of the candidate answers are eliminated, such as the answers that do not match the LAT computed from step 1. The output of this step is a set of machine learning features corresponding to the supporting evidence found. These features will be used as input to a set of machine learning models for scoring the candidate answers.
- Final merging and scoring (machine learning): During this final step, the system identifies variants of the same answer and merges them together. It also uses machine learning models to select the best answers ranked by their respective scores, using the features generated in step 3. These machine learning models have been trained on a set of representative questions with the correct answers against a corpus of documents that has been pre-ingested.
As we continue the discussion on how data science and AI are changing the field of computer science, I thought it was important to look at the state of the art. IBM Watson is one of these flagship projects that has paved the way to more advances we've seen since it beats Ken Jennings and Brad Rutter at the game of Jeopardy!.
