By Topic

IBM Journal of Research and Development

Issue 3.4 • Date May-June 2012

This is Watson

In 2007, IBM Research took on the grand challenge of building a computer system that could compete with champions at the game of Jeopardy!. In 2011, the open-domain question-answering system dubbed Watson beat the two highest ranked players in a nationally televised two-game Jeopardy! match. This special issue provides a deep technical overview of the ideas and accomplishments that positioned our team to take on the Jeopardy! challenge, build Watson, and ultimately triumph. It describes the nature of the question-answering challenge represented by Jeopardy! and details our technical approach. The papers herein describe and provide experimental results for many of the algorithmic techniques developed as part of the Watson system, covering areas including computational linguistics, information retrieval, knowledge representation and reasoning, and machine leaning. The papers offer component-level evaluations as well as their end-to-end contribution to Watson's overall question-answering performance.

Filter Results

Displaying Results 1 - 19 of 19
  • Front Cover

    Publication Year: 2012, Page(s): C1
    IEEE is not the copyright holder of this material | PDF file iconPDF (1554 KB)
    Freely Available from IEEE
  • Table of Contents

    Publication Year: 2012, Page(s):1 - 2
    IEEE is not the copyright holder of this material | PDF file iconPDF (49 KB)
    Freely Available from IEEE
  • Introduction to “This is Watson”

    Publication Year: 2012, Page(s):1:1 - 1:15
    Cited by:  Papers (4)  |  Patents (1)
    IEEE is not the copyright holder of this material | Click to expandAbstract | PDF file iconPDF (3008 KB) | HTML iconHTML

    In 2007, IBM Research took on the grand challenge of building a computer system that could compete with champions at the game of Jeopardy!™. In 2011, the open-domain question-answering (QA) system, dubbed Watson, beat the two highest ranked players in a nationally televised two-game Jeopardy! match. This paper provides a brief history of the events and ideas that positioned our team to take... View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Question analysis: How Watson reads a clue

    Publication Year: 2012, Page(s):2:1 - 2:14
    Cited by:  Papers (1)  |  Patents (1)
    IEEE is not the copyright holder of this material | Click to expandAbstract | PDF file iconPDF (914 KB) | HTML iconHTML

    The first stage of processing in the IBM Watson™ system is to perform a detailed analysis of the question in order to determine what it is asking for and how best to approach answering it. Question analysis uses Watson's parsing and semantic analysis capabilities: a deep Slot Grammar parser, a named entity recognizer, a co-reference resolution component, and a relation extraction component.... View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Deep parsing in Watson

    Publication Year: 2012, Page(s):3:1 - 3:15
    Cited by:  Papers (1)  |  Patents (2)
    IEEE is not the copyright holder of this material | Click to expandAbstract | PDF file iconPDF (631 KB) | HTML iconHTML

    Two deep parsing components, an English Slot Grammar (ESG) parser and a predicate-argument structure (PAS) builder, provide core linguistic analyses of both the questions and the text content used by IBM Watson™ to find and hypothesize answers. Specifically, these components are fundamental in question analysis, candidate generation, and analysis of passage evidence. As part of the Watson p... View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Textual resource acquisition and engineering

    Publication Year: 2012, Page(s):4:1 - 4:11
    Cited by:  Papers (2)
    IEEE is not the copyright holder of this material | Click to expandAbstract | PDF file iconPDF (1275 KB) | HTML iconHTML

    A key requirement for high-performing question-answering (QA) systems is access to high-quality reference corpora from which answers to questions can be hypothesized and evaluated. However, the topic of source acquisition and engineering has received very little attention so far. This is because most existing systems were developed under organized evaluation efforts that included reference corpora... View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Automatic knowledge extraction from documents

    Publication Year: 2012, Page(s):5:1 - 5:10
    Cited by:  Papers (4)  |  Patents (1)
    IEEE is not the copyright holder of this material | Click to expandAbstract | PDF file iconPDF (839 KB) | HTML iconHTML

    Access to a large amount of knowledge is critical for success at answering open-domain questions for DeepQA systems such as IBM Watson™. Formal representation of knowledge has the advantage of being easy to reason with, but acquisition of structured knowledge in open domains from unstructured data is often difficult and expensive. Our central hypothesis is that shallow syntactic knowledge a... View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Finding needles in the haystack: Search and candidate generation

    Publication Year: 2012, Page(s):6:1 - 6:12
    IEEE is not the copyright holder of this material | Click to expandAbstract | PDF file iconPDF (2104 KB) | HTML iconHTML

    A key phase in the DeepQA architecture is Hypothesis Generation, in which candidate system responses are generated for downstream scoring and ranking. In the IBM Watson™ system, these hypotheses are potential answers to Jeopardy!™ questions and are generated by two components: search and candidate generation. The search component retrieves content relevant to a given question from Wa... View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Typing candidate answers using type coercion

    Publication Year: 2012, Page(s):7:1 - 7:13
    Cited by:  Patents (1)
    IEEE is not the copyright holder of this material | Click to expandAbstract | PDF file iconPDF (2267 KB) | HTML iconHTML

    Many questions explicitly indicate the type of answer required. One popular approach to answering those questions is to develop recognizers to identify instances of common answer types (e.g., countries, animals, and food) and consider only answers on those lists. Such a strategy is poorly suited to answering questions from the Jeopardy!™ television quiz show. Jeopardy! questions have an ext... View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Textual evidence gathering and analysis

    Publication Year: 2012, Page(s):8:1 - 8:14
    IEEE is not the copyright holder of this material | Click to expandAbstract | PDF file iconPDF (208 KB) | HTML iconHTML

    One useful source of evidence for evaluating a candidate answer to a question is a passage that contains the candidate answer and is relevant to the question. In the DeepQA pipeline, we retrieve passages using a novel technique that we call Supporting Evidence Retrieval, in which we perform separate search queries for each candidate answer, in parallel, and include the candidate answer as part of ... View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Relation extraction and scoring in DeepQA

    Publication Year: 2012, Page(s):9:1 - 9:12
    Cited by:  Papers (1)
    IEEE is not the copyright holder of this material | Click to expandAbstract | PDF file iconPDF (1439 KB) | HTML iconHTML

    Detecting semantic relations in text is an active problem area in natural-language processing and information retrieval. For question answering, there are many advantages of detecting relations in the question text because it allows background relational knowledge to be used to generate potential answers or find additional evidence to score supporting passages. This paper presents two approaches t... View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Structured data and inference in DeepQA

    Publication Year: 2012, Page(s):10:1 - 10:14
    IEEE is not the copyright holder of this material | Click to expandAbstract | PDF file iconPDF (752 KB) | HTML iconHTML

    Although the majority of evidence analysis in DeepQA is focused on unstructured information (e.g., natural-language documents), several components in the DeepQA system use structured data (e.g., databases, knowledge bases, and ontologies) to generate potential candidate answers or find additional evidence. Structured data analytics are a natural complement to unstructured methods in that they typi... View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Special Questions and techniques

    Publication Year: 2012, Page(s):11:1 - 11:13
    IEEE is not the copyright holder of this material | Click to expandAbstract | PDF file iconPDF (407 KB) | HTML iconHTML

    Jeopardy!™ questions represent a wide variety of question types. The vast majority are Standard Jeopardy! Questions, where the question contains one or more assertions about some unnamed entity or concept, and the task is to identify the described entity or concept. This style of question is a representative of a wide range of common question-answering tasks, and the bulk of the IBM Watson&... View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Identifying implicit relationships

    Publication Year: 2012, Page(s):12:1 - 12:10
    IEEE is not the copyright holder of this material | Click to expandAbstract | PDF file iconPDF (171 KB) | HTML iconHTML

    Answering natural-language questions may often involve identifying hidden associations and implicit relationships. In some cases, an explicit question is asked by the user to discover some hidden concept related to a set of entities. Answering the explicit question and identifying the implicit entity both require the system to discover the semantically related but hidden concepts in the question. ... View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Fact-based question decomposition in DeepQA

    Publication Year: 2012, Page(s):13:1 - 13:11
    Cited by:  Patents (2)
    IEEE is not the copyright holder of this material | Click to expandAbstract | PDF file iconPDF (503 KB) | HTML iconHTML

    Factoid questions often contain more than one fact or assertion about their answers. Question-answering (QA) systems, however, typically do not use such fine-grained distinctions because of the need for deep understanding of the question in order to identify and separate the facts. We argue that decomposing complex factoid questions is beneficial to QA systems, because the more facts that support ... View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • A framework for merging and ranking of answers in DeepQA

    Publication Year: 2012, Page(s):14:1 - 14:12
    Cited by:  Patents (3)
    IEEE is not the copyright holder of this material | Click to expandAbstract | PDF file iconPDF (1601 KB) | HTML iconHTML

    The final stage in the IBM DeepQA pipeline involves ranking all candidate answers according to their evidence scores and judging the likelihood that each candidate answer is correct. In DeepQA, this is done using a machine learning framework that is phase-based, providing capabilities for manipulating the data and applying machine learning in successive applications. We show how this design can be... View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Making Watson fast

    Publication Year: 2012, Page(s):15:1 - 15:12
    Cited by:  Papers (1)
    IEEE is not the copyright holder of this material | Click to expandAbstract | PDF file iconPDF (3389 KB) | HTML iconHTML

    IBM Watson™ is a system created to demonstrate DeepQA technology by competing against human champions in a question-answering game designed for people. The DeepQA architecture was designed to be massively parallel, with an expectation that low latency response times could be achieved by doing parallel computation on many computers. This paper describes how a large set of deep natural-langua... View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Simulation, learning, and optimization techniques in Watson's game strategies

    Publication Year: 2012, Page(s):16:1 - 16:11
    IEEE is not the copyright holder of this material | Click to expandAbstract | PDF file iconPDF (964 KB) | HTML iconHTML

    The game of Jeopardy!™ features four types of strategic decision-making: 1) Daily Double wagering; 2) Final Jeopardy! wagering; 3) selecting the next square when in control of the board; and 4) deciding whether to attempt to answer, i.e., buzz in. Strategies that properly account for the game state and future event probabilities can yield a huge boost in overall winning chances, when compar... View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • In the game: The interface between Watson and Jeopardy!

    Publication Year: 2012, Page(s):17:1 - 17:6
    IEEE is not the copyright holder of this material | Click to expandAbstract | PDF file iconPDF (734 KB) | HTML iconHTML

    To play as a contestant in Jeopardy!™, IBM Watson™ needed an interface program to handle the communications between the Jeopardy! computers that operate the game and its own components: question answering, game strategy, speech, buzzer, etc. Because Watson cannot hear or see, when the categories and clues were displayed on the game board, they were also sent electronically to Watson.... View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.

Aims & Scope

The IBM Journal of Research and Development is a peer-reviewed technical journal, published bimonthly, which features the work of authors in the science, technology and engineering of information systems.

Learn about subscribing to this journal >

Full Aims & Scope

Meet Our Editors

Editor-in-Chief
Clifford A. Pickover
IBM T. J. Watson Research Center