I lost on Jeopardy!, baby

by nkronos on February 15, 2011

The speed with which computers have surpassed the expectations of all but the most visionary science fiction writers has been astounding.  David Levy, both an international master at chess and an early expert on artificial intelligence, famously wagered that in the next ten years–starting in 1968–no computer could win a chess match with him. He won his bet, but was eventually defeated in 1989 after writing that “the idea of an electronic world champion belongs only in the pages of a science fiction book.” Likewise, when Deep Blue faced off against world champion and highest all-time ranked human player Gary Kasparov in 1996, Levy said he’d “stake my life” on Kasparov’s beating the computer 6 to 0. Kasparov won but not in a walk, and the following year a computer became the best chess player in the world.

Despite being situationally complex, however, chess is a finite game and differs from tic tac toe only in numerical possibilities, not in problem class. That computers can play tic tac toe perfectly implies that in theory a computer can do the same with chess; with enough memory and processing power a computer could deduce that even at the beginning of the match–just as with tic tac toe–the outcome is predetermined for one side to win or draw despite how the other player moves.

The reason that this is theoretical and not demonstrable is we are unlikely ever to reach that level of processing power because the Shannon number of chess positions is so astronomically huge–certainly beyond the upper bound of silicon-based processing over the duration of human life on earth at least.

Nonetheless, it is not surprising that an advanced enough computer can beat any human at this kind of problem. What surprised us was how quickly it happened.

The TV quiz show Jeopardy! is a much more interesting case, and so I find the three-match face-off between the computer, Watson, and its two human competitors, Jennings and Rutter, fascinating stuff.  Ever since John Henry duked it out with the steam drill, human beings have watched as their heroic selves have yielded one venue after another to the machine. Intelligence, however, the one trait that allows us dominance over the animals, has always been our exclusive purview. That may no longer be true…or at least true much longer.

The outcome of the match doesn’t really matter all that much. Because if Watson makes a respectable showing–and last night it beat one human challenger and tied the other–then, as with chess, the machine’s victory is only a matter of time. To be sure, certain kinds of questions are more difficult for Watson than for Jennings and Rutter because human thinking works differently than computers in terms of context, but over time those areas can be refined and narrowed, so that the realm of human superiority will constantly diminish.

Watson is enormously handicapped: its native tongue is bits of zeroes and ones, whereas the contestants are playing using the familiar English of Jennings and Rutter. That it may beat them anyway is remarkable.

To me the two most fascinating moments of the first match involve mistakes because of what they reveal about how Watson’s intelligence has developed. The first was a response to the clue “It was the anatomical oddity of US gymnast George Eyser who won a gold medal on the parallel bars in 1904.” Jennings answered incorrectly, “What is missing an arm?” Watson then said “What is leg?” That answer was disqualified because of the lack of the “missing” first part, even though it would have been accepted for a human player–judges reasoned that a human would have not bothered to restate that portion of the original guess, but since Watson cannot hear, it could not be doing the same thing and instead left out an important part of the answer.

That seems just a little human-centric! How can the same answer be correct for a human to respond, yet wrong for the computer? It also highlights the language difficulty for Watson in that, conceptually, “arm” may be sufficient to connote “missing arm” in the context of “anatomical oddity.” To put it another way, suppose Eyser’s arm was cut off at the elbow. Would you then as a human say his arm was “partly missing” or be satisfied with just “missing”? If I counted “missing arm” wrong, demanding “partly” as well, would you object?

The computer may have a similar dilemma in that it “thinks” if “oddity” has been stipulated, there is no need to refine “arm” further.  “Missing arm” and “arm oddity” might seem synonymous to the computer because the arm is odd only because it is missing.

The absence of a thing viewed through silicon reasoning may be a very odd form of the thing indeed.

Second, Watson answered the same wrong answer for a question that Jennings did regarding decades. This was again a handicap of not being able to hear as one would assume a human player wouldn’t make that mistake (although players in actual matches often do). Watson’s engineers talked about working out some way to prevent its doing that in the future, but to me it is the most important moment of the whole match.

Why? Because for a computer to think truly the same way we do, it has to make the same kinds of mistake in its thinking we do.

The second match is tonight.

Comments on this entry are closed.

Previous post:

Next post: