Tuesday, November 1, 2011

The DoT Matrix -- Establishing a Baseline

So I've gone and done it.

A while back Ken Jennings posited in an interview that "Most Jeopardy players know most of the answers." So I decided to put that to the test.

Assume with me, if you will, that:
  • Some games are harder than others, but over the course of a five game stretch, the difficulty level of the clues will pretty much average out.
  • Some players are better than others, but the average skill level of a group of challengers is pretty consistent from one week to the next.
If we grant those assumptions, we can begin to construct a profile for an "average" Jeopardy player by analyzing how often clues go unanswered by all three.

If the probability of a given player not knowing the response to a clue is X, the probability of all 3 in a given game not knowing the correct response is X ^ 3.

So step one in building the DoT Matrix was to find out if Cube Rule is correct. I randomly selected 45 games from the last 3 seasons, where none of the players would eventually win 4 or more games. I then counted "Triple Stumpers" -- unanswered clues -- in each row of the game board.

I did not consider Daily Doubles, since their placement is essentially random and since only one player has an opportunity to respond to them.

The Triple Stumper Rate, or TSR, is our X ^ 3. So the cube root of the TSR is the probability that an average Jeopardy player will not attempt a buzz. 1 - X, then, is our Buzz Rate.

The problem we have at this point is the fact that lack of knowledge is not evenly distributed through the Jeopardy player pool. There are some clues so badly written that nobody is going to get them, and there are other clues in subjects so obscure that very few Jeopardy players ever consider learning them for the show.

I wrote one such category for an SHC a while back about Lawn Mowers. Never again.

But how far do those distribution irregularities skew the players' performance? Quite a bit, as it turns out. I tested the Cube Root Hypothesis against performance in Final Jeopardy -- the only clue inthe game that everybody gets a chance to respond to. If the Cube Root Hypothesis is correct, the correct response rate in Final Jeopardy should be roughly equal to the cube root of TSR.

Spoiler alert: It's not. In fact, the Cube Root Hypothesis underestimates the knowledge level of an average Jeopardy player by about 20%. Which means that if the Cube Root Hypothesis says that a typical Jeopardy player should only know 30 responses per game, he probably knows closer to 36.

So adjusting for CRH error, we find that the typical Jeopardy player will have a correct answer and attempt to buzz 35.56 times on every 60 clues he sees.

So if you've ever wondered how you compare to the players you see every night on your TV, there you go.

NEXT: Using Buzz Probabilities and Coryat Rules to calculate a score for yourself as you play along at home.

2 comments:

MikeDupee said...

FWIW, when I was preparing for the UTOC, I knew an average of 54 questions per game.

Ben Wiles said...

Well, you were one of the all-time greats.

During the five-game stretch surrounding my show, Ken Jennings averaged knowing 54 questions per game, and won on the buzzer 66% of the times he attempted to ring in.

I'll get to how I derived these numbers later, but for now it looks like you and he would be a pretty even match head to head.