# Statistical Mythology - TRUE! - Nothing to See Here

## Jul. 24th, 2008

### 09:45 pm - *Statistical Mythology - TRUE!*

We're watching the movie "21". It seems like it has the potential to be an interesting story.

But then ... the story starts out with the main character's induction into a group of blackjack card counters at his university, MIT, because he wows his professor by first spouting history about Newton-Rhapson method (which apparently people in the US refer to only as Newton's Method! Huh!), which is fine, ~~but then by spouting a ~~**STUPID logical fallacy**, which the professor agrees with him on.

That fallacy?

Well, the question is:

~~Host shows you 3 doors. Behind 1, a prize. Behind the other 2, nothing.~~

You choose a door.

The host opens one of the other 2 doors, showing nothing.

He offers you the option of changing your guess.

Should you change it?

You choose a door.

The host opens one of the other 2 doors, showing nothing.

He offers you the option of changing your guess.

Should you change it?

~~You should change your guess!~~

*hand-waving*

*statistical bullshit*

... so the one you chose first was 33.3%, which means the other remaining one has become 66.7%!

*hand-waving*

*statistical bullshit*

... so the one you chose first was 33.3%, which means the other remaining one has become 66.7%!

I've heard this fallacy often enough, back in college at least.

It makes me really angry that a movie which a lot of people who don't know any better will see depicts a supposedly top-flight MIT student and an MIT professor proclaiming this as statistical fact. I hope I can enjoy the movie in spite of this early huge screw-up.

So, yeah, I was wrong. Argh.

I take it back. Nobody has ever succeeded in explaining this to my satisfaction. It's always sounded like hand-waving (hm, how zen). The statistical explanations have always sounded like voodoo.

I just worked it out on paper, and it actually ... does work that way. WEIRD. But true. I shall explain. Perhaps my reasoning (since it's the only reasoning which has persuaded me) will help persuade others

I've heard people talk of the fact that the host is an intelligent agent, and therefore not subject to statistical analysis and so on and so forth, but none of this has ever pointed out the real reason why switching is best:

**In 2 out of 3 cases, you have forced the intelligent agent to take one, with no option. He only has a choice 1 out of 3 times!**

When you

**have**forced the agent's hand, the remaining card is it. When he had a choice of which card, the one you picked first was it.

Weird? Yes, but true. I came to this realisation after I jotted down the possibilities.

*I = your initial pick*

E = choice eliminated by host

S = the switch option

E = choice eliminated by host

S = the switch option

Door Layout | Dealer Option 1 | Dealer Option 2 | Initial Outcome | Switch Outcome |
---|---|---|---|---|

1 0 0 | 1(I) 0(E) 0(S) | 1(I) 0(S) 0(E) | always win | always lose |

0 1 0 | 0(I) 1(S) 0(E) | -- | always lose | always win |

0 0 1 | 0(I) 0(E) 1(S) | -- | always lose | always win |

(You can re-order things so that you don't always choose the first door, but

**that**is statistically irrelevant, assuming the prize door is properly randomized).

So, ur, wow. 2/3 of the time, the agent was forced to point out the right door, effectively.

Weird, but no longer pure voodoo. If someone had shown me what I worked out, I would have believed them. Other people's explanations

*now*make some sense, in hindsight, but really didn't in the absence of this realisation.

I guess I owe the makers of the movie an apology!

*[Edited to fix my mismatched legend and use of symbols! Oops!]*

**Current Location:**806

**Current Mood:**embarrassed

purpledrazi(Link)purpledrazi(Link):)

mavjop(Link)purpledrazi(Link):P

athelind(Link)The trick is that "the Monty Hall Problem" is NOT a purely statistical situation: Monty

knowswhich door has the prize, and he knows which twodon't.He'snot picking that door at random: if one of the remaining doors has a prize and the other doesn't, he's deliberately going to open the one with no prize, to keep the suspense going. He's anintelligent agent.You had a 1/3 chance of picking the right door the first time. Monty has a

100%chance of picking an EMPTY door. So that means that the door that neither of you picked... has a 2/3 chance of having the prize.Google "game theory three door"; every result (mostly from mathematical sites) agrees.

mavjop(Link)Another friend pointed out http://people.hofstra.edu/Steven_R_Costenoble/MontyHall/MontyHallSim.html ... but that really didn't do it for me. For one thing, over 20 samples of not switching, I got 50%. :P Of course, over a larger sample size of switching, I got ~67%.

I'm glad I finally understand this, and kind of annoyed that I was convinced it was wrong.

athelind(Link)knows the answer and is deliberately playing you.This is also why I can see it as an ideal Thought Problem for dealing with cards, casinos, and "games of chance".

~~So, what made it finally make sense for YOU?~~Oh, wait, you edited the main entry. And, yeah, that makes it eaven clearer. I hadn't made the leap to "you're forcing the Intelligent Agent" in so many words; I hadn't quite consciously factored in MONTY'S odds.Edited at 2008-07-27 05:46 pm (UTC)mavjop(Link)[Edit: Heh! ... and I just responded before I saw your edited comment :> Oops!]

Edited at 2008-07-27 06:28 pm (UTC)lessachu(Link)Pretty much every time some statistician feels the urge to write a newspaper article or column on this problem, a blizzard of letters and e-mail on both sides of the solution happens.

mavjop(Link)I don't know if it's because I'm

weirdin some way or if my explanation reallyisclearer.