?

Log in

No account? Create an account

Statistical Mythology - TRUE! - Nothing to See Here

Jul. 24th, 2008

09:45 pm - Statistical Mythology - TRUE!

Previous Entry Share Next Entry

We're watching the movie "21". It seems like it has the potential to be an interesting story.

But then ... the story starts out with the main character's induction into a group of blackjack card counters at his university, MIT, because he wows his professor by first spouting history about Newton-Rhapson method (which apparently people in the US refer to only as Newton's Method! Huh!), which is fine, but then by spouting a STUPID logical fallacy, which the professor agrees with him on.

That fallacy?

Well, the question is:

The stupid, stupid, stupid answer is:This is crap, plain and simple. The fact of the matter is that YES, the odds have changed since you originally picked (where it was 33.3% for each), but they have changed to 50% each. There is no magic causing ONE of the cards to inherit the probability of the one that was eliminated.

I've heard this fallacy often enough, back in college at least.

It makes me really angry that a movie which a lot of people who don't know any better will see depicts a supposedly top-flight MIT student and an MIT professor proclaiming this as statistical fact. I hope I can enjoy the movie in spite of this early huge screw-up.


So, yeah, I was wrong. Argh.

I take it back. Nobody has ever succeeded in explaining this to my satisfaction. It's always sounded like hand-waving (hm, how zen). The statistical explanations have always sounded like voodoo.

I just worked it out on paper, and it actually ... does work that way. WEIRD. But true. I shall explain. Perhaps my reasoning (since it's the only reasoning which has persuaded me) will help persuade others

I've heard people talk of the fact that the host is an intelligent agent, and therefore not subject to statistical analysis and so on and so forth, but none of this has ever pointed out the real reason why switching is best:

In 2 out of 3 cases, you have forced the intelligent agent to take one, with no option. He only has a choice 1 out of 3 times!

When you have forced the agent's hand, the remaining card is it. When he had a choice of which card, the one you picked first was it.

Weird? Yes, but true. I came to this realisation after I jotted down the possibilities.

I = your initial pick
E = choice eliminated by host
S = the switch option


Door LayoutDealer Option 1Dealer Option 2Initial OutcomeSwitch Outcome
1 0 01(I) 0(E) 0(S)1(I) 0(S) 0(E)always winalways lose
0 1 00(I) 1(S) 0(E)--always losealways win
0 0 10(I) 0(E) 1(S)--always losealways win

(You can re-order things so that you don't always choose the first door, but that is statistically irrelevant, assuming the prize door is properly randomized).

So, ur, wow. 2/3 of the time, the agent was forced to point out the right door, effectively.

Weird, but no longer pure voodoo. If someone had shown me what I worked out, I would have believed them. Other people's explanations now make some sense, in hindsight, but really didn't in the absence of this realisation.

I guess I owe the makers of the movie an apology!

[Edited to fix my mismatched legend and use of symbols! Oops!]

Current Location: 806
Current Mood: embarrassedembarrassed

Comments:

[User Picture]
From:purpledrazi
Date:July 25th, 2008 05:24 am (UTC)
(Link)
hmmmm . . . unless I'm missing something, there are two doors left. One has a prize, the other doesn't. So you have a 50/50 chance whether you chance your answer or not . . . yes?

(Reply) (Thread)
[User Picture]
From:purpledrazi
Date:July 25th, 2008 05:25 am (UTC)
(Link)
never mind . . . I've re-read the thread . . . yep, that's bull

:)

(Reply) (Parent) (Thread)
[User Picture]
From:mavjop
Date:July 25th, 2008 08:03 am (UTC)
(Link)
Stay tuned for my big retraction! I was wrong, and I've worked out why.
(Reply) (Parent) (Thread)
[User Picture]
From:purpledrazi
Date:July 25th, 2008 08:22 pm (UTC)
(Link)
Ack! . . . that's what I get for not doing the math!!

:P
(Reply) (Parent) (Thread)
[User Picture]
From:athelind
Date:July 25th, 2008 05:43 am (UTC)
(Link)
It's counter-intuitive, but that IS what game theory says: you have better odds if you switch.

The trick is that "the Monty Hall Problem" is NOT a purely statistical situation: Monty knows which door has the prize, and he knows which two don't. He's not picking that door at random: if one of the remaining doors has a prize and the other doesn't, he's deliberately going to open the one with no prize, to keep the suspense going. He's an intelligent agent.

You had a 1/3 chance of picking the right door the first time. Monty has a 100% chance of picking an EMPTY door. So that means that the door that neither of you picked... has a 2/3 chance of having the prize.

Google "game theory three door"; every result (mostly from mathematical sites) agrees.
(Reply) (Thread)
[User Picture]
From:mavjop
Date:July 25th, 2008 08:29 am (UTC)
(Link)
Your explanation makes sense to me now, somewhat, but only after I figured it out in a different way. Wacky!

Another friend pointed out http://people.hofstra.edu/Steven_R_Costenoble/MontyHall/MontyHallSim.html ... but that really didn't do it for me. For one thing, over 20 samples of not switching, I got 50%. :P Of course, over a larger sample size of switching, I got ~67%.

I'm glad I finally understand this, and kind of annoyed that I was convinced it was wrong.
(Reply) (Parent) (Thread)
[User Picture]
From:athelind
Date:July 27th, 2008 05:42 pm (UTC)
(Link)
Yeah, it bugged the heck out of me until I finally realized that it's not purely a statistical problem, because it includes someone who knows the answer and is deliberately playing you.

This is also why I can see it as an ideal Thought Problem for dealing with cards, casinos, and "games of chance".

So, what made it finally make sense for YOU? Oh, wait, you edited the main entry. And, yeah, that makes it eaven clearer. I hadn't made the leap to "you're forcing the Intelligent Agent" in so many words; I hadn't quite consciously factored in MONTY'S odds.

Edited at 2008-07-27 05:46 pm (UTC)
(Reply) (Parent) (Thread)
[User Picture]
From:mavjop
Date:July 27th, 2008 06:27 pm (UTC)
(Link)
See my explanation in the post. :) I edited it to strike out all the stuff I said that was not true, and to add the explanation of why (in a way I finally could see as based on facts) it was true.

[Edit: Heh! ... and I just responded before I saw your edited comment :> Oops!]

Edited at 2008-07-27 06:28 pm (UTC)
(Reply) (Parent) (Thread)
[User Picture]
From:lessachu
Date:July 25th, 2008 05:00 pm (UTC)
(Link)
The Monty Hall problem strikes again!

Pretty much every time some statistician feels the urge to write a newspaper article or column on this problem, a blizzard of letters and e-mail on both sides of the solution happens.
(Reply) (Thread)
[User Picture]
From:mavjop
Date:July 25th, 2008 05:37 pm (UTC)
(Link)
The problem is, nobody's ever managed to persuade me by giving any clear explanation which made any sense.

I don't know if it's because I'm weird in some way or if my explanation really is clearer.
(Reply) (Parent) (Thread)