On the accuracy of Dialidol

Since I’ve started building a forecast model for Idol, I’ve been terribly interested in Dialidol, a service that measures votes and busy signals during American Idol voting, and gives a raw score. Their main touted statistic is that they have never flubbed a prediction of the finale: since Season 4 (when they started), they have correctly predicted the winner of the contest every time.

However, if dialidol was a perfect forecast, there would never be a need for anything else, and it’s actually quite far from a perfect forecast. I can think of lots of reasons for this, but the main problem with dialidol is, of course, that it’s got sampling problems. It only measures a small number of people using the software to register votes by calling on a land line. Cell phone calls, texts, and now internet votes, are never counted. Moreover, the fewer people watch Idol (and its ratings have been dropping), the worse that sampling gets due to simply numbers.

The first, and by far the most important question, is how often does Dialidol accurately forecast the person going home: Continue reading

Surprise elimination and the save rule

Casey Abrams had the lowest number of votes last night, even though he was a highly popular contestant, with plenty of name recognition, and had previously never been in the bottom 3. He was voted in by the public, not a wild card. How often does it happen that a contestant is eliminated without his fans knowing that he was in trouble?

Excluding the first two rounds of the finals, and the top 3, it has actually happened 15 times.

Season 1 Tamyra Grey was eliminated from the Top 4
Season 2 None
Season 3 Jon Peter Lewis in the Top 8
Season 4 Nikko Smith in the Top 9
Constantine Maroulis Top 6
Season 5 Mandisa from Top 9
Season 6 Chris Sligh Top 10
Gina Glocksen Top 9
Season 7 Chikezie from Top 10
Ramiele Malubay from Top 9
Michael Johns from Top 8
Season 8 Megan Joy Top 8
Season 9 Didi Benami from Top 10
Andrew Garcia from Top 9*
Michael Lynche from Top 9 (saved)
Siobhan Magnus from Top 6

*During a double-elimination week

You can see that nearly all of these eliminations happened in the Top 8-10. I think this is kind of interesting, actually. I’ve heard of the judge’s save referred to as the Chris Daughtry rule, but this is wrong. Daughtry’s fans had every reason to believe he could be in trouble: he was in the bottom 3 during the Top 7 round, and was eliminated in the Top 4. What this should really be known as is The Constantine Maroulis rule! Similarly, the season 8 save was Matt Giraud, but he was in the bottom 3 in the Top 10 round and saved in the Top 7. That’s not very far removed. If his fans really just weren’t voting because they thought he was safe, they sure don’t have a very long memory.

Surprise eliminations like Casey are inevitable in Idol, particularly in the first few rounds, after people have seen their favorites sail through and start voting for their second favorites—ones they like, but aren’t sure are safe. However, these make it damn hard to predict what is going to happen in the first volatile weeks. The judges save has been invoked twice rightly, once wrongly, and in each case for men.

In the future, any predictive model is going to need to take into account dialidol or some other measurement, because the results alone would never have had any indicator that Casey was in danger, let alone the losing competitor. He was male, went first (historically not a position that was dangerous), had lots of pre-exposure, played an instrument, chose interesting songs—last night never should have happened. The fact that it did shows just how strange the voting public is.

Final 11 Post game: one of the craziest results shows ever

Three people in the bottom 3 who had never been there before. It’s only happened twice in 57 before (3% of the time):

Season 4, Final 9, Scott Savol, Nikko Smith, Vonzell Solomon
Season 5, Final 9, Elliott Yamin, Paris Bennett, Mandisa

I’m inclined to say that model could never have predicted that, but the system clearly needs a lot more refinement. Part of the problem is that the ratings were based on a few individuals, instead of a broad base survey. I’ll rerun the numbers when those numbers become available.

Still, just a shocking result. Without having seen the show, I can’t believe it.

Final 11 Results: Quick and dirty predictions

A very preliminary logistical regression model predicts the bottom 3 is Haley, Naima, and Paul. Naima is predicted to go home tonight, but only barely. Once I’ve ironed out some of the details, I’ll explain the reasoning here, and I hope to have some error analysis along with it.

Ranking Name
11 Naima
10 Haley
9 Paul
8 Stefano
7 Pia
6 Lauren
5 Thia
4 Casey
3 Jacob
2 Scotty
1 James

Final 11 – Minority Report

I’m not really inclined to agree that Naima is the next most likely to be voted off. She’s just more memorable than some of the other girls, although her performances for the past couple of weeks have been at best uneven (and, to be perfectly honest, pretty bad at times). That said, when you say something about Naima, people who watch Idol will know immediately who you’re talking about. The judges call this “flavor” or, perhaps, “flava.” When you say “Haley” or perhaps even “Thia” it may take people a moment to recall who you’re talking about. I’m guessing this might have played some part in Karen’s early elimination; Karen is just such a boring name, and didn’t really express who she was as a person (not that I demand everyone have a stereotypically ethnic name should they fail of being a white person).

I confess I hope the next person to be eliminated is a guy, and I do hope that it is Paul. I can really do without him, and again, I feel like when someone starts talking about “Paul” I don’t instantly know who they mean. “Oh, you mean the real quiet dude who you can barely hear?” I have a feeling that having an interesting name (Ace Young, for example) may play some small factor, though we have done no data analysis looking into that theory. “Paul McDonald” just isn’t it. It doesn’t even have the pleasing assonance of “Casey Abrams.” For that as well as his generally mediocre-to-crappy singing and performance style, I would say that if American sends a guy home, it will and should be Paul.

If the anti-female bias does come into play, I would definitely put my metaphorical  money on Haley over Naima, but I have been wrong many times before.

Hey America, try shedding some mediocre dudes for awhile!