Song choice revisited

As regular readers of this blog will know, I don’t ascribe as much weight to the idea of song choice as other sites. The popular site “What Not To Sing” seems to rest on the idea that song choice is critical, but I just don’t see it. As people have said many times, if you can sing, you can sing. People have pulled out great performances on strange songs like Hemorrhage (In My Hands) (Chris Daughtry, a 94 WNTS rating in the Men’s Semifinals of Season 5), and people have crashed and burned with what are ostensibly good choices (how about Paige Miles’ performance of Against All Odds (Take a Look at Me Now) in Season 9’s Top 11?).

But those are just cherry picking. Suppose we look back on all seasons and try to suss out what the import of song choice is. Unfortunately, we’re going to run into a problem right away, in that any given song will only have been sung a few times in all of Idol history. The most sung is I Have Nothing by Whitney Houston, with only 8. I’m not going to be drawing any conclusions with a sample set of 8.

The obvious analytic way out of this is to group songs by common factors, or otherwise quantify songs along some dimension. One way to do this is using the “Whitburn score” (my term, don’t bother googling it). Suppose a song charts for one week on the Billboard Hot 100, at position 100. Then we give that song a Whitburn Score of 1. If it never charts we give it a Whitburn score of 0. We define the score as the sum of 101 minus the Hot 100 position for each week the song charts. Imagine Dragons’ Radioactive charted for 74 weeks, and has a Whitburn Score of almost 6000. I’m Yours, Rolling in the Deep, Smooth, Somebody That I Used to Know also all rate near the top.

So the first question I have to ask is: does the frequency with which someone is safe in the contest depend at all on a metric like this? If the answer is yes, we ask an even more interesting question: is the more familiar song safer or less safe?

Let’s use the logit link function to do a simple linear fit to the historical data. We fit whether a performance was safe or not versus a bunch of potentially relevant variables, such as the order of the performance, the WNTS Rating, whether or not the contestant was in the bottom group previously, and some metrics that rate the popularity of a song

glm(formula = Safe ~ Season + Order + TotPerfs + WNTS.Rating + 
    Bottom.Prev + VFPercent + YearOfRanking + YearlyRank + WeeksCharted + 
    Charted40OrBetter + Charted10OrBetter + HighestPosition + 
    RadioPlays + WhitburnScore, family = binomial, data = SongData)

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-2.6702  -0.9666   0.4720   0.8418   1.8900  

                    Estimate Std. Error z value Pr(>|z|)    
(Intercept)       -1.501e+01  1.415e+01  -1.061   0.2887    
Season            -7.699e-02  3.333e-02  -2.310   0.0209 *  
Order              1.134e-01  2.760e-02   4.108 3.98e-05 ***
TotPerfs           4.324e-02  4.411e-02   0.980   0.3270    
WNTS.Rating        3.007e-02  4.157e-03   7.233 4.73e-13 ***
Bottom.Prev       -8.484e-01  1.730e-01  -4.903 9.44e-07 ***
VFPercent          4.024e-02  7.135e-03   5.640 1.70e-08 ***
YearOfRanking      7.042e-03  7.218e-03   0.976   0.3292    
YearlyRank         1.626e-03  1.906e-03   0.853   0.3936    
WeeksCharted       2.216e-02  2.172e-02   1.020   0.3077    
Charted40OrBetter  9.316e-03  3.306e-02   0.282   0.7781    
Charted10OrBetter -3.449e-02  2.450e-02  -1.408   0.1592    
HighestPosition   -1.717e-02  9.983e-03  -1.720   0.0854 .  
RadioPlays        -1.142e-05  7.494e-05  -0.152   0.8789    
WhitburnScore     -4.273e-04  4.336e-04  -0.985   0.3244    
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 1296.6  on 1007  degrees of freedom
Residual deviance: 1058.4  on  993  degrees of freedom
  (823 observations deleted due to missingness)
AIC: 1088.4

Number of Fisher Scoring iterations: 5

The initial findings are not promising. By far the most significant variables are WNTS Rating, Votefair popularity, and whether or not the person had previously been in the Bottom Group. We can see that many of the variables are not significant. For instance, whether the song is actually played on the radio, which you might think would matter. In fact, it’s quite likely that makes no difference (at least on the entire data set).

Now let’s limit the model to just variables that seem to have a snowball’s chance in hell of mattering:

glm(formula = Safe ~ Order + WNTS.Rating + Bottom.Prev + VFPercent + 
    WeeksCharted + HighestPosition + WhitburnScore, family = binomial, 
    data = SongData)

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-2.6673  -0.9666   0.4831   0.8441   1.8820  

                  Estimate Std. Error z value Pr(>|z|)    
(Intercept)     -1.2971209  0.2918839  -4.444 8.83e-06 ***
Order            0.1263050  0.0258266   4.891 1.01e-06 ***
WNTS.Rating      0.0279386  0.0039534   7.067 1.58e-12 ***
Bottom.Prev     -0.9486016  0.1578972  -6.008 1.88e-09 ***
VFPercent        0.0395262  0.0066878   5.910 3.42e-09 ***
WeeksCharted     0.0403195  0.0182330   2.211  0.02701 *  
HighestPosition -0.0089462  0.0037305  -2.398  0.01648 *  
WhitburnScore   -0.0006699  0.0002277  -2.942  0.00326 ** 
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 1333.5  on 1037  degrees of freedom
Residual deviance: 1097.9  on 1030  degrees of freedom
  (793 observations deleted due to missingness)
AIC: 1113.9

Number of Fisher Scoring iterations: 5

We can see that it’s still really dependent on how popular the contestant is and how well they sang the song. I am fairly confident that the idea that “song choice is everything” is really not the case.

But anyway, this article is focusing on song choice only, and we do see some effect of how well the song did on the Billboard Hot 100 charts, like the Whitburn Score. If we wanted to model the contest only that way, we could fit based on only song-related variables

glm(formula = Safe ~ WhitburnScore, family = binomial, data = SongData)

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-1.5172  -1.3885   0.8725   0.9611   1.3168  

                Estimate Std. Error z value Pr(>|z|)    
(Intercept)    0.7706726  0.0756893  10.182  < 2e-16 ***
WhitburnScore -0.0002011  0.0000519  -3.875 0.000107 ***
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 2405  on 1830  degrees of freedom
Residual deviance: 2390  on 1829  degrees of freedom
AIC: 2394

Number of Fisher Scoring iterations: 4

Of course, the Whitburn score now seems much more important when the other variables are not taken into account, as it’s easier to pass the significance test (NB: some of that is due to the fact that number of data points is now higher, since not all variables are known for all performances, such as Votefair). In fact, it’s quite likely that the WNTS Rating is related to the Whitburn score. We can test this using a linear regression, but before we do that, let’s look at the parameter estimate generated by R. Safe is being fit versus Whitburn Score, which is significant … but you are less safe the more popular the song was! The probability of being safe is actually reduced by singing a song that charted a lot.

To see how these are related, let’s do that linear regression now:

lm(formula = WNTS.Rating ~ WhitburnScore, data = SongData)

    Min      1Q  Median      3Q     Max 
-53.237 -16.237  -0.292  17.131  45.543 

                Estimate Std. Error t value Pr(>|t|)    
(Intercept)   55.2371786  0.7801521  70.803  < 2e-16 ***
WhitburnScore -0.0036882  0.0005484  -6.726 2.33e-11 ***
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 21.78 on 1829 degrees of freedom
Multiple R-squared:  0.02414,   Adjusted R-squared:  0.0236 
F-statistic: 45.24 on 1 and 1829 DF,  p-value: 2.327e-11

Indeed, WNTS Rating is decreased by increasing Whitburn Score, less popular songs are rated more highly. People actually rate unfamiliar songs better than familiar ones!

What conclusions can we draw from this? Someone’s chances overall are mostly dependent on how well they sang, how popular they are, and not on what song they chose. But to the degree that song choice does matter, you are better off choosing a song the audience is only marginally familiar with, or a song they’ve never heard.

Here’s the data in csv form.

Bookmark the permalink.

Comments are closed