A reaction on "On a First-name Basis with Success? Your Mom Chose Your Name Wisely."

Earlier this week, the Business section of the Flemish quality newspaper 'De Standaard' reported that the shorter the first name, the higher the income (see here). The article showed a pricture of Bill Gates, with the caption: "Was using the nickname 'Bill' the key to the success of William Henry Gates?". The newspaper was refering to research carried out by TheLadders, a "job-matching service for career-driven professionals" and reported here. Basically, they analyzed data around first names from TheLadders’ nearly 6 million members and salary level.
The blog is more tongue in cheek than De Standaard article led us to believe, but the blog has found its way in social media, being liked and tweeted more than thousand times, and was caught up by the popular (and sometimes serious) press. There are, however, a few concerns with this research. Let me mention them one by one:

• The first concern is an obvious one: "Correlation is not causation". It's been said many times before, so  I don't need to do much explaining here, but it remains surprising to see that a lot of the reactions on the research really focused on the causal consequences. The Bill Gates example mentioned above is a case in point. The author's conclusion "to all prospective mothers, our advice is to keep Baby’s name short and sweet – your child will thank you when they’re raking in the money one day" was meant as humor, I suppose.
• The second concern is related to the first one, and that's spurious correlation: The observed relationship might be caused by a third, unseen factor (sometimes referred to as a "confounding factor" or "lurking variable"). To the rearcher's credit the fact that he did a separate analysis for women and for men already eliminates gender as such a confounding factor. But nonetheless it is perefectly imaginable that length of first name is related to age or ethnicity, two factors that have been reported to influence salary in previous research.  I don't have the data available right here, but I'm sure that it wouldn't be hard to figure out whether certain age cohorts of people were given longer names than other cohorts, for instance because long names might become fashionable or out of fashion again. Likewise, it can't be hard to show that certain enhnicities have in general longer or shorter names than others. 
• Another concern is the poor and confusing scientific language that is used. One example is "We wanted to prove the null hypothesis that what your mother names you makes a difference.". Null is misplaced here. I would rather say that we attempt to disprove the null that there is no difference. Sentences like "The definitive proof for this theory can be seen in Sara vs. Sarah, Michele vs. Michelle, or Philip vs. Phillip –  one letter less positively correlates with increased salary." should be avoided as well.
• Another statistical concern is that  if you use 6 million observation almost everything will become significant, but that does not mean the reported effect is substantial as well.
• While 6 million observations is huge, it does not mean that they are representative for the global population.
• And if you consider the 6 million  people as your population (to avoid the problem in the previous bullet), the is no real need to use inductive statistics in the first place.
• The graph with the average salaries by length of first name is somewhat misleading because the y-axis is not given and it is not exactly clear what scale is being used. 
  
• While the regression coefficient is given, it would have been good to report the $R^2$ as well. 

To illustrate how easy it is to report effects like these, while they're (likely) not there or very weak, let me get back to the election data I used in a pevious post in this blog. It's data on the results of the municipal elections in Flanders in 2012.
For each of the about 37000 candidates from more than 300 municipalities I calculated the number of characters in their first name and related that to the number of votes they received relative to the result of their party in their municipality. Below I tried to mimic the graph produced by TheLadders (and shown above). It is a histogram of frequencies superimposed with the average percentage of votes.

As you can see, the two graphs look pretty similar both in terms of pattern of frequencies and in terms of the decreasing red line. This graph (wrongly) suggests that there is a tendency that longer first names have a lower (relative) number of votes. Before you jump to conclusions, please read further.
I used a couple of tricks to exagerate the effect. Many of these tricks were applied by the TheLadder author as well:

1. By omitting labels on the y-axis, it is difficult to judge the scale.
2. There are names with only two letters and with more than 9. In my example I used the same buckets as TheLadder. If I had allowed names with 2 characters as well, the line would have looked different. In Flanders the majority of those names are "An", which is a female name. I don't know how that works in the US, but there are names like 'Al' that must come up frequently in the huge database TheLadder has used
3. The red line really hides the enormous variation in percentage of votes within each class of 'number of characters of the first name'.
4. By not reporting the $R^2$ of the model, I deny the reader to evaluate the strength of the model themelves. In this case here, while the regression coefficient is significant, the $R^2$ of the regression model explains less than 1% of the variance, so the effect is clearly not substantial. Notice that the opposite is not true. Having a high $R^2$ does not always indicate a strong relationship either, but that is a different story.
5. I've scaled my graph such that the decrease in the red line looks more dramatic (in all fairness I have to say that TheLadder author didn't do that).  

One more thing: Here's a list of candidates with names of length 11 with the frequency below each name:

Jean-Pierre Marie-Paule Jean-Claude Christianne Marie-Josée Anne-Sophie Christopher Marie-Elise 

         62          13           8           7           6           5           3           3 

Jean-Michel Ann-Pascale Anne-Claire Bernhardien Cemil-Jimmy Christianna Christoffer Danny-Spock 

          2           1           1           1           1           1           1           1 

Gust-Julien Guy-Maurice Hanne-Loren Hendrik-Jan Jan-Laurens Jean-Hubert Liese-Lotte Luigia-Gina 

          1           1           1           1           1           1           1           1 

Marie-Berte Marie-Josee Marie-Laure MarieJeanne Maximiliaan Pieter-Paul Salah-Edine Sebiha-Abla 

          1           1           1           1           1           1           1           1 

 

If you are familiar with names in Flanders you will agree that a lot of the higher frequency names are probably from older people, illustrating that behind "length of first name" lies a lurking variable, i.e. age.

I'm just giving this illustration, to show how easy it is to fool the (untrained) eye. Data scientists should avoid this kind of tricks and go for clarity instead.

Conclusion: Anyone who writes an article to pay respect to their mother has my support, but the article of TheLadder probably did not deserve to be spread as far as it did, and it certainly should not end up in a quality newspaper that De Standaard claims to be.