All Things Data Science

 Read my post on substack:  Calibration, weighting and post-stratification in audience measurement

With the rise of the 'Data Scientist', a lot has been said about the definition, role, qualifications and skills of the Data Scientist, and how to hire them. A somewhat neglected topic is how to manage data scientists. Indeed, data scientists, by their very nature, are hard to manage. They love to resolve problems, but those problems are not always the business problems you want them to tackle. They are ace players, but they're not always the best team players and some of them can sometimes have difficulty in dealing with (higher) management. They can have bright ideas, but they often lose interest when it comes to implementing those ideas in a profit making activity. They will find clever solutions for you, but they don't always excel in making sure that a structured process is place, let alone the administrative follow up that comes with it. Some of them were hired as 'rock-stars' and have developed an ego that goes with that... On the other hand, they are...

Some time a go I captured quite a lot of geo-located tweets for a spatial statistics project I'm doing. The tweets I collected were all confined to be in Belgium. One of the things I looked at was the language of tweets. As you might know, Belgium officially has three languages, Dutch, French and German. Of course, when you analyze a large set of tweets, you can't manually determine the language, on the other hand blindly relying on Twitter's language detection algorithm doesn't feel good either. That's why I set up a little experiment to assess to what extent Twitter's language detection algorithm can be trusted, in the context of  my geo-location project. I stress this because I don't have the ambition to make overall judgments on how Twitter takes care of language detection. First, let's look at the languages as determined by the Twitter language detection algorithm of the 150,000 or so tweets I collected. The barchart below shows the frequency of...

Forget Paul the Octopus, Chanakya the Fish, and all other football predicting animals. It appears that the stock market can be used to predict the outcome of  football matches!  Now that the stock markets are not doing so well, at least we can use them to make a few bucks on football betting sites. Take last weekend's game, for instance, when Germany played Greece. It turns out that the score evolution of this game followed a very similar pattern than the German Greek spread from half March to half June 2012. The pattern is obvious in the graph below. The correlation between the two series is an impressive 93%. The attentive reader will notice that there is a gap between the 45 and 60 minutes marks. Indeed the stock markets predicted that the third and fourth German goals would have been scored earlier. Specialists are investigating whether this has to do with the break after the first half . But, other than that small gap, the fit between the two lines is very close. These...

Currently there is a debate in Belgium about the construction of Uplace, a huge shopping mall on the outskirts of Brussels. A marketing professor of the Vlerick School, Gino Van Ossel, did a survey which showed that, contrary to popular believe, quite a big group was actually in favor rather than opposing the plans. So far so good, but the professor's methodology was questioned in the media. The arguments used against the survey findings were not very strong, I believe, and I will not discuss them here. However, a part of the reasoning used by Gino Van Ossel looked rather odd to me and made me think about a more general problem that I would like to discuss here. Those of you who understand Dutch can find all the details on www.marketingblog.vlerick.com . In short, he found that in a sample of 654, representing the total Belgium, 33% was in favor, while in the region where the shopping mall would located, with a sample of 182, 46% were in favor. This was against the popular believ...