1. Of course, round 1 of the Netflix competition has ended, but did you know there’s a round 2 — also with a $1 million prize? Round 2 will be a time-limited contest involving sparse datasets. The full details for the Netflix 2 prize will be announced in the near future on their website. Once the contest has been officially started, it will have a progress prize at 6 months and then finish at 18 months.

2. There’s a statistical methods competition called the OMOP Cup: Method Competition. It’s organized by the Observational Medical Outcomes Partnership. The purpose is to improve on current methods of utilizing real-time data to ensure drug safety. There are two parts to the competition (taken from the website):

• Challenge 1 explores how well your method works when provided an entire dataset, so the goal is accurate classification of which drugs are associated with which outcomes.
• Challenge 2 evaluates the timeliness of detection of drug-event associations by having your methods run against data sequentially as it accumulates over time.

The total prize money is $20,000. Visit the OMOP Cup: Method Competition website for full details.

3. Every year, KDD (Knowledge Discovery and Data-Mining ) sponsors a data-mining competition with a cash prize of around $5000. The competition is usually announced in Spring, so apologies for mentioning it now – you will have to wait until 2010. You can look at info on past competitions here.

For the past 50 years, it has been postulated that computer learning algorithms would out-pace the human mind in deriving laws of behavior from large, complicated datasets. Prizes (like the Leibniz Prize) were even been offered for the first program to fundamentally change mathematics. Despite many earnest attempts, Hal has yet to be created.

However, the modern availability of cheap memory and speed has meant that the processing power of researchers has grown exponentially, allowing for more sophisticated learning algorithms. Researchers have been able to learn from these algorithms, refine them and create even more elaborate methods until voila! A computer has been able to derive laws of physics with nothing more than experimental data. 

To be more specific, the researchers, Schmidt and Lipson, fed experimental data on several simple systems such as a weight and spring or a 2-arm pendulum into the program. The program also had knowledge of simple mathematical operations. Using a method similar to Monte Carlo simulations, it kept trying and optimizing different forms of equations until it had derived equations that described the systems. In the process, it also expressed some underlying laws like conservation of momentum and Newton’s 2nd law of motion.

In another example of data-mining, researchers were able to answer one part of a hieroglyphic mystery that has perplexed archeologists to this day. The Indus Script from 4,000 years ago has remained undeciphereable. Some linguists have insisted that it is no language at all, but merely political ciphers (like the Democtratic donkey or Republican elephant) that were important in that day. The problem is that the longest chain of Indus Script contains only 27 characters, making it extremely difficult to crack. A group of researchers has now managed to “prove” that it is a real language by showing that the entropy level of the order of the Indus characters is very similar to human language. 

Rao, a machine learning expert, and his team, fed their learning algorithm several different modern and ancient languages and measured the entropy of the word order. They also fed it non-verbal communication like DNA and FORTRAN code. These non-verbal communications turned out to have either extremely low or high entropy. The Indus language matched the mid-entropy level of human languages.

In other fascinating news, did you know that you can’t tie your shoelaces? If you don’t believe me, visit Ian’s shoelace site. Apparently, most of us use Granny knots to tie our shoelaces instead of the “correct” Shoelace knot.

Now you know.

Citations:

“Distilling Free-Form Natural Laws from Experimental Data.” By Michael Schmidt and Hod Lipson.  Science, Vol. 324, April 3, 2009.

Wired Science article on the topic: http://www.wired.com/wiredscience/2009/04/newtonai

“Entropic Evidence for Linguistic Structure in the Indus
Script.” By Rajesh P. N. Rao, Nisha Yadav, Mayank N. Vahia, Hrishikesh
Joglekar, R. Adhikari and Iravatham Mahadevan. Science, Vol. 324 Issue
5926, April 24, 2009.

Wired Science article on the Indus language:   http://www.wired.com/wiredscience/2009/04/indusscript#comment-152291563

Ian’s shoelace website: http://www.fieggen.com/shoelace/index.htm

Professor Noshir Contractor from Northwestern University and his colleagues recently data-mined the massive 60-terabyte dataset released by EverQuest II. In case someone out there has never heard of EverQuest, it’s an online roll-playing game occupying the spare time of about 45 million people.

Noshir found some interesting results as a result of his efforts. First off, those 45 million people are NOT mostly teens. The average age was much higher. And despite having the entire world literally at their finger-tips, people still tended to network with their geographical neighbors.

“People end up playing with people nearby, often with people they already know,” Contractor said. “It’s not creating new networks. It’s reinforcing existing networks. You can talk to anyone anywhere, and yet individuals 10 kilometers away from each other are five times more likely to be partners than those who are 100 kilometers away from each other.”

I would have expected some grouping due to language barriers, but the geographical localization far exceeds that. I wonder if the even stronger statement could be made that players tend to play with people they previously knew?

A survey was also distributed to 7,000 players. Using the results of the survey, Professor Contractor found disproportionate rates of self-reported depression vs. the general population. Additionally, he found that players tended to understimate how much time they devoted to the game, and that women are generally the most devoted and content (but apparently don’t like to play with other women!).

Hmmm…. So perhaps the guys are depressed because after long hours of playing EverQuest with mainly other guys they realize after the game has been switched off they still don’t have a girlfriend? Whereas women spend their time mostly getting attention from lots of admiring EverQuest men so it makes up for not having a boyfriend?

One thing this shows is that patterns can exist in data, but the reasons behind those patterns may still be a mystery. I wonder if Professor Contractor has plans to follow up his research with a further survey to try to tease out of players why they interact the way they do – now that he knows how they interact!

Whatever the next step, it was an interesting series of finds extracted from a massive dataset. Please refer to the original article written by Megan Fellman from Northwestern University’s news center. You can also visit Professor Noshir Contractor’s blog.

The NY Times recently described an interesting policy by American Express. The credit card company had been lowering the credit limits of customers who had shopped at certain retail outlets. Using their proprietary dataset gathered over their customer base, American Express had identified certain retailers whose customers had a hard time paying their credit card bills. They concluded that all customers who shopped at these stores were a credit risk and correspondingly lowered their credit limits. The catch here is that American Express did not reveal their retailer black list. So customers had no way of knowing which stores to avoid. Walmart? Niemann-Marcus? Baskin-Robbins?

Once the story appeared into the press American Express recanted its policy. In fact, they went one step further and insisted this had never been their policy, despite thousands of letters to “curtailed” customers that explicitly detailed otherwise. Read the original NY Times report of the surprising use of American Express data mining.

In other news about creative uses of data mining…

Microsoft Live search is thinking about inserting social technology into its searching service. They found that a relatively new technique called “groupization” turned up more relevant results in an internal test-run. The idea is to use a person’s social network to influence the results that are returned to a user. A user searches using a set of keywords which are then correlated with the results the user’s social group found relevant. While Microsoft was keen on the idea, they were also worried that the implementation on a large scale might be nigh impossible. You can read a summary of the idea at the online website ars technica.

Personally, I have my doubts that this would really contribute to my personal search results. I’ve noticed that the advertisements on the social networking site Facebook are 180-degrees different from what I’m actually interested in. I get a bunch of ads for the acai berry diet, movies,  and how to get long eye-lashes. In case anyone from Facebook is reading this, I don’t want to detangle my eye-lashes in the morning! I think the low relevance of the ads on Facebook have to do with the fact that it’s a social networking site and therefore relatively “fluffy”. Perhaps Microsoft could create its own pre-defined community titles and a user could click their interests/hobbies when they create an account profile. Then based on these broad categories, Microsoft could perform the “groupization”. This might be a scalable approach to their strategy.