Geek Challenge Results: March Madness Part II

A big question that lingered after looking at the results of the geek challenge was how much luck played into the outcome against how strong any specific entry was over any other.  Most entries relied heavily on seeding and used other factors to tweak the effect of the overall seeding. Only Dan Freve chose a different approach, weighting strength of schedule heavily and actually using a negative coefficient on the tournament seed. His formula is reasonable, somewhat mimicking the RPI rating used by the NCAA’s selection committee.

First, we looked at how much randomness gave optimum results. To normalize the analysis, we looked at it as a ratio of the randomness coefficient K divided by the maximum coefficient for rating.

There was a wide variation in K values, ranging from the low of Ken (0.35 times his max coefficient) to a high of Kevin (3 times his max coefficient).  We found a near optimum K given the actual tournament results and random number sets for each entry by simple brute force, subject to using a resolution of 1/20th the maximum coefficient value for that entry.  The optimal K seemed to lie roughly around the maximum coefficient for most entries. Bruce was noticeable exception, as his random numbers did not favor a Kentucky team that his coefficients favored and that won the tournament. Choosing a low randomness in his case allowed his entry to minimize the impact of his unlucky random numbers. More on just how unlucky Bruce’s brackets were is discussed below.
 

Alex, Bruce, and Kevin did themselves damage by choosing high K’s, while the rest of the field obtained better than 95% of their points with their chosen randomness factor. In David’s case, choosing a relatively high K had little impact on his score.

Each entry was given a unique set of 630 random numbers to pick the 63 games in each of 10 brackets. So it is possible that our winner, Dean, either had great skill in choosing coefficients, or was lucky in the random number draw, or some combination of the two. In order to figure out how much of Dean’s victory was luck and how much was choosing a good entry, we ran the random numbers to find the results for 10 different outcomes using this year’s brackets to estimate the mean score and standard deviation for each entry. Using those statistics, we determined which entries were relatively lucky, and what the win probability for each entry was. We then re-ran the analysis using last year’s bracket to see how robust the results are year to year. 

The mean is an indication of how good an entry is for the specific teams and outcome of this year’s tournament. The deviation is how much luck someone had, with positive numbers being good luck and negative numbers bad luck. So Dan had the most luck, but it wasn’t enough to overcome Dean’s high mean and modest luck.

Dean’s entry dominated this year’s tournament and he had an estimated 50% chance of winning. He would have been in the mix last year as well. Bruce’s entry also stood a fair chance to win in both years. David and Alex stood a better chance last year, while Dan’s entry was much better for this year’s tournament. Ken and Kevin not only stood at the bottom of the heap in terms of average, but with low standard deviations, our brackets had little chance of making a significant impact.

Bruce and Alex led the unlucky crowd. I was surprised by just how unlucky Bruce was, as in a normal distribution only 1 in 14000 entries would be worse than Bruce’s. I had to double check his entry and that our score distribution was in fact close to normal over the range of data we produced. We ran the other 6 entries on Bruce’s random numbers and it produced similarly bad outcomes (-2.7 to -4.8 deviations) for every entry except Kevin’s—where it performed really well. (+1.0 deviation) (Note: Had I just used Bruce’s set of random numbers for every entry, I would’ve won by a landslide.)  The conclusion reached is that while our distribution is close to normal within about 2 standard deviations of the mean, larger outliers can occur more often than would be expected. This is because in some situations, the results obtained can be very dependent on one or two random numbers if they affect the bracket that results in your maximum score. In Bruce’s case, his random numbers were very unfavorable to Kentucky. Even though he rated them highly, they did not wind up winning a single one of his brackets--costing him 320 points on his best bracket and total score.

Finally, we asked the question of what were the best values possible. To answer this question, we used the same methodology above and ran the 10 entries over a broad range of parameters to narrow in on what appeared to be an optimal region. Our initial run varied all parameters from -16 to +16 using increments of powers of 2. We found that the strategy of weighting heavily on seed and tweaking the other parameters was in fact a good strategy. We set this value to 8 and varied the other parameters around 0 to try and find a near to optimal solution.

Based on the results we refined the optimum parameters down to:

W = 0 to 1
 X = -3 to -2
 Y = 2 to 3
 Z = 8
 K = 6 to 8

The combinations that produced the top mean scores are shown below. As the results show, we were able to make improvements on Dean’s entry. Since these results are very dependent on the specific outcome of this year’s tournament, there is no guarantee that they will be the best choices for next year’s tournament.

Thanks to everyone who entered. We’re already thinking ahead to next year, where you may be asked to pick the winning formula of geography, mascot, and school colors to predict the tournament’s winner.