The Signal and Noise by Nate Silver

Nate Silver has become a fairly important person in certain circles recently. As a statistician Silver devised a system for measuring performance in baseball. Then in 2008 he earned more praise when he nailed the vote in the Presidential election. During this election series his Political Forecasting Blog Five Thirty Eight has been syndicated by The New York Times.

For my own sanity and as an Obama supporter I have not paid much attention to all of the polls that are released each day and instead only follow Silver's blog which takes a great deal of information, including those same polls, and comes up with a likelihood of an Obama victory in both the popular vote and the electoral college.

In this heavily buzzed book Silver tackles the subject and science, if you will, of prediction and forecasting. Chapters dedicated to weather, sports, poker, global science, and others show examples of the use of probability theory to make accurate forecasts.

The book is fascinating. As Silver talks about the weather he praises the gains made in Weather prediction. He also points out what most of us know, that local weather forecasts are a joke, and that the best forecasts come from the sources that are less profit driven. The National Weather Service, who has no agenda other than being correct, is much more clairvoyant if you will than even The Weather Channel which is multiples more accurate than your local forecaster.

Silver talks about the fact that many forecasters have an agenda that makes their accuracy much lower. Talking heads on Fox or MSNBC are perfect examples of this, the forecasts on these channels in almost all cases match the agenda of the networks themselves.

There is so much information in this book that it cannot all be summarized but the highlights for me include an explanation of Bayes Theorem. This methodology, a simple algebraic equation according to Silver, can be the best tool to make any prediction. It talks about Prior Probabilities and Posterior Probabilities and it works beyond compare. It is not, to me, a simple algebraic equation but with slow thought I was able, with my wife's help to make it work. As she said she was not Captain of the Math team for nothing. Probabilistic theory is the best way to make predictions because it recognizes that no one can be truly objective, in any prediction there is some level of subjectivity. With Prior Probabilities this is taken into account.

As an aside I realized that I used this theory without knowing it in my fantasy baseball leagues. I also realized that not everyone does. Prior opinion, matched and compared against current results can bring a great deal of knowledge.

Another methodology called Power Theory is fascinating. Originally coming to the fore in earthquake prediction, a science that is beyond difficult, power theory offers some significant measure of prediction can be found. We are not close to being able to predict earthquakes far in advance, but we can get a general sense of the likelihood of catastrophic earthquakes in any given area. Late in the book we also see that in the methodology of predicting terror events the same theory can be used. When one thinks of this, it cannot be rational, but the charts and graphs shown prove that indeed, what power theory shows us, is that just because something has not happened, power theory can show us that it is indeed possible and in some cases, likely.

The last illustration I want to talk about is a particular illustration of Baye's Theorem that applies to false positives in the mammograms of younger women. While not being for me enough to increase the age of mammogram recommendation, a false positive can be proven wrong while not catching cancer cannot be corrected, it does show in a clear way the use of Baye's Theory in the real world.

This is a fascinating book, full of great information. It is not a skimmer, one has to pay attention and in my case at least some real thought is required. It is well worth it.