« 01.07.2010 - 31.07.2010 | Main | 01.05.2010 - 31.05.2010 »

28/06/2010

The Black Swan: preliminary thoughts

I had some time off recently and had a chance to read Nassim Taleb's book The Black Swan. Every decade or so I have my weltanschauung radically rearranged by a particularly profound book; Norretranders The User Illusion in 2000 and Gleick's Chaos in 1989 (which lead to the book From Clocks to Chaos by Glass and Mackey which almost made me quit medicine for research again). It's interesting that these major changes happen about once/decade which is also roughly the interval between my making major life alterations.

Black swans are the totally unexpected, unpredictable events that occur a lot more frequently than people think. The name comes from the assertion, prior to humans setting foot in Australia, that all swans are white. Looking back at my life I find that every major change was the result of a black swan event (the majority positive). Black swans are the "unknown unknowns" that complicate ones life and, on a personal level can be the unexpected heart attack or stroke or winning a lottery and on a social scale the collapse of the world financial system (a work in progress), near total disruption of air travel to Europe by a volcano in Iceland and the use of hijacked airliners to attack the world trade center on 11/9/2001. Curiously (not so curious after you read the book), it's the "experts" who have been the least likely to predict such events.

Humans live under the delusion that their lives are orderly and predictable. This illusion is created by the reality generating apparatus that creates our percieved reality (not something that Taleb dealt with in detail but the primary subject of Norretranders book). This reality generating apparatus has a number of design faults which result in it ignoring the possibility of black swans or minimizing their impact. This results in a number of fallacies which Taleb explores in great detail and it finally made me aware of why most physicians are stuck on linear relationships between variables -- this seems to be built into the human brain. I worked with non-linear systems during my research days so, to me, non-linear relationships are obvious but now I understand why I get so frustrated with much of the medical literature.

One of the key concepts in Taleb's book is the division of events as either belonging to "Mediocristan", the nicely behaved (from a statistical viewpoint) part of the world where conventional statistics works and measured variables fall on a Gaussian probability distribution (bell curve). His example is that of human heights which fall into a very narrow range and the probability of finding a 12' human is so close to zero that it is zero for practical purposes. Then there is "Extremistan" the dwelling place of the black swans. Events from extremistan can't be described by Gaussian statistics and are things like number of books that an author will sell and one that is known to most people is the Harry Potter series of novels which were the creation of the previously unknown author J. K. Rowling. Coming from seemingly nowhere, the Harry Potter series of books had sold some 400 million copies by June 2008. Considering that a book which sells 100,000 copies is considered a best seller the Harry Potter phenomenon clearly belongs to extremistan.

I realize the section above is a crappy review of the book and plan on revising it RSN. What struck me about The Black Swan was that I was familiar with many of the underlying concepts but had never seen them laid out in this fashion. It's a book that I plan on re-reading in the very near future to nail down the key points. It's already influenced my thinking quite a bit especially in the area of climate science.

Taleb destroys complicated financial models as they are useless and haven't predicted a single recession over the last 30 years. When one reads the book, it is clear that the analytic approach used is totally flawed and increasing the speed of computers used to run the models or the amount of information they start with won't make a bit of difference. Black swans are outside of the reality of the models and can't be predicted; all that can be done is to prepare for them and try to mitigate the effects when they do occur.

This situation is uncannily close to the highly detailed climate models that are said to predict global catastrophe unless we all give up industrial civilization and go back to living in a primitive fashion that doesn't result in release of CO2 into the atmosphere. I'm sorry to say that physicians appear to have been particularly gullible in this regard with major Canadian medical organizations coming out in support of processes which would destroy modern medicine as we know it if the drastic CO2 lowering called for by various watermelon groups (green on the outside, red on the inside) was to take place. The distortion of science as a result of the delusion of anthropogenic global warming (AGW) may be severe enough to destroy science. Fortunately, there are many voices of sanity on the internet with Watts Up With That (WUWT) being one of the most popular blogs pointing out that the AGW emperor has no clothes.

The earth's climate (considered over a period of centuries) belongs to extremistan, not mediocristan but the global climate models (GCM) are firmly rooted in mediocristan. They are less than useless and it will be interesting to see how future generations view this particular form of mass insanity that affected the world from 1990-2010. The only benefit that the AGW movement has given the rest of humanity is the insight that the earth's climate is a lot less stable than we think but we're not the ones who are making the changes. The thinking exhibited by the AGW proponents is isomorphic with the anthropocentric view pre-Galileo that the earth must be the center of the universe. I wish we had the ability to control the earth's climate but that is a long long way off.

One of the latest attempts by the AGW forces was the publication of a paper in Proceedings of the National Academy of Sciences (PNAS) which tried to discredit scientists who don't believe in the preposterous claims of AGW by stating that 97% of climate scientists are believers in AGW and have a better publication record than the "disbelievers". Science is not a matter of consensus, it's about whose theory has predictive value and is resistent to attack. The paper in PNAS is a sign of desperation among the AGW adherents. I haven't read the PNAS paper as I can't get access to it online but the story was covered on WUWT at this link and there was a "hit list" of "disbelievers" at another site which I can't find the link for now. The posting I made is given below and I noticed as I was writing it that I was suddenly using concepts from The Black Swan. What follows is a corrected version of what I posted at WUWT.


That the PNAS has seen fit to publish a paper in which a central premise of a paper is the fact that 97% of scientists support the theory of AGW is a sad day for science. I don't care if 99% of scientists support a particular viewpoint; it just takes one scientist who demonstrates that the theory is false to produce a paradigm shift. There are numerous historical examples of where this was the case: 150 years ago 99% of physicians would have likely disbelieved the germ theory and it took the work of Semmelweis and Pasteur to finally convince the medical profession that they were on the wrong track. To make matters more embarrasing, Pasteur was not a physician.

Of course that is in the field of medicine which some can argue isnt a true science as consensus positions in many subspecialties are the norm (especially psychiatry). In the field of physical sciences one would expect true science to reign. AGW is a very sloppy theory and seems to be setup so it is non-falsifiable; IMHO Steve McIntyre has falsified AGW by his debunking of the hockey stick graph. AGW is such an amorphous theory that it seems anything that happens fits the theory; we have a colder than average winter and that is part of global warming. It is hotter than usual in the summer again global warming. It rains more, again global warming but then droughts are also forecast by global warming.

What has become clear is that most of climate science has become very unscientific. True science is open to anyone who is intelligent enough to take the basic premises of a branch of science, play with them and come up with novel testable theories. The sign of a non-scientist is the denigration of an individuals credentials when they come up with unexpected testable conclusions from data that "true scientists" have not. Steve McIntyre seems to frequently be attacked on this basis as his methodology for destroying one of the essential foundations of AGW can't be disproved by the "experts" in this area and Steve McIntyre is not an academic. Many of the novel discoveries in science have been made by non-experts or people with expertise in other fields as they are thinking out of the box whereas experts have great knowledge in a tiny area of science and no longer think much beyond their area of expertise.

I dont care if the person who comes up with the right answer is a cab driver, plumber or solar physicist; the main thing is if theyre right or not. We've seen failure after failure of AGW predictions and an ever increasing nebulous theory that now seems indistinguishable from a pseudo-scientific justification for unbridled statism.

From what I've seen, the well funded AGW establishment is unlikely to produce anything of scientific significance but is capable of producing immense harm. One has the situation where there are a small group of like-thinking "experts" who are the recipients of the funding and also serve as peer-reviewers of what are considered to be the most influential journals in the field (and thus determine which papers get published). These "experts" have graduate students who, through the process of confirmation bias and often not so subtle pressure from their thesis supervisor, reject the outliers they find in their thesis projects and come up with yet more support for the pet theory of their supervisor. This technique is fine if one is dealing with phenomena from mediocristan (the realm where normal distributions apply and conventional statistics works) but totally fails in extremistan which is the home of black swans, or unknown unknowns and also the regime corresponding to earths climate. We have been total failures in modelling the far more constrained financial systems with numeric models even with incredibly fine-grained data. The idea that one can use the same models (but faster computers) to model the highly chaotic climate system of the earth for which we know only imperfectly a few of the controlling parameters is the height of hubris (or insanity).

Throwing money at a large collection of climate "experts" would be analagous to creating a multiprocessor system to solve a problem in which every processor solves the identical problem with just minor differences. WUWT is the equivalent of a multiprocessor system in which the highly varied components solve differing problems in the realm of climate science and such a collection of individuals is far more likely to solve some of the difficult problems in this realm (or more importantly poke holes in poor theories) than the well funded multiprocessor system that is only working on a very tiny portion of the problem. This type of democratic science is very messy but far more likely to come up with solutions to climate science than the monolithic rigid science which is now indistinguishable from government. Democracy and capitalism are very messy multiprocessor parallel processing networks for solving social and economic problems but they are orders of magnitude better than central control solutions.

What I've found is that every time I change fields I come up with my most significant insights in the new field right at the beginning while I'm learning it; I'm glad I kept all my writings when I first got into medical school as they make me realize how myopic I've become about medicine now that I do medicine full time. To try to stay out of the expert trap I try to do some form of major career change every 10 years or so.