« Blog related | Main | Computers »

16/11/2010

Demolishing "scientific concensus"

Ran into a posting on WUWT by Geraldo Luis Lino which I would have to say is the perfect answer to those who view "scientific consensus" as being of importance in AGW. Science doesn't operate via concensus, it operates on the basis of which theory fits the facts the best. As we come up to the 1 year anniversary of climategate, it is encouraging to see that the whole AGW house of cards is collapsing and, hopefully, with it will end the wet dreams of statists hoping to create a totalitarian state on this planet by demonizing CO2.

We've seen the demise of the Chicago climate exchange which disappeared once the price of CO2 dropped below $0.05/ton. This market was supposed to be worth more that $1 trillion/year but cap and trade laws didn't get passed and it's history. The French have killed the "super ministry for the environment" which was to be in charge of "sustainable development" (meaning that it would extort money from the general public for transfer to watermelon criminals). This is good news and one wonders about WTF BC has a carbon tax which is the legacy of junk science and unbridled statism.

I'd forgotten about the "100 authors against Einstein" in 1931 as the German National Socialists used the full power of scientific concensus to reject "Jewish science" as embodied in the theory of relativity. The Germans never did find out how wrong scientific concensus was in this case, but the Japanese did experience the fruits of "Jewish science" at Hiroshima and Nagasaki.

While it's nice to know that one is right, there can be no letup of the fight against the watermelons who created this scientific hoax of the millenium. It's now time for the fraud trials to begin and hopefully the totalitarians that were engaged in this criminal enterprise will begin serving very long jail terms soon.

Posted by Boris Gimbarzevsky at 2:17.56
Edited on: 16/11/2010 2:46.40
Categories: Climatology

30/07/2010

Carbon taxes - made in BC idiocy

Had to fuel up the jeep today and while fuming about the waste of time that is required to pay before pumping gasoline (an idiotic workplace regulation arbitrarily imposed by unelected unaccountable statist bureacrats) noticed that every liter of gasoline now has 4.54 cents of carbon taxes included. Of course this $0.0454 is multiplied by 1.12 for the Victoria statists to gouge yet more money from people. Then there is that same "carbon tax" imposed on natural gas which goes up yearly nicely coincident with global cooling which will enrich the Victoria kleptocrats even more and taxes on electricity to fund "sustainable" electricity generation projects. We didn't stop using windmills because the world ran out of wind; nothing shows the true agenda of the watermelons who dispoil the natural environment with massive noisy wind turbines that do quite a good job of slicing up large numbers of birds. These are the same moonbats who go after oil companies for a few dead birds on tar sands ponds whereas the carnage of windmills gets swept under the rug. I guess in the warped watermelon reality chopping birds into pieces is "sustainable" and "natural". And, BTW, wind power does a great job in destabilizing the electrical grid and the wind don't blow when it's really cold. I guess people freezing to death because of no "sustainable energy" is natural too. What I want is a nice clean nuclear reactor nearby as that waste heat is going to come in very handy as world temperatures continue to decline.

Given the revalations of climategate it is hard to believe that any politician in this country could truthfull say that AGW scam is scientifically based. What is real is the insatiable appetite for statists for money and, they are currently taking advantage of "the environment" in imposing tax after tax on things. I've botten totally fed up with this process and, if the Wild Rose Alliance wins the next Alberta provincial election, I'll be moving to Alberta.

Lying Gordon Campbell (head moonbat in BC) promised he wouldn't bring in the HST, but we know how much politicians promises are worth. Now I guess it's into the recall phase of the anti-HST campaign and as soon as I find out where to sign the petition to recall the local lieberal idiot MP's, I'll be supporting this move. I want to see the BC lieberal party utterly destroyed as they represent the worst possible mix of statism, watermelon ideology, crony capitalism and stupidity that one is going to find anywere except Ontario. BC politics seems to consist of governments of either the BC communist party or a party that just wants to keep the commies out of power. What we really need now in BC is a TEA party.

/rant

28/06/2010

The Black Swan: preliminary thoughts

I had some time off recently and had a chance to read Nassim Taleb's book The Black Swan. Every decade or so I have my weltanschauung radically rearranged by a particularly profound book; Norretranders The User Illusion in 2000 and Gleick's Chaos in 1989 (which lead to the book From Clocks to Chaos by Glass and Mackey which almost made me quit medicine for research again). It's interesting that these major changes happen about once/decade which is also roughly the interval between my making major life alterations.

Black swans are the totally unexpected, unpredictable events that occur a lot more frequently than people think. The name comes from the assertion, prior to humans setting foot in Australia, that all swans are white. Looking back at my life I find that every major change was the result of a black swan event (the majority positive). Black swans are the "unknown unknowns" that complicate ones life and, on a personal level can be the unexpected heart attack or stroke or winning a lottery and on a social scale the collapse of the world financial system (a work in progress), near total disruption of air travel to Europe by a volcano in Iceland and the use of hijacked airliners to attack the world trade center on 11/9/2001. Curiously (not so curious after you read the book), it's the "experts" who have been the least likely to predict such events.

Humans live under the delusion that their lives are orderly and predictable. This illusion is created by the reality generating apparatus that creates our percieved reality (not something that Taleb dealt with in detail but the primary subject of Norretranders book). This reality generating apparatus has a number of design faults which result in it ignoring the possibility of black swans or minimizing their impact. This results in a number of fallacies which Taleb explores in great detail and it finally made me aware of why most physicians are stuck on linear relationships between variables -- this seems to be built into the human brain. I worked with non-linear systems during my research days so, to me, non-linear relationships are obvious but now I understand why I get so frustrated with much of the medical literature.

One of the key concepts in Taleb's book is the division of events as either belonging to "Mediocristan", the nicely behaved (from a statistical viewpoint) part of the world where conventional statistics works and measured variables fall on a Gaussian probability distribution (bell curve). His example is that of human heights which fall into a very narrow range and the probability of finding a 12' human is so close to zero that it is zero for practical purposes. Then there is "Extremistan" the dwelling place of the black swans. Events from extremistan can't be described by Gaussian statistics and are things like number of books that an author will sell and one that is known to most people is the Harry Potter series of novels which were the creation of the previously unknown author J. K. Rowling. Coming from seemingly nowhere, the Harry Potter series of books had sold some 400 million copies by June 2008. Considering that a book which sells 100,000 copies is considered a best seller the Harry Potter phenomenon clearly belongs to extremistan.

I realize the section above is a crappy review of the book and plan on revising it RSN. What struck me about The Black Swan was that I was familiar with many of the underlying concepts but had never seen them laid out in this fashion. It's a book that I plan on re-reading in the very near future to nail down the key points. It's already influenced my thinking quite a bit especially in the area of climate science.

Taleb destroys complicated financial models as they are useless and haven't predicted a single recession over the last 30 years. When one reads the book, it is clear that the analytic approach used is totally flawed and increasing the speed of computers used to run the models or the amount of information they start with won't make a bit of difference. Black swans are outside of the reality of the models and can't be predicted; all that can be done is to prepare for them and try to mitigate the effects when they do occur.

This situation is uncannily close to the highly detailed climate models that are said to predict global catastrophe unless we all give up industrial civilization and go back to living in a primitive fashion that doesn't result in release of CO2 into the atmosphere. I'm sorry to say that physicians appear to have been particularly gullible in this regard with major Canadian medical organizations coming out in support of processes which would destroy modern medicine as we know it if the drastic CO2 lowering called for by various watermelon groups (green on the outside, red on the inside) was to take place. The distortion of science as a result of the delusion of anthropogenic global warming (AGW) may be severe enough to destroy science. Fortunately, there are many voices of sanity on the internet with Watts Up With That (WUWT) being one of the most popular blogs pointing out that the AGW emperor has no clothes.

The earth's climate (considered over a period of centuries) belongs to extremistan, not mediocristan but the global climate models (GCM) are firmly rooted in mediocristan. They are less than useless and it will be interesting to see how future generations view this particular form of mass insanity that affected the world from 1990-2010. The only benefit that the AGW movement has given the rest of humanity is the insight that the earth's climate is a lot less stable than we think but we're not the ones who are making the changes. The thinking exhibited by the AGW proponents is isomorphic with the anthropocentric view pre-Galileo that the earth must be the center of the universe. I wish we had the ability to control the earth's climate but that is a long long way off.

One of the latest attempts by the AGW forces was the publication of a paper in Proceedings of the National Academy of Sciences (PNAS) which tried to discredit scientists who don't believe in the preposterous claims of AGW by stating that 97% of climate scientists are believers in AGW and have a better publication record than the "disbelievers". Science is not a matter of consensus, it's about whose theory has predictive value and is resistent to attack. The paper in PNAS is a sign of desperation among the AGW adherents. I haven't read the PNAS paper as I can't get access to it online but the story was covered on WUWT at this link and there was a "hit list" of "disbelievers" at another site which I can't find the link for now. The posting I made is given below and I noticed as I was writing it that I was suddenly using concepts from The Black Swan. What follows is a corrected version of what I posted at WUWT.


That the PNAS has seen fit to publish a paper in which a central premise of a paper is the fact that 97% of scientists support the theory of AGW is a sad day for science. I don't care if 99% of scientists support a particular viewpoint; it just takes one scientist who demonstrates that the theory is false to produce a paradigm shift. There are numerous historical examples of where this was the case: 150 years ago 99% of physicians would have likely disbelieved the germ theory and it took the work of Semmelweis and Pasteur to finally convince the medical profession that they were on the wrong track. To make matters more embarrasing, Pasteur was not a physician.

Of course that is in the field of medicine which some can argue isnt a true science as consensus positions in many subspecialties are the norm (especially psychiatry). In the field of physical sciences one would expect true science to reign. AGW is a very sloppy theory and seems to be setup so it is non-falsifiable; IMHO Steve McIntyre has falsified AGW by his debunking of the hockey stick graph. AGW is such an amorphous theory that it seems anything that happens fits the theory; we have a colder than average winter and that is part of global warming. It is hotter than usual in the summer again global warming. It rains more, again global warming but then droughts are also forecast by global warming.

What has become clear is that most of climate science has become very unscientific. True science is open to anyone who is intelligent enough to take the basic premises of a branch of science, play with them and come up with novel testable theories. The sign of a non-scientist is the denigration of an individuals credentials when they come up with unexpected testable conclusions from data that "true scientists" have not. Steve McIntyre seems to frequently be attacked on this basis as his methodology for destroying one of the essential foundations of AGW can't be disproved by the "experts" in this area and Steve McIntyre is not an academic. Many of the novel discoveries in science have been made by non-experts or people with expertise in other fields as they are thinking out of the box whereas experts have great knowledge in a tiny area of science and no longer think much beyond their area of expertise.

I dont care if the person who comes up with the right answer is a cab driver, plumber or solar physicist; the main thing is if theyre right or not. We've seen failure after failure of AGW predictions and an ever increasing nebulous theory that now seems indistinguishable from a pseudo-scientific justification for unbridled statism.

From what I've seen, the well funded AGW establishment is unlikely to produce anything of scientific significance but is capable of producing immense harm. One has the situation where there are a small group of like-thinking "experts" who are the recipients of the funding and also serve as peer-reviewers of what are considered to be the most influential journals in the field (and thus determine which papers get published). These "experts" have graduate students who, through the process of confirmation bias and often not so subtle pressure from their thesis supervisor, reject the outliers they find in their thesis projects and come up with yet more support for the pet theory of their supervisor. This technique is fine if one is dealing with phenomena from mediocristan (the realm where normal distributions apply and conventional statistics works) but totally fails in extremistan which is the home of black swans, or unknown unknowns and also the regime corresponding to earths climate. We have been total failures in modelling the far more constrained financial systems with numeric models even with incredibly fine-grained data. The idea that one can use the same models (but faster computers) to model the highly chaotic climate system of the earth for which we know only imperfectly a few of the controlling parameters is the height of hubris (or insanity).

Throwing money at a large collection of climate "experts" would be analagous to creating a multiprocessor system to solve a problem in which every processor solves the identical problem with just minor differences. WUWT is the equivalent of a multiprocessor system in which the highly varied components solve differing problems in the realm of climate science and such a collection of individuals is far more likely to solve some of the difficult problems in this realm (or more importantly poke holes in poor theories) than the well funded multiprocessor system that is only working on a very tiny portion of the problem. This type of democratic science is very messy but far more likely to come up with solutions to climate science than the monolithic rigid science which is now indistinguishable from government. Democracy and capitalism are very messy multiprocessor parallel processing networks for solving social and economic problems but they are orders of magnitude better than central control solutions.

What I've found is that every time I change fields I come up with my most significant insights in the new field right at the beginning while I'm learning it; I'm glad I kept all my writings when I first got into medical school as they make me realize how myopic I've become about medicine now that I do medicine full time. To try to stay out of the expert trap I try to do some form of major career change every 10 years or so.

25/04/2010

Accessing Canadian temperature data

One of the things that has come out of climategate is the huge amount of data manipulation that has taken place in order to make regional temperature data fit with the AGW model. This has been demonstrated numerous times on WUWT. Canadian climate data is available on the National Climate Data and Information Archive. If one goes there, temperature data from thousands of weather stations is available for ones perusal. This is very nice except when one attempts to download large chunks of data. The site works fine for the first few downloads (done through the bulk-data option) and then seems to hang. I've tried multiple different web browsers and different IP addresses and found the same problem and it appears that this behavior is deliberate and designed to prevent people from downloading more than a few months of data at a time.

In order to look for trends and cycles in raw climate data, one needs to use as large a date range as possible. Thus far the longest temperature record I've found in the Canadian data thus far is for Edmonton which goes back to 1880 (I'm sure there are earlier records but I just haven't explored the site enough yet). The temperature data at this site appears to be uncorrected and there is some urgency in downloading it all before it becomes "homogenized" and "adjusted". Thus, in December of 2009, when the nasty little adjustments being made to regional temperature data started to surface, I wrote a quick and dirty program, climate_scraper to download data in variious formats from the National Climate Data and Information Archive.

Curiously, if one copied the query strings from this site into a browser and manually changed the date and station parameters, there didn't seem to be any throttling of the data flow from the site. All that my climate_scraper program does is to automate this process and one gives it the station ID# and year range that one wants data for as well as the type of data desired. Right now the only options are for hourly data and daily data. The climate data server is rather simple minded and, when one gives it a year for which it has no data, it will return a file of dates with comma seperated null strings. Needless to say, a file of hourly data for an invalid date is just a waste of time and space so make sure you have the right date range when you use the program.

Climate_scraper is written in VB6 (the best version of VB the M$ put out before they went to the incompatible VB.NET) and the download, available on my website, contains both an executable file as well as full source code. I'm releasing this code under GPL v3 which basically means it is yours to do with as you please except, if you create a derivate work based on my code you have to indicate that you made the changes and release all your code with the executable file.

What motivated me to finish up this program was the WUWT posting about possible errors in the Eureka, Nunavut weather station data for July 2009. This station is used by GISS to calculate temperatures for essentially the whole of N. America north of this station. Fortunately, the manager of the Eureka station has posted explanations on WUWT which has been very helpful in sorting out what's going on with the temperature at this location. Such willingness to share information with WUWT is unusual in the scientific climate community and the manager, Rai LeCotey, is to be commended for his openness. When I read the first posting regarding the Eureka station, I was motivated to write the code to download hourly data from any station that has this data. (This is the h option for DataType).

I've been hesitant to release this program for general use primarily because of my concern that if everyone decides to make their own copy of the Canadian temperature data the increased server load will be noticed and this means of access would be terminated. It would be impossible to close off entirely without rewriting all of the javascript code to access data but I'm sure that it would be easy enough to restrict the number of downloads an individual user was allowed to do during a given time period. For this reason, I would ask people to limit themselves to looking at their local area and thinking about setting up a mirror server that has the whole dataset. This isn't going to be me as I don't have the time, or anything close to the bandwidth required for such a project. The data format for temperature data is an incredibly inefficient one and here is a line of temperature data from the Eureka station:

"2009-7-14 19:00","2009","7","14","19:00","14.40","","4.10","","50.00","","36.00","","22.00","","64.40","","101.87","","","","","","Mainly Clear"

This, incidentally, was the temperature which started the whole post at WUWT. While this data format might not be a concern on someones local 1 Tb disk, it is grossly inefficient for internet transfers. Note that the date appears twice and 41 bytes are used to hold the date which, even if one uses M$'s bloated VB Date variable, only 8 bytes would be required. 19 variables then follow and each of them could be represented in 2 bytes as a signed (or unsigned in some cases) integer variable which would give a binary representation requiring 46 bytes/line instead of the 145 bytes that are currently required (may be off by a few bytes as I counted them manually). It appears that this is a standard format for representing temperature data but when I have a chance to start playing with this data, there is no way that I'm going to store it in this format on my computers

One problem with data conversion is that errors may creep in and this is what we are trying to avoid. One possible way of flagging errors would be to create a pseudo-checksum of all of the numeric values in the ascii format temperature reading and if converting all of the binary variables to text results in something different then clearly there is a problem. I'll leave this problem for whoever takes on the task of coming up with an open-source world temperature data archive.

Going through this data by hand is a gargantuan task, but with the average desktop machine being significantly more powerful than the supercomputers of even 20 years ago, automatic analysis of huge amounts of data is possible in short amounts of time. The main things we're interested in are temperature trends and correlations between geographically close stations as well as temperature outliers which might be errors or may be the result of the numerous artifacts that plague weather stations in urban areas that have been looked at in Anthony Watts surface stations project. This project demonstrate the power of thousands of interested amateurs to exhaustively document the potential temperature errors in every surface weather station in the US. Analysis of temperature data is something which any amateur scientist armed with a computer and and often self-written software can do and potentially make valuable contributions to climate science. Open-source code has been described as peer-reviewed code and the contributions of extended peer-review by amateur climatologists to this area have been embarrasing to the inbred climatologic establishment. Steve McIntyre has done outstanding work, WUWT has been a hotbed for dissemination of independent climate information and E.M. Smith has audited GISS FORTRAN code finding major bugs as well as doing an impressive amount of other climatologic work. I'm happy to make my minor contribution to the process to find climate truth.

Unless someone with more time than me decides to take on the Kamloops temperature data, I plan on analyzing this data when RSN. TIme is one thing that is in very short supply for me.

One warning about the program is that it does run fairly slowly. Data transfer from the internet control returned data is done a byte at a time primarily because I didn't want to write the code to move chunks of data at once instead; I had my code working inside of an hour and I was more interested in looking at the climate data than I was in coming up with the ideal download program. Also, be warned that the VB6 code is not very nice as I used the first thing that worked rather than striving for elegant code. I don't normally program like that and was inspired by some of the climategate FORTRAN code from FOI2009.zip. My code is at least as good as that FORTRAN code and better documented.

Now that you've read through my verbose and rambling description of the program (no time to write a shorter entry) here is where you can get the code. Right now it is only available as a 7zip format file and one can download 7zip at this link.

25/4/2010 T:=14:04

Sorry for the error in the download link; Thingamablog used "downloads.html" instead of "download" resulting in a 404 error for anyone who tried this. Puzzling as the program has been quite good until now. Lack of appropriate testing on my part.

Posted by Boris Gimbarzevsky at 2:23.15
Edited on: 25/04/2010 13:54.58
Categories: Climatology, Computers

27/03/2010

Human Achievement Hour

Today is the day that AGW believing moonbats expect people to sit in the dark for an hour in the so-called "earth hour". That this group of morons expect people to give up an hour of their time to destabilize the electrical grid is asinine. If they are so enamored of such primitive lifestyles, then why don't they move to North Korea where every day is earth hour?

During past "earth hours" I've used the time productively to ensure that my house wiring was working properly and to test out a number of high current appliences. It's also a good time to ensure that every light bulb in the house is working and see if they will survive an hour of continuous use.

Now we have an official theme for this hour and it is Human Achievement Hour. The widespread distribution of electrical power is one of humanity's great achievements and we should celebrate it periodically. For those who want to sit in dark unheated hovels eating locally grown grass go right ahead but don't expect me to join you.

Posted by Boris Gimbarzevsky at 10:51.13
Edited on: 27/03/2010 18:38.30
Categories: Climatology, Junk science