« Climatology | Main | Junk science »

23/01/2011

DPlot program

Every once in a while I will buy some software and the DPlot program that I purchased about a month ago is one of the best pieces of software that I've come across in decades. It's well worth the $195 US price tag and comes with very reasonable license terms -- install it on as many of your computers as you like if you're the only one using it. It's available for online purchase at DPlot.com and there is a free trial version.

I spend a lot of time crunching data from my various personal research projects and the greatest bottleneck that I've run into is creating graphs from the data. What I've been using, when I need a nice graph, is CricketGraph for the Mac. To run this program I have to launch my Mac emulatorBasiliskII, get the data transferred to a "disk" in the emulator, run CricketGraph, paste the resultant PICT file into M$ Word for the Mac and then export it as an rtf file which then needs to be copied from the Mac emulator "disk" drive and pasted into Open office. Needless to say, this is an incredibly tedious way to creating graphs and one of the reasons I'm constantly writing crude data display programs which are fine to look at data but leave much to be desired aesthetically.

My first attempt to deal with this problem was to use the MSChart control which comes with VB. For anyone who's thinking of doing this my advice is DON'T!!!! MSChart is an incredibly complex control which, after one has learned the arcane details of how to program it, doesn't allow you to print what you've created on the screen!!! This control is one of the worst POS that I've encountered from M$.

Anthony Watts put up a rave review about DPlot on WUWT in December 2010 and I was intrigued enough to download the demo version. Once I started using it I realized that I finally had something almost as good as CricketGraph. I say almost as good as one can't shift data up and copy it to another column to create Poincare plots like CricketGraph can do, but OTOH, DPlot will do FFT's on data and produces incredibly nice looking graphs. Now I can spend my time writing fast programs to manipulate data and, while I haven't tried it yet, DPlot is scriptable from VB. DPlot easily handled a 24 hour 3-axis accelerometry graph with over 3 million points which I found very impressive.

One application of DPlot I did today was to calibrate the USB temperature monitors which I bought from the Weather Shop (link at WUWT). These are really neat units that sample temperatures from once every 10 seconds to every hour or so up to 16383 readings. 2 of the 3 original monitors I bought 2 years ago still work (the onlly complaint I have about them is that they use a very hard to find 3.6 V Li battery)

Last night did a calibration of monitors 2 and 3 and results are quite impressive. Monitor 3 has some electrical tape and wire around the rear end and the effects of this on the monitor thermal parameters are seen during calibration process!

The monitors were both restarted after download of data and, after a short period at downstairs room temperature, were placed in downstairs fridge freezer for about 1 hour. Sampling frequency was 1/minute. After being removed from freezer, I had the bright idea of putting them on Athena keyboard as this is above room temperature. This was a bad idea as the keyboard is not at a uniform temperature as there was 4 F degree difference between the two monitors when they were read out. After spending about 5 hours on keyboard, they were placed on my table behind futon which is not heated by any computers. They were left there all night.

Since the keyboard was a non-uniform temperature, this section of data was deleted from the data finally used for calibration. There was hysteresis noted when USB3 was plotted against USB2 and will have to re-run the calibration to see if this was due to the electrical tape on USB3. There is some deviation at the high end of the range, but the least squares fit equation was:

USB3 = 0.998*USB2 + 0.321

Correlation coefficient was 0.9989 for R^2 of 0.9978

The two monitors are highly correlated as expected but I find it hard to believe how close the two units are to each other. I would have been ecstatic if I'd had a calibration curve this good when I was building my own DAS's in 1982. For all practical purposes, the temperatures are identical.

Standard error about the line is 0.558 so every temperature reading is +/- 0.56 F degrees. This is about what I'd expect from a thermometer with 1 degree F resolution. Considering that each unit was only $60 US this is very impressive and demonstrates how good off the shelf electronics have become in the last 30 years.

Calibration temperature range was from 13 to 84 F degrees and should try to get to 120 F to see if there are any non-linearities at the extremes.

The hysteresis indicates that the thermal time constant of the thermometers is >1 minute and have to keep this in mind when using them to monitor temperatures. There's not much point in using the 10 second resolution of the thermometer except for rapidly varying temperatures (like hanging from air-conditioner vent) where one doesn't get an absolute temperature but rather a partially integrated temperature. One would need a much faster thermal time constant to sample this, eg my thermocouple on DMM.

And here's the calibration graph described above.

 

Posted by Boris Gimbarzevsky at 23:34.07
Edited on: 24/01/2011 1:03.37
Categories: Computers

30/05/2010

Moronic Sony Keyboard

At the end of last year I indulged myself in a new toy, the Sony Vaio touchscreen computer which has a very impressive display, 4 Gb of RAM, a 3 GHz dual core 64 bit processor and has been a great machine in all respects except that the keyboard supplied with the unit is an absolute POS. (It also runs windoze 7 which takes weeks to clean up but I've blogged about that earlier).

Having been typing stuff into computers for the last 40 years or so, I'm quite particular about my keyboards. Of the keyboards that I've used, IBM makes by far the best keyboards and they are almost indestructible. Given how much of a difference in typing speed a good keyboard makes, I don't mind shelling out top dollar for a keyboard that feels very natural to use. I hadn't really thought about keyboards very much as every machine I've used until recently has had a good keyboard. HP makes nice keyboards that don't last as long as the IBM indestructables and I can adapt to the cramped keyboard of the TC1000 and then find it enjoyable to have a bit of room for my fingers on the TC4400, my current tablet PC.

The Sony Vaio was to be my primary work machine as it is so nice to have a 1920x1080 screen in front of ones eyes (actually my work area has 3 monitors but the Vaio is the highest resolution). I thought that this would be a great machine until I attempted to use the keyboard which is an absolute POS. To see why, look at the image below.

 

The red circle marks the problem. WTF is this idiotic key and WTF is it where the Shift key should be? What idiot decided that the Caps_Lock key should be oversized? Who even uses the Caps_Lock key? THIS IS ONE OF MY VERY RARE USES OF THE CAPS_LOCK KEY BUT I PREFER NOT TO YELL WHEN I POST.

On a good keyboard I can type very fast, it is enjoyable and the backspace key doesn't get much of a workout. On the VGP-WKB10 keyboard shown above, it seems that the BS key was the most hit key on the keyboard as my left 5th digit kept hitting the idiotic new key when I wanted to use the shift key. To make matters even worse, the Sony keyboard design department imbeciles decided to add another extra key where the Enter key left hand border should be. My character entry rate with the Sony keyboard was so slow that I resorted to typing on a proper keyboard on another machine via a VNC connection to the Vaio (Ultra VNC works just fine with W7). This is obviously not what I had planned on doing. I ended up wearing out my TC4400 even more as it has a very nice keyboard shown below.

Some 5 months after I got this machine, finally, I found a keyboard that I can use with it and can now think of creative ways of destroying the crap keyboard that shipped with this computer. I was hoping to have a wireless keyboard but ended up finding a USB keyboard (the Vaio no longer seems to have a standard keyboard connector included). The keyboard that I bought today is a Lexma LK-7300 and it has about the nicest touch that I've found in a keyboard. The TC100 felt similar but too cramped and the Lexma keyboard fits my hands quite well. Haven't timed my typing speeds on the Lexma keyboard yet but it would appear that they are about 3 times faster than the POS Sony keyboard. The only unknown with the Lexma keyboard is longevity; it is Chinese made and my experience with Chinese made goods is that they generally fall apart in a short period of time.

The "new and improved" keyboard configuration that Sony shipped with this machine has been appearing on other laptops and this is very concerning. I was going to buy another laptop to replace my aging TC4400 which has seen a great deal of use but every time I find a computer that I think would work, I see the idiotic keyboard style that Sony has used. I'm not about to retrain my motor system to reach further on the left side of the keyboard; I already have enough of a problem having to remove the Windoze key (which again seems to serve absolutely no usefull function) on keyboards from which it is easy to remove. If one could easiiy remap keys then this wouldn't be an issue but there appears to be no program out there which can remap keyboard scan codes the way that I want to. The M$ program which purports to do this just resulted in my wasting 2 hours of time and having nothing to show for it at the end. Time to go on Ebay and stockup on TC4400's while they're still available.

Posted by Boris Gimbarzevsky at 22:41.07
Edited on: 30/05/2010 23:01.48
Categories: Computers

25/04/2010

Accessing Canadian temperature data

One of the things that has come out of climategate is the huge amount of data manipulation that has taken place in order to make regional temperature data fit with the AGW model. This has been demonstrated numerous times on WUWT. Canadian climate data is available on the National Climate Data and Information Archive. If one goes there, temperature data from thousands of weather stations is available for ones perusal. This is very nice except when one attempts to download large chunks of data. The site works fine for the first few downloads (done through the bulk-data option) and then seems to hang. I've tried multiple different web browsers and different IP addresses and found the same problem and it appears that this behavior is deliberate and designed to prevent people from downloading more than a few months of data at a time.

In order to look for trends and cycles in raw climate data, one needs to use as large a date range as possible. Thus far the longest temperature record I've found in the Canadian data thus far is for Edmonton which goes back to 1880 (I'm sure there are earlier records but I just haven't explored the site enough yet). The temperature data at this site appears to be uncorrected and there is some urgency in downloading it all before it becomes "homogenized" and "adjusted". Thus, in December of 2009, when the nasty little adjustments being made to regional temperature data started to surface, I wrote a quick and dirty program, climate_scraper to download data in variious formats from the National Climate Data and Information Archive.

Curiously, if one copied the query strings from this site into a browser and manually changed the date and station parameters, there didn't seem to be any throttling of the data flow from the site. All that my climate_scraper program does is to automate this process and one gives it the station ID# and year range that one wants data for as well as the type of data desired. Right now the only options are for hourly data and daily data. The climate data server is rather simple minded and, when one gives it a year for which it has no data, it will return a file of dates with comma seperated null strings. Needless to say, a file of hourly data for an invalid date is just a waste of time and space so make sure you have the right date range when you use the program.

Climate_scraper is written in VB6 (the best version of VB the M$ put out before they went to the incompatible VB.NET) and the download, available on my website, contains both an executable file as well as full source code. I'm releasing this code under GPL v3 which basically means it is yours to do with as you please except, if you create a derivate work based on my code you have to indicate that you made the changes and release all your code with the executable file.

What motivated me to finish up this program was the WUWT posting about possible errors in the Eureka, Nunavut weather station data for July 2009. This station is used by GISS to calculate temperatures for essentially the whole of N. America north of this station. Fortunately, the manager of the Eureka station has posted explanations on WUWT which has been very helpful in sorting out what's going on with the temperature at this location. Such willingness to share information with WUWT is unusual in the scientific climate community and the manager, Rai LeCotey, is to be commended for his openness. When I read the first posting regarding the Eureka station, I was motivated to write the code to download hourly data from any station that has this data. (This is the h option for DataType).

I've been hesitant to release this program for general use primarily because of my concern that if everyone decides to make their own copy of the Canadian temperature data the increased server load will be noticed and this means of access would be terminated. It would be impossible to close off entirely without rewriting all of the javascript code to access data but I'm sure that it would be easy enough to restrict the number of downloads an individual user was allowed to do during a given time period. For this reason, I would ask people to limit themselves to looking at their local area and thinking about setting up a mirror server that has the whole dataset. This isn't going to be me as I don't have the time, or anything close to the bandwidth required for such a project. The data format for temperature data is an incredibly inefficient one and here is a line of temperature data from the Eureka station:

"2009-7-14 19:00","2009","7","14","19:00","14.40","","4.10","","50.00","","36.00","","22.00","","64.40","","101.87","","","","","","Mainly Clear"

This, incidentally, was the temperature which started the whole post at WUWT. While this data format might not be a concern on someones local 1 Tb disk, it is grossly inefficient for internet transfers. Note that the date appears twice and 41 bytes are used to hold the date which, even if one uses M$'s bloated VB Date variable, only 8 bytes would be required. 19 variables then follow and each of them could be represented in 2 bytes as a signed (or unsigned in some cases) integer variable which would give a binary representation requiring 46 bytes/line instead of the 145 bytes that are currently required (may be off by a few bytes as I counted them manually). It appears that this is a standard format for representing temperature data but when I have a chance to start playing with this data, there is no way that I'm going to store it in this format on my computers

One problem with data conversion is that errors may creep in and this is what we are trying to avoid. One possible way of flagging errors would be to create a pseudo-checksum of all of the numeric values in the ascii format temperature reading and if converting all of the binary variables to text results in something different then clearly there is a problem. I'll leave this problem for whoever takes on the task of coming up with an open-source world temperature data archive.

Going through this data by hand is a gargantuan task, but with the average desktop machine being significantly more powerful than the supercomputers of even 20 years ago, automatic analysis of huge amounts of data is possible in short amounts of time. The main things we're interested in are temperature trends and correlations between geographically close stations as well as temperature outliers which might be errors or may be the result of the numerous artifacts that plague weather stations in urban areas that have been looked at in Anthony Watts surface stations project. This project demonstrate the power of thousands of interested amateurs to exhaustively document the potential temperature errors in every surface weather station in the US. Analysis of temperature data is something which any amateur scientist armed with a computer and and often self-written software can do and potentially make valuable contributions to climate science. Open-source code has been described as peer-reviewed code and the contributions of extended peer-review by amateur climatologists to this area have been embarrasing to the inbred climatologic establishment. Steve McIntyre has done outstanding work, WUWT has been a hotbed for dissemination of independent climate information and E.M. Smith has audited GISS FORTRAN code finding major bugs as well as doing an impressive amount of other climatologic work. I'm happy to make my minor contribution to the process to find climate truth.

Unless someone with more time than me decides to take on the Kamloops temperature data, I plan on analyzing this data when RSN. TIme is one thing that is in very short supply for me.

One warning about the program is that it does run fairly slowly. Data transfer from the internet control returned data is done a byte at a time primarily because I didn't want to write the code to move chunks of data at once instead; I had my code working inside of an hour and I was more interested in looking at the climate data than I was in coming up with the ideal download program. Also, be warned that the VB6 code is not very nice as I used the first thing that worked rather than striving for elegant code. I don't normally program like that and was inspired by some of the climategate FORTRAN code from FOI2009.zip. My code is at least as good as that FORTRAN code and better documented.

Now that you've read through my verbose and rambling description of the program (no time to write a shorter entry) here is where you can get the code. Right now it is only available as a 7zip format file and one can download 7zip at this link.

25/4/2010 T:=14:04

Sorry for the error in the download link; Thingamablog used "downloads.html" instead of "download" resulting in a 404 error for anyone who tried this. Puzzling as the program has been quite good until now. Lack of appropriate testing on my part.

Posted by Boris Gimbarzevsky at 2:23.15
Edited on: 25/04/2010 13:54.58
Categories: Climatology, Computers

15/03/2010

Capturing youtube files

I wasn't planning on doing this but it's another one of these things that just happened. I'm down to 24 Gb of free HD space on my laptop and decided it was time to see what is taking up so much space (80 Gb HDD on my TC4400). A few things were easy like the 8 Gb of photos which I don't need to carry around with me but what was surprising was the temporary internet file folder which weighed in at 2 Gb. I didn't recall setting the disk cache to this high a value and so it was time to investigate. In DOS, it is easy to look at this folder which is located at:

C:\Documents and Settings\<user>\Local Settings\Temporary Internet Files\Content.IE5

if you use M$ IE as your browser (I somehow trashed both Firefox and SeaMonkey on my laptop so it's either Opera or IE). To see what is going on you need to look for system files:

  • dir /a:s
  • Volume in drive C has no label.
  • Volume Serial Number is xxxx-yyyy
  • Directory of C:\Documents and Settings\<user>\Local Settings\Temporary Internet Files
  • 14/Mar/2010 10:34 <DIR> .
  • 14/Mar/2010 10:34 <DIR> ..
  • 14/Mar/2010 10:34 <DIR> Content.IE5
  • 14/Mar/2010 10:34 67 desktop.ini
  • 1 File(s) 67 bytes
  • 3 Dir(s) 24,069,357,568 bytes free

Obviously on your machine <user> will be the user account you're logged in under. All the material we want is in the directory Content.IE5. In DOS, when one switches to this directory one finds:

  • >dir /a:s
  • Volume in drive C has no label.
  • Volume Serial Number is xxxx-yyyy
  • Directory of C:\Documents and Settings\<user>\Local Settings\Temporary Internet Files\Content.IE5
  • 14/Mar/2010 10:34 <DIR> .
  • 14/Mar/2010 10:34 <DIR> ..
  • 11/Mar/2010 12:07 <DIR> 0N27AIOW
  • 12/Mar/2010 21:45 <DIR> 45YRCP6B
  • 14/Mar/2010 10:34 <DIR> 4DC7X5IG
  • 14/Mar/2010 10:37 <DIR> 4HUJWLER
  • 11/Mar/2010 20:26 <DIR> 4XAZ0LU3
  • 10/Mar/2010 07:57 <DIR> 4Z43ITS1
  • 11/Mar/2010 12:02 <DIR> 5JUZA1AO
  • 09/Mar/2010 00:47 <DIR> 63QPONG1
  • 11/Mar/2010 12:07 <DIR> 65CVAXY5
  • 10/Mar/2010 01:22 <DIR> 8B41MPUX
  • 11/Mar/2010 12:07 <DIR> 9TXD5YA8
  • 11/Mar/2010 23:18 <DIR> A7WPSVYD
  • 11/Mar/2010 12:10 <DIR> ANC8X53I
  • 11/Mar/2010 20:26 <DIR> C56ZOHQZ
  • 11/Mar/2010 20:26 <DIR> CP63CX2Z
  • 14/Mar/2010 10:41 <DIR> CPIZ8DAR
  • 11/Mar/2010 12:10 <DIR> CX4IQS8H
  • 14/Mar/2010 10:34 67 desktop.ini
  • 07/Mar/2010 17:30 <DIR> E387GLSJ
  • 07/Mar/2010 17:30 <DIR> ERI7UD2Z
  • 11/Mar/2010 23:18 <DIR> I72LG8XG
  • 07/Mar/2010 17:30 <DIR> K1GJ4N03
  • 14/Mar/2010 11:02 <DIR> MDC90HAZ
  • 14/Mar/2010 10:43 <DIR> N7TSMG0O
  • 26/Feb/2010 21:56 <DIR> OB0ZCBEN
  • 07/Mar/2010 17:30 <DIR> OD0PIBKL
  • 11/Mar/2010 12:10 <DIR> OEWPWNY9
  • 11/Mar/2010 19:55 <DIR> QKXZB2I9
  • 11/Mar/2010 22:19 <DIR> S1MR4L27
  • 07/Mar/2010 17:31 <DIR> SKGAXR8D
  • 11/Mar/2010 12:02 <DIR> TKUZOKX5
  • 11/Mar/2010 12:10 <DIR> U5YRIBWN
  • 11/Mar/2010 12:10 <DIR> WBINMDPR
  • 1 File(s) 67 bytes
  • 34 Dir(s) 24,069,357,568 bytes free

Note that dir /a:s is needed to list the system files which contain IE web cache; a simple dir will suggest that desktop.ini is the only file in that folder.

I had never seen that many files in content.ie5 before and didn't feel like writing a script to copy them so I decided to see if I could make them visible to Explorer. If you doubleclick on "Temporary Internet files" you get a useless directory which has little resemblence to what is actually in the web cache; doubleclicking on an flv file will take you to the original site which may no longer have that file and hence it is less than useless. I suspect this was a crude attempt at hiding temporary files.

To access Content.IE5 and get a DOS type directory listing, it is merely necessary to rename the desktop.ini file in "Temporary internet files" to something else:

  • s>attrib -H -S desktop.ini
  • C:\Documents and Settings\<user>\Local Settings\Temporary Internet Files>dir
  • Volume in drive C has no label.
  • Volume Serial Number is xxxx-yyyy
  • Directory of C:\Documents and Settings\<user>\Local Settings\Temporary Internet Files
  • 31/Jan/2008 23:33 67 desktop.ini
  • 1 File(s) 67 bytes
  • 0 Dir(s) 24,075,079,680 bytes free
  • C:\Documents and Settings\<user>\Local Settings\Temporary Internet File
  • s>type desktop.ini
  • [.ShellClassInfo]
  • UICLSID={7BD29E00-76C1-11CF-9DD0-00A0C9034933}
  • C:\Documents and Settings\<user>\Local Settings\Temporary Internet Files>ren desktop.ini *.nii

The attrib -H -S statement clears the hidden and system attributes of file desktop.ini which allows it to be renamed to desktop.nii. That's all it takes to be able to see all of IE's web cache folders like any other folders on the disk. Presumably the UICLSID line is to tell the shell to display the contents of the folder as the idiotic set of internet files that shows up without this modification.

Once I had fixed this windoze bug I started clicking on the various webcache folders and was surprised to see how much garbage was there. I've noticed that certain sites tend to be very cpu intensive when they should be doing nothing and there were hundreds of copies of the same picture in some of the folders which was presumably downloaded time and time again from a poorly designed website. There were also a lot of flv files.

Flv files are flash video files and to play them one needs FLVPlayer.exe (you'll have to google the filename as the URL that comes with FLVPlayer setup file no longer works). What I found was that most of the files were Youtube videos still left in the cache. So, I figured that this is an easy way to get Youtube videos as I like to keep a copy of certain things that I watch. I launched IE, went to Youtube and, sure enough, there was an FLV file created in one of the webcache folders that grew in size as the video downloaded. I waited for it to finish downloading it so I could copy it and --- the file disappeared!

This was odd as I could still play the video. So off to look at more places where it could be hiding. It was found in the "local settings\Temp" folder but there was only one problem; the file was locked in some way. Not only that, but when one terminated the instantiation of IE that was displaying that file, it disappeared. Most annoying. Changing file permissions to forbid anyone from deleting the file didn't work so it was time to get more aggressive.

Process explorer was then used to bring up a list of open handles in the copy of IE that was playing the file and the handle for the filename was manually closed: note, this is a dangerous operation and if you randomly close file handles in processes you will potentially cause a BSD on your machine. The only way to recover from a BSD is to reboot (and on my laptop this means pulling the battery with power cable unplugged). You've been warned.

Once I closed the file handle, IE no longer knew about this file (and I killed that instantiation of IE with process explorer to make really sure that file would stay there) and it was left behind on my disk but, unfortunately, it was still locked and I couldn't read it or copy it (or delete it which is a major problem with this method of saving youtube flv files). I know there's a simpler way to do this, but I used the tools at my disposal and launched Restorer2000, had it enumerate the files on C: drive and copied the "locked" flv file to another location. Restorer2000 is one of the most usefull programs that I've ever bought online and it has saved me months of time as I tend to mess up a lot of HDD's for some unknown reason.

The flv files hanging around in the IE webcache folders were downloaded from youtube as late as Feb 2010. The fact that suddenly the whole manner in which youtube flv files are stored on disk now means that Youtube is actively working to prevent people from saving files from their computer. This is futile and idiotic to say the least. If one is able to display a video file on ones monitor, then somewhere in ones machine there exists a representation of that file which can be saved to a disk file on ones machine. I don't care if DRM is used or if the code is obfuscated; if you can see it you can make your own copy. This might mean snooping on the video card, it might mean copying process memory and any number of things that are easy to do. M$ is trying to make all these things much harder for the average user to do, but as long as one has raw access to the HDD, all such efforts are ultimately futile. Vista and W7 allow raw disk I/O which can only be defeated by encrypting the disk drive contents. Anyone who relies on M$ disk encryption technology is an idiot as this is supporting their drive to treacherous computing (they use the oxymoron "trusted computing"). /rant

One of the reasons I have for saving youtube flv files is that files that google doesn't like have a bad habit of disappearing from youtube. Case in point: I had a link to the hilarious video "Hide the Decline" put there by M4GW(Minnesotans for Global Warming) on my climategate page. I clicked on that page today to grab the flv and the link no-longer works. It was the first time I had seen an error of this type in youtube and someone engineered the takedown of Hide the Decline. It didn't take long to find multiple copies of this video on youtube and I grabbed the flv using the method outlined above. I'll be putting this on my own webserver (when I have the time) where it will be protected from the malignant influence of fat Al who will hopefully see all of his carbon billions disappear RSN.

The next step is to start digging into file permissions to see how I can gain read and delete access to the flv files stored via my, quick but very dirty method, so I can delete the files after I've copied and renamed the file. I'm sure there is a simple way of doing this but I have put off doing various medical reports for too long and this is something that's going to have to wait for another day.

Addition on 28/3/2010

Just remembered that CACLS will allow one to edit file access control lists. The youtube file (now undeletable) in Temp directory had the following ACL:

  • C:\Documents and Settings\<user>\Local Settings\Temp\flaCE.tm
  • BUILTIN\Administrators:F
  • ATHENA\<user>:F
  • NT AUTHORITY\SYSTEM:F

To get access to the file so one can copy it/rename it and eventually delete it, it is necessary to remove BUILTIN\Administrators and "NT AUTHORITY\SYSTEM" from the ACL. This is done with 2 CACLS calls:

  • C:\Documents and Settings\<user>\Local Settings\Temp>cacls flace.tmp /e /r builtin\administrators
  • processed file: C:\Documents and Settings\<user>\Local Settings\Temp\flaCE.tmp
  • C:\Documents and Settings\<user>\Local Settings\Temp>cacls flace.tmp /e/r "NT AUTHORITY\SYSTEM
  • processed file: C:\Documents and Settings\<user>\Local Settings\Temp\fl aCE.tmp

Then one can do things like:

  • C:\Documents and Settings\<user>\Local Settings\Temp>ren flace.tmp *.flv
  • C:\Documents and Settings\<user>\Local Settings\Temp>dir *.flv
  • Volume in drive C has no label.
  • Volume Serial Number is xxxx-yyyy
  • Directory of C:\Documents and Settings\<user>\Local Settings\Temp
  • 14/Mar/2010 11:59 10,700,649 flaCE.flv
  • 1 File(s) 10,700,649 bytes
  • 0 Dir(s) 23,513,124,864 bytes free

I also seem to recall using CACLS to obtain access to certain windoze directories such as System Volume Information but am not sure since my System Volume Information folder is completely accessible to me now.

Posted by Boris Gimbarzevsky at 1:17.40
Edited on: 28/03/2010 22:17.21
Categories: Computers

26/02/2010

Google loves spammers

Google loves spammers. I can say this with complete assurance as I've sent 4 applications to Google asking them to cease and desist with the misuse of my name and not a reply from them at all. The whole issue concerns a temporary blog I created on blogger when I was shutting down my practice in Vancouver. This was at drgimbarzevsky.blogspot.com. If one clicks on this site, one now finds that it is associated with the Canadian Pharmacy spam criminals who also forge a lot of spam with drgimbarzevsky.com as the origin. I'm hopefully making a bit of an impact on their operations as all mail directed at drgimbarzevsky.com is sent to the abuse address of the most recent chicom server they're spamming from. Every few months I re-enable mail to come to one of my email addresses, find the IP of their newest server and then change my MX records to divert the millions of emails/year that are bounced back to me as a result of poorly configured mail servers (it's a simple operation to do a nslookup of "drgimbarzevsky.com" to see if it matches the IP address of the actual sender. If they don't match the header is forged and there is absolutely no reason to send it back to me).

I decided to look where this group of assholes is hanging out now and, lo and behold, they're now using a US server! Must be getting really hard up as usually they use chicom servers which seem to be happy to send any amount of spam as long as they get paid. I've got a list of Chinese IP addresses and don't let any of them through to any of my machines and building the equivalent of the great wall on the internet around China would result in a massive decrease in spam. OTOH this US based machine could be part of the botnet and all my email to the abuse department will do is result in one less zombie.

Here's the latest start of "medical services plan" which is what one gets if one clicks on drgimbarzevsky.blogspot.com

  • 20 يونيو, 2008
  • Medical Services Plan
  • Canadian Pharmacy with high quality medications!Bonus pills added to your order!
  • Canadian Pharmacy offers high quality medications: Viagra, Cialis, Levitra, and many medications! Various payment methods available: Visa, Wire transfer, echeck, Western Union or MoneyGram.

The first link maps to:

http://199.80.55.18/go.php?data=vIVssHkxsXkyFdN9lixkOyuCqJvC4FMUNGUzu2HO093LQ09BsGLvkmQWpffaqwwLL6AhPdKnyhiPw8WsIQQH4OGcD8r8eQh6lW3aKsvWMT64k%2FwhlGoGornoWPpRJnWz5hh4HFZqLXnn4IJdJU2MuM9CQuACXQuGe6ngV%2BXql7T1WumGCahjf%2B39dNpoXoiYQ5aZ29K%2BGUVqZiCy7LuHlH68%2FYAAw0EXpAwGLLFfbEvZT47ZBL98St5v%2BpbigpbufjCZLOfQlCDuZFffqRSJ57xpkv%2BBM43T8RzYuD3IQ2n%2B5jxwOta7C9r3yRCN49dA2UgQiZQWa%2Bq0BEe4VB1IJhTUGYtFcYdQqSBYDSzS2icQ2Cedw1A2HS8818foy%2F%2F68IAOmlxbEMAtAXdckIVLGzy7VOODJIvrIQYXfDGWmcnPrnDUSjV3pXSqMX9VI%2BwTGt17vByG%2Br6YkCBIuENNgcZFs0t9uIh9Iggkr9MEUiKxDL%2Fz7bvm%2Fb3n2subCBE8CtoBWl6Z5QJKw%2BK%2BgTOaXZDlVPzA8zuwnPf9%2BTP6CXi6aTv%2F6Uco6YYRictFg6iQ%2F2mEYCfDuXs3Ti2uBLooncjCOQOeRBSDhUAi1DlU8%2BGa8BvDMWgdFFlJackls6FJOEL1UCPk%2FM03WTYxppnt7FzVRrNzRfUvPXUvaOmZN9Svj1heYXgFQuigS8MSmXC0ABaYCujgmMi%2F7ypTSr8I3%2BsRkoaKFTbf0P9FYX34h03%2BgEoZrIaRCoEEmYy9lJqCMDIdr6NYwQc5JVzlB8J0bClI0v013yzg28lUCI3DjJCZJrgXBy%2BZt8iL6diUF1JHtWzlZNL%2Fe1qObKqcv3w9i2n2mKLzaRdpkEKUegFHkww4ekIKR2u1RfueebM179do2%2BiIRITtGKLG08tK3kOSTcyjVRTTlrbpAu35N8Pw21YhKxQwyIhe451%2BvhLJb3rJ5wPndRwc5Zg0WjFLoboV9%2B9rzbhgtu2qkL5JSMFdtGtJKZCgdPGJ6ukwdlfgP4rEYZewNUm7czKm9xtrjVxc%2B1T%2Bmn%2FwPpw3YABt%2FyyQWazfRtsUt4ACLAxX%2FdwPfTsGNZSDmv6QzlSYkdqdoRs%2FMY4uR%2BAH7yQr5rJQAC3A0q%2B3MCSeYKumqUuOQLbaPxbgq3teyghCQOH3Tzx2dcshGcVUTwPp5lepDkXVKRLXqfE%3D

Lets see who owns 199.80.55.18. nslookup gives:

  • 02/26/10 20:04:35 dns 199.80.55.18
  • nslookup 199.80.55.18
  • Canonical name: c-n080-u0390-18.webazilla.com
  • Addresses: 199.80.55.18

Then let's do a whois on this IP address:

  • OrgName: WZ Communications Inc. Org
  • ID: WZCOM
  • Address: 131 W Wilson Street
  • Address: Suite 600
  • City: Madison
  • StateProv: WI
  • PostalCode: 53703
  • Country: US
  • NetRange: 199.80.52.0 - 199.80.55.255
  • CIDR: 199.80.52.0/22
  • OriginAS: AS40824
  • NetName: WZCOMM-US
  • NetHandle: NET-199-80-52-0-1
  • Parent: NET-199-0-0-0-0
  • NetType: Direct Allocation
  • NameServer: NS1.WEBAZILLA.COM
  • NameServer: NS2.WEBAZILLA.COM
  • NameServer: NS3.WEBAZILLA.COM
  • Comment: Please send abuse complaints to
  • RegDate: 2009-03-05
  • Updated: 2009-09-25
  • OrgAbuseHandle: WZCOM1-ARIN
  • OrgAbuseName: WZCOMM Abuse
  • OrgAbusePhone: +1-408-404-3912
  • OrgAbuseEmail: abuse@webazilla.com
  • OrgTechHandle: WZCOM-ARIN
  • OrgTechName: WZCOMM NOC
  • OrgTechPhone: +1-408-404-3912
  • OrgTechEmail: bk@webazilla.com

Email to abuse address sent right after this posted. Lets see what happens when one contacts the server:

  • 02/26/10 19:28:14 Browsing http://199.80.55.18/go.php?data=vI
  • Fetching http://199.80.55.18/go.php?data=vI ...
  • GET /go.php?data=vI HTTP/1.1
  • Host: 199.80.55.18
  • Connection: close
  • User-Agent: Sam Spade 1.14
  • HTTP/1.1 302 Found
  • Date: Sat, 27 Feb 2010 03:28:15 GMT
  • Server: Apache
  • X-Powered-By: PHP/5.2.12
  • Expires: Mon, 26 Jul 1997 05:00:00 GMT
  • Last-Modified: Sat, 27 Feb 2010 03:28:15 GMT
  • Cache-Control: no-store, no-cache, must-revalidate
  • Cache-Control: post-check=0, pre-check=0
  • Pragma: no-cache
  • Location:
  • Content-Length: 0
  • Connection: close
  • Content-Type: text/html

Not that exciting so lets try http://199.80.55.18/go.php?data=deathtospammers which gives the same result. Interestingly trying the huge datastring crapped out SamSpade so perhaps it was designed as a means of preventing people from using tools like SamSpade to probe this spamsite. Most likely the data string has the origin of the link encrypted within it so the spammers know which are the productive blog pages and probably some botnet control info but I'm just guessing here. No, I'm not bored enough to start trying to crack this data as I've got way more important stuff to do.

Never, never, never click on one of those links using M$ IE as your chances of acquiring malware are very high. So lets try Opera hoping that this group of criminals hasn't got any Opera exploits. To ensure that nothing sneaks through lets also start Wireshark and log all of the traffic to and from my machine in the process.

Interestingly there's a totally different webpage displayed when one goes through Opera than when goes through Firefox. Pasting the above URL into Opera goes through at least one other IP address (all information captured by Wireshark running in the background) and eventually one is on the site: http://www.canadasmedicine.com/?c=click&ck=26492

Let's see what unethical registrars and ISP's are involved. First nslookup:

  • 02/26/10 20:22:00 dns canadasmedicine.com
  • Canonical name: canadasmedicine.com
  • Addresses:
  • 88.86.119.225
  • 62.248.113.106

Interesting, 2 IP addresses but we'll get to those in a moment. First lets find out who the alleged owner of this domain is:

02/26/10 20:22:21 whois canadasmedicine.com@whois.geektools.com

  • whois -h whois.geektools.com canadasmedicine.com ...
  • GeekTools Whois Proxy v5.0.4 Ready.
  • Checking access for xxx.yyy.zzz.aaa... ok.
  • Checking server [whois.crsnic.net]
  • Checking server [whois.tucows.com]
  • Results:
  • Registrant:
  • N/A
  • 13 Av Des Cedres
  • Ville DAvray, 92410
  • FR
  • Domain name: CANADASMEDICINE.COM
  • Administrative Contact:
  • Berder, Ronan ronan@33drugs.com
  • 13 Av Des Cedres
  • Ville DAvray, 92410
  • FR
  • +44.02081338455
  • Technical Contact:
  • Berder, Ronan ronan@33drugs.com
  • 13 Av Des Cedres
  • Ville DAvray, 92410
  • FR
  • +44.02081338455
  • Registration Service Provider:
  • Hover, help@hover.com
  • 416.538.5498
  • http://help.hover.com
  • Registrar of Record: TUCOWS, INC.
  • Record last updated on 19-Jan-2010.
  • Record expires on 06-Jun-2010.
  • Record created on 06-Jun-2008.
  • Registrar Domain Name Help Center:
  • http://domainhelp.tucows.com
  • Domain servers in listed order:
  • NS2.33DRUGS.COM
  • NS1.33DRUGS.COM
  • NS3.DRUGCUSTOMER.COM
  • NS5.ALPHA-NS.COM
  • NS4.DRUGCUSTOMER.COM
  • Domain status: ok

The spammer says he's French but who knows where he's really from. All that this record tells us is that we have to do some more digging. Lets start with the domain servers and 33DRUGS.COM seems like a logical place to start.

  • 02/26/10 20:29:14 dns 33drugs.com
  • Canonical name: 33drugs.com
  • Addresses:
  • 62.248.113.106
  • 88.86.119.225

If those IP's seem familiar it's because they're the same two that popped up when we did nslookup on canadasmedicine.com. Lets find where these are located:

  • Results:
  • % This is the RIPE Database query service.
  • % The objects are in RPSL format.
  • %
  • % The RIPE Database is subject to Terms and Conditions.
  • % See http://www.ripe.net/db/support/db-terms-conditions.pdf
  • % Note: This output has been filtered.
  • % To receive output for a database update, use the "-B" flag.
  • % Information related to '62.248.113.96 - 62.248.113.111'
  • inetnum: 62.248.113.96 - 62.248.113.111
  • netname: AIRMARK
  • descr: AIR-MARK MARKA HAVACILIK LTD
  • descr: TURKEY
  • country: TR
  • admin-c: HE337-RIPE
  • tech-c: HE337-RIPE
  • status: ASSIGNED PA
  • mnt-by: AS9121-MNT
  • source: RIPE # Filtered
  • person: HAKAN ERMAN
  • address: MARKA HAVACILIK TURIZM VE TASIMACILIK TIC.LTD.STI.
  • address: INONU CAD.GUMUSKONAK APT.NO:44/8 GUMUSSUYU
  • address: TAKSIM ISTANBUL
  • e-mail: hakan@air-mark.com
  • phone: +90 212 245 44 65
  • fax-no: +90 212 245 44 86
  • nic-hdl: HE337-RIPE
  • source: RIPE # Filtered
  • % Information related to '62.248.0.0/17AS9121'
  • route: 62.248.0.0/17
  • descr: TR-TELEKOM-960902
  • origin: AS9121
  • mnt-by: AS9121-MNT
  • source: RIPE # Filtered

So one of our webservers is in Turkey and there are only 15 IP addresses in the IP block which suggests that either someone is making a bit of money on the side by sending spam, or more likely, one of the computers in that IP block is a zombie and part of a botnet. Now lets find the other server:

  • Results:
  • % This is the RIPE Database query service.
  • % The objects are in RPSL format.
  • %
  • % The RIPE Database is subject to Terms and Conditions.
  • % See http://www.ripe.net/db/support/db-terms-conditions.pdf
  • % Note: This output has been filtered.
  • % To receive output for a database update, use the "-B" flag.
  • % Information related to '88.86.96.0 - 88.86.127.255'
  • inetnum: 88.86.96.0 - 88.86.127.255
  • netname: CZ-SUPERNETWORK-20060214
  • descr: SuperNetwork s.r.o.
  • country: CZ
  • org: ORG-SS59-RIPE
  • admin-c: ZC10-RIPE
  • tech-c: ZC10-RIPE
  • tech-c: DC1000-RIPE
  • tech-c: ML7551-RIPE
  • status: ALLOCATED PA
  • mnt-by: RIPE-NCC-HM-MNT
  • mnt-lower: SUPERNETWORK-MNT
  • mnt-routes: SUPERNETWORK-MNT
  • mnt-domains: SUPERNETWORK-MNT
  • source: RIPE # Filtered
  • organisation: ORG-Ss59-RIPE
  • org-name: SuperNetwork s.r.o.
  • org-type: LIR
  • address: Bilejova 407
  • address: 46303
  • address: Liberec
  • address: Czech Republic
  • phone: +420296826296
  • fax-no: +420482731466
  • e-mail: zdenek@superhosting.cz
  • admin-c: DC1000-RIPE
  • admin-c: ZC10-RIPE
  • admin-c: ML7551-RIPE
  • mnt-ref: SUPERNETWORK-MNT
  • mnt-ref: RIPE-NCC-HM-MNT
  • mnt-by: RIPE-NCC-HM-MNT
  • source: RIPE # Filtered
  • person: Zdenek Cendra
  • address: U Pergamenky 2
  • address: Praha 7
  • address: 170 00
  • address: Czech Republic
  • e-mail: zdenek@superhosting.cz
  • phone: +420296826296
  • nic-hdl: ZC10-RIPE
  • mnt-by: SUPERNETWORK-MNT
  • source: RIPE # Filtered
  • person: David Cermak
  • address: PRO-ZETA spol. s r.o. Praha
  • address: Mukarovska 1567/25
  • address: Praha 10
  • address: 100 00
  • address: Czech Republic
  • e-mail: david@prozeta.cz
  • phone: +420222742702
  • nic-hdl: DC1000-RIPE
  • mnt-by: SUPERNETWORK-MNT
  • source: RIPE # Filtered
  • person: Miroslav Laus
  • address: U Pergamenky 2
  • address: Praha 7
  • address: 170 00
  • address: The Czech Republic
  • e-mail: mirek.laus@superhosting.cz
  • phone: +420296826296
  • nic-hdl: ML7551-RIPE
  • source: RIPE # Filtered

This IP block obviously belongs to an ISP in the Czech Republic and, is probably another zombie PC. I'm sure that if you perform nslookups on canadaspharmacy.com on a daily basis you'll find that the servers change frequently. The whole process is an attempt to try to prevent people from tracking them down and I'm sure that there's no link between the online pharmacy spammers and this trail of evidence aside from control packets sent to the botnet and the only way that one would catch them would be to have a packetsniffer running constantly on one of the botnet zombies to record the control packets. If I was doing this, I'd also route control packets through multiple zombies before they ended up at the target machine and use either an internet cafe computer as the source of the control packets or on open WiFi link found while warwalking. Wouldn't surprise me in the least if this further level of paranoia was part of their SOP.

One of the enabling factors in this whole process is Google; one knows Google must be crooked when Al Gore is on their board of directors. This slimeball has raked in millions with the AGW scam and who knows what other shady deals are going down with Google; after all they are actively helping the chicoms censor internet access in China. Since I've notified Google multiple times regarding the misuse of my name in relation to canadaspharmacy.com and because they have done nothing, I can only conclude that they have some financial stake in this and are thus involved in illegally distributing spam. The other alternative is that those report forms on blogger are routed to dev/null and they could care less what someone puts on blogger unless it gets significant exposure in the MSM. It will be interesting to see if Google comes after me for the accusations I've made in this blog post. There is one potentially inflammatory comment if one sentence is taken out of context. Taking things out of context is what mindless programs are really good for. If Google's software looks for people accusing Google of performing nefarious acts then maybe I'll hear from Googles legal department the next time googlebots crawl my site. At that point we can talk about what it takes for me to remove my non-flattering references to Google both in this post and in the Keywords

I have some ideas of what the fate of spammers should be. SamSpade is a very nice tool with lots of convenient internet related tools in one package and I highly recommend it as well as Wireshark. Didn't realize how many SMB packets were flowing through my home network until I ran wireshark and captured >400,000 packets during the time it took me to write this. Time to take a look at WTF the offending machines are doing as they don't need to talk that much with one another.

Posted by Boris Gimbarzevsky at 19:30.22
Edited on: 26/02/2010 21:42.27
Categories: Computers, Medical

16/02/2010

Stellaris Design Contest

In 2005 I entered a Circuit Cellar Ink magazine design contest as I had the urget to build some sleep monitoring hardware and the system that I came up with didn't win any prizes or even get mentioned, but it allows me to monitor sleep leg movement and do breath by breath monitoring alone with recording SaO2 and HR. A partial writeup is in my first abortive blog attempt here.

On 28/1/2010 as I checked my email I saw there was a new design contest, this time with the TI Stellaris LM3S9B96 CPU which is an ARM M3 core with 96 Kb of SRAM and 256 Kb of flash RAM as well as the obligatory A/D converters, serial ports, timers, I2C ports etc that seem to come along with every new embedded systems design nowadays. I naievely assumed that the DK-LM3S9B96 was the part I'd be getting to play with for the design contest and typed up my idea for a project and sent it off in about 10 minutes.

This CPU is powerfull enough to finally implement an idea of mine that has been kicking around in my head for 15 years or so and that is an ambulatory physiologic monitoring system. If I get a digital holter monitor out of this project I'll be happy, but what I was also planning on doing besides EKG was to have 2 3-axis accelerometers (one on each leg), temperature sensor (finally will be able to hook up that I2C output sample high precision temperature sensor), ambulatory pulse oximetry, ambient sound, etc. The concept is fairly simple, attach a bunch of sensors of various types to a person and feed all of their outputs to an embedded processor which will timestamp all the data chunks and write them out to some form of storage. Right now flash memory cards come in huge capacities and a 4 Gb card would be good for a few days of monitoring. Until recently, the only way one could get all of this physiologic data was to have theperson sitting in a lab next to a computer.

I could have easily done this project in 1987 as long as the person didn't walk more than a few feet from the data acquisition system which ran on a PDP-11/23 Minc system controlled by a PDP-11/34 which had massive, for the time, storage capacity of 2 60 Mb drives! If one threw in the tape drive which was often used as the primary means of storage of experimental data, the total system probably weighed in at over 1000 lb. The PDP-11 was a real joy to program and is still my favorite processor.

To put the memory space one is dealing with in perspective, the PDP-11 had 56 Kb of RAM that was directly addressable unless one used bank switching to access more memory which was, IIRC, 512 Kb on our PDP-11/23. This RAM wasn't used during data acquisition as it was simpler to just stream data from the 11/23 to the 11/34 over a parallel interface and thencefrom to disk or tape.

So in one sense, all the software has already been written and it's just a matter of porting it to a different hardware platform, or so I thought. In contrast to the HCS08 based Zigbee boards that I used in 2005, the EKK-LM3S9B96 comes with thousands of pages of documentation and an IDE that seems to have a lot steeper learning curve than the HSC08 Metroworks CodeWarrior IDE. The LM3S9B96 is a 32 bit processor with a rich instruction set and is not something that I could start hacking the same evening as I got the development kit as was the case with the HCS08. It looks like I'm in for a few evenings of reading before I can even power up the device and run the embedded systems version of the ubiquitous "Hello world" program.

One of the things that one is expected to do with the Stellaris development kit is to use the SafeRTOS as part of the project. This will be a real novely for me as any embedded designs I've thus far done have had no OS besides the boot loader to flash new programs into flashram. I've been used to writing all of my interrupt routines and using a periodic timer tick to run a low priority subroutine to make sure data is moving through the device as it should. I'm tempted to just use SafeRTOS in debugging and data display mode and turn it off for ambulatory data acquisition.

What never ceases to amaze me is how much more powerfull embedded systems are every year. Unfortunately they get more complex also and I'm tempted to stick with CPU's that I've had a long and comfortable relationship with like the 680x0. I have a couple of NetBurner development kits and the project I built on them had 90% of the time trying to figure out how to set switches in the compiler and how to reflash parts from my laptop instead of actual programming. That was the nice thing about the PDP-11; just fire up a text editor, write FORTRAN or assembly code, compile it and then debug through the console. The debug capabilities of the new embedded systems are way more powerfull than my stepping through code one instruction at a time or doing octal dumps of RAM on an LA36 printer like I did in the 1980's, but it took way less time to learn how to do it.

One of the reasons for going with the ARM is that this is a very widely used CPU architecture for embedded systems; my Palm TX uses and ARM processor and they are ubiquitous in other devices. At some time I have to learn the architecture of the ARM and what better way to do it than put myself under the time pressure of having to produce a working project by 23/6/2010. The other thing I need to do is learn about how USB devices work and how to interface to them on a low level. I grew up with serial ports and still have many devices that I use that communicate through RS-232 links. New computers no longer have serial ports (it really pissed me off when the parallel port was removed) and instead rely on USB or firewire for external inputs. RS-232 I/O is very easy to code but when one gets to USB the problem is non-trivial. I have to learn the low level details of the USB specification and get into low level windoze driver calls to grab data from USB connections. USB 2.0 is a lot faster than RS-232 which, in my experience, maxes out about 110 Kbaud but to do low level USB I/O in VB is going to take me a week or two of intense study.

The other thing that the EKK-LM3S9B96 has is an ethernet interface and I've never done any low level coding of ethernet capable devices. I've spent a lot of time looking through Wireshark output to trace down network problems. The Stelleris development kit comes with a rich set of drivers for doing ethernet I/O, but if one wants to debug projects a detailed understanding of TCP/IP is necessary so time to find my book on the subject.

Probably the most annoying thing about this project is that documentation no longer comes in book form. The LM3S9B96 programmers manual alone is over 1200 pages and I'm not about to print this out. Then one has all of the technical documents, other documents that deal with the peculiarities of the Keil IDE, etc. What I really used to look forward to was to lie back with a programmers manual for a new CPU architecture and just browse. You can't do this with an electronic document, not at least in the way that I like reading a book. The most serious problem with reading a PDF of a book is the damage to laptops that occurs during the process. One of the hazards of reading in a supine position like I do, often at the end of a 12 hour workday, is that I'll fall asleep during the process. As muscle tone decreases the book falls out of my hands and onto the floor. No problem with a paper book although I did sustain chest injuries from dropping the one volume edition of Harrison's Principles of Internal Medicine during medical school. I've trashed laptop HDD's by falling asleep reading an on-screen PDF. What is needed is a very sturdy PDF viewer which will have the screen resolution of a book, the ability to make notes in the margins (all of my books are extensively annotated) and survivability from either being dropped or sat on by a 200 lb physician. Just remembered that I have a ToughBook laptop that's not currently being used for anything -- this platform should survive in the book emulation application.

Posted by Boris Gimbarzevsky at 22:39.50
Categories: Computers

13/02/2010

Electronic medical records

Updated version to come RSN.

Posted by Boris Gimbarzevsky at 23:20.36
Edited on: 16/05/2010 20:55.06
Categories: Computers, Medical

22/01/2010

Windows 7 rant

A couple of weeks ago I purchased a Sony touchscreen computer because I like the idea of using a finger to control things on a screen. I've used tablet PC's since 1993 and a pen is my preferred interface to a GUI instead of a mouse. I'm not a big fan of windoze since M$ has been progressively crippling their OS's since win NT came out but I was convinced to try windoze 7 (W7) as "it's better than Vista".

While I like Linux in principle, my Linux boxes aren't used for my everyday software development since I decided that I really liked VB in 1993 (before that I was using Hypercard on the Mac but it was s l o w on my Mac+). VB6 was the last version of VB that M$ put out and also the last IDE that I bought from them. I like VB6 and it suits my needs perfectly. It produces code which runs fast enough that I rarely have to use assembly language routines to speed up critical sections. One of the benefits of M$ bloatware is that it has made the average desktop of today far superior to the supercomputers of 20 years ago as it takes this much computational power to run this monstrosity from Redmond. The Sony machine is a dual core 64 bit 2.9 GHz Intel E7500 CPU with 4 Gb of RAM. It is the fastest machine that I've ever owned.

The first inkling that something was amiss was when I launched process explorer and noticed that baseline CPU load was 5-10% which is ludicrously high. My XP machine OS overhead is <1% and it uses a much slower CPU. Investigation was in order.

W7 seems to be designed on the infuriating principle that applies to most electronic hardware now, a sealed box with "No user servicable parts inside Servicing to be done only by qualified service personnel" prominently displayed on the back. When I buy something, usually the first thing I do is to take it apart to see how it works. When I buy something, it's mine. As is usually the case with my computer purchases, the machine was a demo model which has windoze activated so I didn't get hungup at the setup step where I can't go further (I won't agree to M$'s license terms about not disassembling the OS). A bit of tinkering led me to settings where I could turn off all "protection" features which are of absolutely no use to anyone and seem to be based on the premise that the average user surfs virus infested web sites on a regular basis and is too stupid to see when they've downloaded malware. My main antiviral program is in my wetware using information from process explorer, regmon/diskmon as well as Wireshark.

I uploaded my set of hacking tools which, once I found out how to run them in administrator mode, allowed me reasonable access to the machine. WinHex wouldn't let me display all process memory but I did have raw disk access. The reason for the high baseline CPU use turned out to be a huge number of programs that were run on a regular basis that served absolutely no usefull function that I could see. Every minute or so a program would run which scanned all windoze components to ensure that they were "properly licensed". Other programs ran useless "diagnostics" on the machine to update the "windoze experience" number which has absolutely no meaning to me (OTOH, the 2.8 Gflops/core and 5700 MIPS/core and 46 Gflops GPU is meaningfull).

The first 2 days I had the machine were spent in turning off one service after another and disabling programs which had been scheduled to run periodically. This is in marked contrast to Win 2000 or XP which are ready to run after being loaded onto a machine. One process which was exceedingly annoying was the "media server" service which consumed 50% of CPU time for several hours before I terminated it and I have no idea what it was doing but it is yet one more of the useless bits of bloatware that M$ has seen fit to ship with their OS.

Once I had the baseline CPU useage down to a saner value, it was time to start installing VB6 and VC5 onto the machine. I got a compatibility warning about VB6 and, sure enough, the IDE had only very limited functionality on the Sony machine. The display was too large and, while my compiled VB6 programs ran, the text was displayed at too large a size. The settings to change the displayed font don't go below 100% so Sony must assume not only that the average computer user is a moron, but they are blind as well. As my first test I thought I'd change the size of the text labels in the buttons on a simple program. I clicked on the font selection icon in the properties window and waited, and waited and waited until I got fed up and went to process explorer to find out WTF was going on here. VB6 was stuck in an endless loop with a single thread consuming 50% of all CPU time: VB6.EXE!VB_CALLBACK_REVOKE_+0x2EA3F. The only way to get out of this endless loop was to kill VB6 as terminating the offending thread resulted in a non-functional VB. What is even more odd about this is that when I tried an installation of VB6 on a W7 64 bit AMD Athlon 235e based system I had absolutely no difficulty in changing the font size in controls.

An internet search using multiple search engines for "VB_CALLBACK_REVOKE" was notable that there was not one hit for this search term. (Once the googlebots crawl my blog there will be a reference to the term). It confirmed my impression that Sony may make good hardware, but their people shouldn't be allowed near a compiler as any Sony program I've ever used is absolute crap.

What was even more puzzling was that neither of the machines would call functions from an assembly language dll I had written while in interpreted mode, but the compiled program had no difficulty calling this DLL. This seemed to be limited only to my DLL's as references to system DLL's were properly handled in interpreted mode. VC5 seemed to run just fine on both machines and I was able to recompile cpuid.dll which is a library of functions to access the CPU's timestamp counter and CPUID instruction among other things.

Fine, so VB6 didn't run, and then the next step was to install virtualbox on my system and create a W2K image disk and run VB6 on that. It's a bit of a pain to have to pass everything through shared folders but Virtualbox does give me a bit more security as I like to twiddle with arcane system settings and it's easier to restore a virtual machine than to resinstall an OS on a real machine (I use appropriate security measures when they are needed, not indiscriminately).

That problem solved I next wanted to see if I could get hardware virtualization running to speed up my virtual machines even more (despite the fact that in software virtualization the W2K system is way faster than the last hardware W2K system I used) and found, to my disgust, no support for hardware virtualization on the Sony Vaio. Obviously I wasn't the only person really pissed off at Sony about this as the outrage was quite noticeable when I Binged this topic and Sony has relented and come out with a Bios that supports hardware virtualization. I've downloaded it but will wait until I've gotten another UPS for this machine before I reflash the BIOS as the last thing I need is a power failure during the process (again, appropriate security precautions).

Usually when I personalize windoze I don't keep track of what I've done to get the desktop to the way it should be for me but this time I'm documenting the process and will see if I can come up with a script which will take a fresh W7 machine and automatically convert it to one which I want to use not some dumbed down POS.

The other major annoyance I have with M$ (no room to list them all) is that they have totally changed the desktop in W7 compared to XP. At least in XP one could fairly easily go from the annoying XP appearance to a "classic" windoze desktop and the additions were evolutionary. In W7 M$ appears to have changed basic functionality for no other reason that "we can do this if we want". If M$ wants to see how to do real GUI design, take a look at the Mac. I am very impressed that I have no difficulty in using OSX when the last major Mac use on my part was 10 years ago on a MacIIvx. Apple established a set of consistent user interface settings which make interaction with the computer so much easier. Instead of spending time on computationally intensive fluff that no-one will use, M$ would have been far better off to allow for customization of the desktop to suit individual users. I'm 10x more productive on an XP or W2K system because everything is where I expect it to be. Linux GUI's are individually customizable and there is no reason why M$ can't offer the same flexibility

The only reason that I'm still using windoze is VB. Until W7, one of the first steps in getting a new computer was to install VB6 on the machine and then begin coding. For a while the changes which happened were for the better: VB3 was what convinced me to ditch Hypercard, VB4 allowed one to create compiled 16 bit and 32 bit programs (I didn't see much use for VB5) and VB6 was the culmination of progress in development of visual basic. It was no mean feat to take a language which was once sneered at by serious programmers and make it into a powerfull and easy to use development platform for non-time critical applications (for that I write assembly language routines). My early medical software involved lots and lots of input forms and VB was by far the easiest way to produce these quickly.

There is some strange urge within M$ that once they have produced a good product to totally abandon it and to roll out an inferior and incompatible "new and improved" version. I'm referring to VB.NET. I see absolutely no reason for .NET as if I want to run programs in a sandbox I'll use Java or a virtual machine. M$ seems to have a severe case of the "not invented here" syndrome and they came up with the .net framework in response to Sun's success with java. I made the mistake of downloading an evaluation version of M$'s Visual Studio onto one of my machines and it took years to erradicate the last traces of this product. One of the assumptions that the installation program made was that I wouldn't want to keep old versions of VB and VC on my machine and proceeded to cripple them. VB.net is totally incompatible with VB6. I took one small program written in VB6 and spent most of a day attempting to get it to run under VB.net and never succeeded. I've been programming for 40 years and can still run the FORTRAN code that I wrote in high school although I have to retype it as there aren't any punched card readers easily available now. What VB.net requires is that one totally rewrite one's VB6 programs to fit into the procrustean vb.net bed. This seems to be the only solution that numerous people have come up with and it is absolutely assinine. After this experience I started the frustrating process of trying to remove the .net based version of visual studio from my machine and started learning java.

This is already too long for a blog entry but it is possible to go on endlessly about the bad design choices that M$ has made. The thing that I've always liked about software is that if one doesn't like something it is possible to write new code which performs things in the desired manner. M$ seems determined to ensure that such capabilities are not within the hands of the average user. I'm now learning about the architecture of W7 which seems to basically be NT at the core with an absolutely idiotic number of non-essential additions. Part of this is to run DRM software as part of the OS but this is easily countered and I use non-M$ DVD players and TV card interfaces. I like the touchscreen interface as it is very intuitive and as soon as a version of Linux comes out that can run on this machine I'll be ditching W7 and running Linux instead. I suspect I'll still have to keep W7 around as the bulk of the population can't concieve of any other OS besides windoze and I'll always have to deal with this idiotic OS or some bastard progeny of same either at the hospital or in my clinic. I've cracked security on every dumbed down version of windoze that has thus far come out and don't expect W7 will be any different. True, the security does get better with every iteration, but I like tough problems. Once I have the highest level of privileges available on the system (which is not Administrator) I'm happy and can get on with doing some real work with the system.

Posted by Boris Gimbarzevsky at 19:44.31
Edited on: 22/01/2010 21:46.09
Categories: Computers