[ Home | Articles | Account | People | Projects | FAQ ] Name: Russell Coker
Member since: 2001-02-19 14:53:50
Last Login: 2009-02-24 04:55:31
Homepage: http://etbe.coker.com.au/
Notes: I do general Linux programming and sys-admin work. I am mostly known for my work on NSA Security Enhanced Linux.
DomainKeys and OpenSSL have Defeated Me
I have previously written about an error that valgrind reported in the STL when some string operations were performed by the DKIM library [1]. This turned out to be a bug, Jonathan Wakely filed GCC bug report #40518 [2] about it, Jonathan is one of many very skillful people who commented on that post.
deb http://www.coker.com.au lenny gcc
I’m still not sure whether that bug could actually harm my program, Nathan Myers strongly suggested that it would not impact the correct functionality of the program but mentioned a possible performance issue (which will hurt me as the target platform is 8 or 12 core systems). Jaymz Julian seems to believe that the STL code in question can lead to incorrect operation and suggested stlport as an alternative. As I’m not taking any chances I built GCC with a patch from Jonathan’s bug report for my development machines and then built libdkim with that GCC. I created the above APT repository for my patched GCC packages. I also included version 3.4.1 of Valgrind (back-ported from Debian/Unstable) in that repository.
Nathan Myers also wrote: “Any program that calls strtok() even once may be flagged as buggy regardless of any thread safety issues. Use of strtok() (or strtok_r()) is a marker not unlike gets() of ill thought out coding.” I agree, I wrote a program to find such code and have eliminated all such code where it is called from my program [3].
I think it’s unfortunate that I have to rebuild all of GCC for a simple STL patch. My blog post about the issue of the size and time required to rebuild those packages [4] received some interesting comments, probably the most immediately useful one was to use --disable-bootstrap to get a faster GCC build, that was from Jonathan Wakely. Joe Buck noted that the source is available in smaller packages upstream, this is interesting, but unless the Debian developers package it in the same way I will have to work with the large Debian source packages.
I have filed many bug reports against the OpenSSL packages in Debian based on the errors reported by Valgrind [5]. I didn’t report all the issues related to error handling as there were too many. Now my program is often crashing when DomainKeys code is calling those error functions, so one of the many Valgrind/Helgrind issues I didn’t report may be the cause of my problems. But I can’t report too many bugs at once, I need to give people time to work on the current bug list first.
Another problem I have is that sometimes the libdkim code will trigger a libc assertion on malloc() or free() if DomainKeys code has been previously called. So it seems that the DomainKeys code (or maybe the OpenSSL code it calls) is corrupting the heap.
So I have given up on the idea of getting DomainKeys code working in a threaded environment. Whenever I need to validate a DomainKeys message my program will now fork a child process to do that. If it corrupts the heap while doing so it’s no big deal as the child process calls exit(0) after it has returned the result over a pipe. This causes a performance loss, but it appears that it’s less than 3 times slower which isn’t too bad. From a programming perspective this was fairly easy to implement because a thread of the main program prepares all the data and then the child process can operate on it – it would be a lot harder to implement such things on an OS which doesn’t have fork().
DomainKeys has been obsoleted by DKIM for some time, so all new deployments of signed email should be based on DKIM and systems that currently use DomainKeys should be migrating soon. So the performance loss on what is essentially a legacy feature shouldn’t impact the utility of my program.
I am considering uploading my libdomainkeys package to Debian. I’m not sure how useful it would be as DomainKeys is hopefully going away. But as I’ve done a lot of work on it already I’m happy to share if people are interested.
Thanks again for all the people who wrote great comments on my posts.
Web Hosting After Death
Steve Kemp writes about his concerns for what happens to his data after death [1]. Basically everything will go away when bills stop being paid. If you have hosting on a monthly basis (IE a Xen DomU) then when the bank account used for the bill payment is locked (maybe a week after death) the count-down to hosting expiry starts. As noted in Steve’s post it is possible to pay for things in advance, but everything will run out eventually.
One option is to have relatives keep the data online. With hard drives getting bigger all the time it wouldn’t be difficult to backup the web sites for everyone in your family to a USB flash device and then put it online at a suitable place. Of course that relies on having relatives with the skill and interest necessary.
The difficult part is links, if the domain expires then links will be broken. One way of alleviating this would be to host content with Blogger, Livejournal, or other similar services. But then instead of the risk of a domain being lost you have the risk of a hosting company going bankrupt.
It seems to me that the ideal solution would be to have a hosting company take over the web sites of deceased people and put adverts on them to cover the hosting costs. As the amount of money being spent on Internet advertising will only increase while the costs of hosting steadily go down it seems that collecting a lot of content for advertising purposes would be a good business model. If the web sites of dead people are profitable then they will remain online.
It wouldn’t be technically difficult to extract the data from a blog server such as Wordpress (either from a database dump or crawling the web site), change the intra-site links to point to a different domain name, and then put it online as static content with adverts. If a single company (such as Google) had a large portion of the market of hosting the web sites of dead people then when someone died and had their web site transferred the links on the other sites maintained by the same company could be automatically adjusted to match. A premium service from such a company could be to manage the domain. If they were in the domain registrar business it would be easy to allow someone to pay for 10 or 20 years after their death. Possibly with a portion of the advertising revenue going towards extending the domain registration. I think that this idea has some business potential, I don’t have the time or energy to implement it myself and my clients are busy on other things so I’m offering it to the world.
Cory Doctorow has written an article for the Guardian about a related issue – how to allow the next of kin to access encrypted data when someone is dead [2]. One obvious point that he missed is the possibility that he might forget his own password, a small injury from a car accident could cause that problem.
It seems strange to me that someone would have a great deal of secret data that needs strong encryption but yet has some value after they are dead. Archives of past correspondence to/from someone who is dead is one category of secret data that is really of little use to anyone unless the deceased was particularly famous. Probably the majority of encrypted data from a dead person would be best wiped.
For the contents of personal computers the best strategy would probably be to start by dividing the data into categories according to the secrecy requirements. Publish the things that aren’t secret, store a lot of data unencrypted (things that are not really secret but you merely don’t want to share them with the world), have a large encrypted partition that will have it’s contents lost when you die, and have a very small encrypted device that has bank passwords and other data that is actually useful for the executors of the will.
One thing that we really need is to have law firms that have greater technical skills. It would be good if the law firms that help people draw up wills could advise them on such issues and act as a repository for such data. It seems to me that the technical skills that are common within law firms are not adequate for the task of guarding secret electronic data for clients.
Valgrind and OpenSSL
I’ve just filed Debian bug report #534534 about Valgrind/Helgrind reporting “Possible data race during write” [1]. I included a patch that seems to fix that problem (by checking whether a variable is not zero before setting it to zero). But on further testing with Valgrind 3.4.1 (backported from Debian/Unstable) it seems that my patch is not worth using, I expect that Valgrind related patches won’t be accepted into the Lenny version of OpenSSL.
I would appreciate suggestions on how to fix this, the problem is basically having a single static variable that is initialised to the value 1 but set to 0 the first time one of the malloc functions is called. Using a lock for this is not desirable as it will add overhead to every malloc operation. However without the lock it does seem possible to have a race condition if one thread calls CRYPTO_set_mem_functions() and then before that operation is finished a time slice is given to a thread that is allocating memory. So in spite of the overhead I guess that using a lock is the right thing to do.
deb http://www.coker.com.au lenny gcc
For the convenience of anyone who is testing these things on Debian and wants to use the latest valgrind, the above Debian repository has Valgrind 3.4.1 and a build of GCC to fix the problem I mentioned in my previous blog post about Valgrind [2].
if (default_RSA_meth == NULL)
default_RSA_meth=RSA_PKCS1_SSLeay();
I have also filed bug #534656 about another reported race condition in the OpenSSL libraries [3]. Above is the code in question (with some C preprocessor stuff removed). This seems likely to be a problem on an architecture for which assignment of a pointer is not an atomic operation, I don’t know if we even have any architectures that work in such a way.
static void impl_check(void) {
CRYPTO_w_lock(CRYPTO_LOCK_EX_DATA);
if(!impl)
impl = &impl_default;
CRYPTO_w_unlock(CRYPTO_LOCK_EX_DATA);
}
#define IMPL_CHECK if(!impl) impl_check();
A similar issue is my bug report bug #534683 [4] which is due to a similar issue with the above code. If the macro is changed to just call impl_check() then the problem will go away, but at some performance cost.
I filed bug report #534685 about a similar issue with the EX_DATA_CHECK macro [5].
I filed bug report #534687 about some code that has CRYPTO_w_lock(CRYPTO_LOCK_EX_DATA); before it [6], so it seems that the code may be safe and it may be an issue with how Valgrind recognises problems (maybe a Valgrind bug or an issue with how Valgrind interprets what the OpenSSL code is doing). Valgrind 3.3.1 reported many more issues that were similar to this, so it appears that version 3.4.1 improved the analysis of this but didn’t do quite enough.
I filed bug report #534706 about the cleanse_ctr global variable that is used as a source of pseudo-randomness for the OPENSSL_cleanse() function without locking [7]. It seems that they have the idea that memset() is not adequate for clearing memory. Does anyone know of a good research paper about recovering the contents of memory after memset()? I doubt that we need such things.
I filed bug report #534699 about what appears to be a potential race condition in int_new_ex_data() [8]. The def_get_class() function obtains a lock before returning a pointer to a member of a hash table. It seems possible for an item to be deleted from the hash table (and it’s memory freed) after def_get_class() has returned the pointed but before int_new_ex_data() accesses the memory in question.
I filed bug report #534889 about int_free_ex_data() and int_new_ex_data() which call def_get_class() before obtaining a lock and then use the data returned from that function in a locked area[9] (it seems that obtaining the lock earlier would solve this).
I filed bug report #534892 about another piece of code which would have a race condition if pointer assignment isn’t atomic, this time in err_fns_check() [10]. In my first pass I didn’t bother filing bug reports about most of the issues helgrind raised with the error handling code (there were so many that I just hoped that there was some subtle locking involved that eluded helgrind and my brief scan of the source). But a new entry in my core file collection suggests that this may be a problem area for my code.
I think that it is fairly important to get security related libraries to be clean for use with valgrind and other debugging tools – if only to allow better debugging of the code that calls them. I would appreciate any assistance that people can offer in terms of fixing these problems. I know that there are security risks in terms of changing code in such important libraries, but there are also risks in leaving potential race conditions in such code.
As an aside, I’ve filed a wishlist bug report #534695 requesting that valgrind would have a feature to automatically add entries to the suppressions file [11]. As a function that is considered to be unsafe can be called from different contexts, and code that is considered unsafe can be in a macro that is called from multiple functions there can be many different suppressions needed. Pasting them all into the suppressions file is tedious.
Microsoft Open Source Information Evening
I have just attended a Microsoft Open Source Information Evening. It was in some ways one of the stranger things that I have experienced in my computer career.
Firstly there was the location, it was in a function room in the CBD, it was convenient for public transport and had good service but seemed likely to be quite expensive. A MS employee said that they believed that some people wouldn’t want to enter an MS office – I can’t imagine why they think that they could convince people who refuse to enter the MS office of anything if they got them to attend. As there were only about 6 people who weren’t from MS it seems likely that they paid something in excess of $200 per head for each non-MS delegate (I can’t imagine two function rooms, two dedicated hotel employees manning the bar, and a supply of food for a larger audience costing less than $1200).
If they had spent $100 per head for us all to have dinner at a good restaurant then I think that the result would have been better. They might want to consider running targeted meetings in future with a small number of people personally invited to dinner at a good restaurant. That said, the dinner of duck canapes and asian-style chicken noodles that they provided was pretty good.
I suggested that they should find other ways of promoting such events as the audience was obviously smaller than they desired. One suggestion that I made was that they create a blog about what MS in Australia is doing in relation to Linux and to offer the RSS feed URL to the people who run Planet Linux Australia. They were reluctant to accept that idea and stated that they don’t want to be seen to be forcing their presence where they are not wanted. That is a good approach (and a contrast to some activities of MS in the past). But I believe that it is misguided in terms of RSS feeds. When you create a blog you make the RSS feed available and then the people who run syndication services have the option of using it. The Linux community is on the side of open discussion, I don’t think that we have anything to fear from hearing what MS people have to say. While my opinion of MS has improved this evening, I still have no interest in using any of their software. Linux just works really well and satisfies all of my needs.
There were a bunch of smart MS people there, they seemed to really care about their work and want to improve things. Their pitch was about how Open Source software works on Windows, they showed demos of the installation process for a variety of PHP programs and showed Python code being used in a MS web environment. Most of the presentation time involved technologies developed outside of MS, while there was obviously a lot of MS code involved in getting Python, Ruby, PHP, etc working well the focus was mostly on the free software. They also mentioned some of their work in opening APIs so that free software programs can access Exchange servers (among other things). I didn’t pay a great deal of attention to the technology as I’m never going to use it. I was more interested in their approach which was positive and respectful and the general trend of what they are doing.
It seems that there is an increasing number of people within MS who realise that free software is not going away and that their customers demand that things work together.
They also didn’t display any of the arrogance for which MS is known. When one of the delegates predicted that MS would take a fall the way IBM did there was no argument about that possibility, instead there was a discussion about how MS software can be used with software from other sources to meet the current and future needs of customers.
The discussion of software patents was generally not very productive, I got the impression that they were not permitted to give anything that I would have considered to be a good answer to any of the questions. They did show examples of software that they have released with RAND terms for patents and other situations in which there would be no patent liabilities. But it seems that MS as a whole has no interest in getting any of the patent problems fixed. I can only hope that IBM, NEC, or one of the other big patent companies will give MS a demonstration of why software patents are bad.
Finally I was given a couple of 8GB USB sticks and a copy of MS Expression Studio 2. If anyone wants the unopened copy of Expression Studio they can make me an offer by email.
Unreasonably Large Source Packages
For the past few hours I’ve been going a build of the GCC packages on a dual-core Opteron system with 2.5G of RAM and a pair of reasonably fast SATA disks in a RAID-1 array. The machine is reasonably powerful so presumably such a build would take a significantly larger amount of time on a laptop or an older machine – my primary development machine is an old laptop and is thus unsuitable for such things.
My aim is to do a build with the patch for GCC bug 40518 [1] – which is a small patch to the STL.
Presumably the people who are seriously involved in GCC development don’t do this, they would be doing a build of a small sub-set that matches the code that they are working on. But as someone who is not involved in the project such an approach doesn’t seem viable, by using the Debian build tools to rebuild all packages from the source package I can reliably get a good build.
It would be convenient if these large source packages could be split into smaller packages. It shouldn’t be necessary to compile the C compiler (presumably with the full double-compile process), as well as the C++, Objective C, and Fortran compilers when I only want to compile the STL (libstdc++). It also shouldn’t be necessary for me to hack around a build system when all I want to do is to apply and test a single patch.
It seems to me that the current situation discourages contributions. If I can build a package in a reasonable amount of time on my laptop (Pentium-M 1.7GHz with 1.5G of RAM) then I can work on it at any time and in any place. If it requires hours of build time on my biggest machine then I can only work on it when at home and only when I have hours to spare (or if I have enough of a need to come back to it the next day).
So if there are two bugs that have equal importance to me and one of them happens to be in part of the GCC family then the probability that I will work on the GCC bug is close to zero.
I realise that packaging GCC etc is really hard work. But it seems that making it easier for more people to contribute would alleviate the burden slightly.
etbe certified others as follows:
Others have certified etbe as follows:
[ Certification disabled because you're not logged in. ]
FOAF updates: Trust rankings are now exported, making the data available to other users and websites. An external FOAF URI has been added, allowing users to link to an additional FOAF file.
Keep up with the latest Advogato features by reading the Advogato status blog.
If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!
del.icio.us
Digg
Google bookmark
reddit
Simpy
StumbleUpon
Furl
Newsvine
Technorati
Tailrank