robustness testing
In a recent weird moment in #swfdec, we decided we wanted to do memory allocation robustness testing in Swfdec. Swfdec has some beginnings of a memory management facility, which is similar to malloc returning NULL, but is supposed to be able to live in a glib world: If you need memory, you ask if it is available by calling wfdec-as-context-use-mem. In theory that function returns FALSE if your Flash file used up too much memory. Of course, those failure paths were very untested, because returning FALSE has not been implemented. So what we needed was a function, that would be able to return FALSE at any allocation, just like being able to return NULL at some specific malloc.
I knew there were some projects that do rigorous memory checking in their test suites, so I had a look at how they do it. Apparently, there doesn’t exist a tool for doing that, as everybody has their own solution. Cairo for example uses Chris Wilson’s memfault valgrind skin, which injects malloc failures into running code (comand line options). D-Bus uses evironment variables for practically the same thing:
Both allow you to specify to return NULL from the Nth malloc (with N specified by you). So to get complete coverage of your error handling code, you need to run your program again and again with N increased by one until you have failed for every malloc. That is very slow, as some of the Flash files we ran do 150.000 allocations - and we don’t even do good memory management yet, I expect that to increase by a factor of 10 once we refactor this. And it’s extra slow, if you run it in a valgrind skin. So we needed a faster solution.
The problem that takes so much time is of course easy to identify: You need to run all the code with succeeding malloc calls again and again until you arrive at the failure. What would be a lot faster is being able to return NULL from malloc, see if everything turned out ok, and then continue normally from the malloc. It turns out this is easy. It took 100 lines of code to do this for the above-mentioned swfdec memory function. What I do is forking the process on every malloc. The child returns NULL, the parent continues successfully. That way, you spawn a child for every malloc failure and the child just runs the failure handler. All usual code is only run once - in the parent.
There’s many things that are great using this approach. For a start, it is a lot faster. And not only that, it scales linearly with the amount of allocations done as opposed to the old approach, which was O(N^2). It also provides an easy way to do bookkeeping: the parent reaps all the children and checks if they exited succesfully, otherwise it prints an error message. So you even get a nice output.