28 Sep 2009 uriel   » (Journeyer)

While profiling Atrack and after writing some code to monitory the live memcache usage stats and display some pretty graphs, I noticed all the CPU time was spent picking and unpicking stuff from memcache, this is really expensive in App Engine because cPickle is disabled, and you are forced to use the excruciatingly slow pure python pickle module.

So for now I used a bunch of calls to string.split() and string.join() to serialize my data into a string before feeding it to memcache, this seems to improve things considerably and my CPU quota can breathe for now.

Next I would like to benchmark how well various serialization methods perform in App Engine, I have seen some comparisons of protocol buffers, json, etc, but I suspect the results wont apply to the pure-python environment in App Engine, and I think none included bencode (which I have learned to love after having to use to implement the bt tracker protocol).

In related news, I built an App Engine based site that monitors open trackers, used mako for the templates, which turned out to be bearable.

Latest blog entries     Older blog entries

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!