While profiling Atrack and after writing some code to monitory the live memcache usage stats and display some pretty graphs, I noticed all the CPU time was spent picking and unpicking stuff from memcache, this is really expensive in App Engine because cPickle is disabled, and you are forced to use the excruciatingly slow pure python pickle module.

So for now I used a bunch of calls to string.split() and string.join() to serialize my data into a string before feeding it to memcache, this seems to improve things considerably and my CPU quota can breathe for now.

Next I would like to benchmark how well various serialization methods perform in App Engine, I have seen some comparisons of protocol buffers, json, etc, but I suspect the results wont apply to the pure-python environment in App Engine, and I think none included bencode (which I have learned to love after having to use to implement the bt tracker protocol).

In related news, I built an App Engine based site that monitors open trackers, used mako for the templates, which turned out to be bearable.

Been playing lately with App Engine. My first experiment is Atrack, a (BitTorrent) tracker that relies purely on memcached, it is fast and efficient, and less than 100 lines of code.

There is demo open installation in appspot, add http://bittrk.appspot.com/announce (or if you are paranoid https://bittrk.appspot.com/announce) to any torrents you like, I could use the stress testing!

So far I'm tracking a few thousand torrents (after torrentfreak broke the news) and barely using %1 of the App Engine quota.

Of course one of the cool things about atrack is that there is no way for me to really what torrents or how many torrents exactly are being tracked! This makes me happy but it is hard to shake the curiosity to know who and what for people are using your code! Ah, the wonders of privacy :))

While reading on the Bittorrent tracking protocol I realized that the problem it solves is much more general and a subset could easily be used for decentralized online gaming, chat systems, and to help other 'grid' like applications find each other inside the cloud.

So I started to write up an spec for Ntrack, a generalized and simplified tracker scheme, that should be backwards compatible with bittorrent clients and trackers, but makes all the bt-specific functionality optional. This is pretty much the subset implemented by atrack; I'm working on a distributed chat system that uses it and I hope others come up with other apps that take advantage of this.

P.S.: I know others have built trackers on app engine in the past, but they seem to be out of quota most of the time, and by looking at the source this is not surprising given how over-engineered they are. *sigh*

Share and enjoy.

This year Plan 9 from Bell Labs is a mentoring organization at the Google Summer of Code!

We are interested in all kinds of projects related to Plan 9, Plan 9 from User Space, 9P and Inferno. We have a list of ideas but we are happy to help you come up with a project that suits your interests and skills best.

Aside from the 4500$ provided as part of GSoC, we will send all participants a fantastic Glenda T-Shirt!

Hurry up, the deadline to apply is the 24 of March!

The first Swedish wmii9con will take place the 18/19 of February in Stockholm.

The main aim is to have fun and meet other Plan 9 and wmii users. It will be more more of a party with a bunch geeks than any sort of conference, but if you are interested in learning more about Plan 9 feel free to drop by and have some drinks with us.

I also will be speaking at FOSDEM the following week and a bunch of 9fans will co me along.

In other news the new Plan 9 web site has been quietly launched which much cleaner design and the full "sources" source tree is available over the web. We also have a more friendly address: 9fans.net

The new website is generated from troff(!) source using a new tool htmlroff , it still needs a bit of polishing but works great and the html version the papers look much nicer, for example see the Security in Plan 9 paper.

I have been quite busy with various things, but I promise to try keeping this diary more updated.

Experimental Plan 9 from Bell Labs changes and patches mailing list: plan9changes

The archives have been populated with most of the changes of the last few months to have a pseudo-changelog.

It even has RSS and Atom feeds for those that like that kind of thing, not me ;)