I'm in Potsdam, at the Hasso-Platner Institute, attending the Workshop on Self-Sustaining Systems. Sounds esoteric? I thought so, too, though that didn't stop me from submitting a paper about the build process of Steel Bank Common Lisp and how that affects the user and developer community.

The call for contributions invited

submissions of high-quality papers reporting original research, or describing innovative contributions to, or experience with, self-sustaining systems, their implementation, and their application.

and my paper, as well as the technical details of the SBCL build, essentially argued for considering the user and developer community as part of the system: software is really just a parasite, but its host is not the hardware it's running on, but the humans who allow it to ‘reproduce’. (A related point was made by Ian Piumarta of the Viewpoints Research Institute, about the survival of the species rather than the individual being the important thing, in the talk before mine).

My talk was a bit rushed and confused, I think; it suffered from insufficient preparation time. What I was utterly flabbergasted by was the attendance: I was expecting a complement of about 12 people, maybe mostly the authors and their colleagues. Instead, there were about 70 people at the opening, and while some of those were local PhD students or other easy prey, there were still about 50 people attending the research talks. If I had expected that, I might have adjusted the material and the presentation a little bit, perhaps with a bit more advocacy (a captive audience is valuable, after all!), but at least in response to my audience poll, very few people confessed to being unfamiliar with Lisp, so maybe the advocacy would have been unnecessary.

In any case, I got some fair feedback, and I had the chance to have a nice chat with Kim Rose about the use of Squeak Smalltalk and related tools in education. Luke Gorrie introduced me to some other of his Smalltalk chums – that did seem to be the most popular language, based on a highly informal sample.

Dan Ingalls gave a demo of the Sun Labs Lively Kernel, which frankly makes my idea of writing a web backend for McCLIM simultaneously achievable and passé; a whole Morphic system built on SVG and Javascript, complete with animated stars and clocks with Roman numerals, is just the ticket at 9 o'clock in the morning. “Runs best in Safari 3 or Firefox 3.0 beta 5”.

The other talks were about implementing 3-lisp (a reflective dialect of Lisp), a Squeak VM in PyPy, Huemul Smalltalk, and Pico (an interpreted Lisp). It's very interesting to see the development of these systems; my main problem is that many of the presenters say things like “this implementation is not efficient (yet)” or “this implementation is incomplete at the moment” – this makes it very hard to judge whether the necessary future development will invalidate some of the nice things about the systems, such as their size, speed or dynamicity. I suppose that's in the nature of the research beast, though.

After all that, and some more chats, time to head off home (briefly, to rehearse for Arne's Judgment of Paris in the English Music Festival) before setting off again to Bordeaux and the European Lisp Symbolsium.

I spent the weekend in Amsterdam, doing ordinary touristy things on the days when I wasn't attending the 2008 edition of the European Common Lisp Meeting. I suspect that most readers of this diary are likely to prefer a summary of the ECLM to the details of my wanderings on the (rather cold) Saturday, so that is what I shall provide.

Walking through the streets of continental cities early (around 08:00) on a Sunday Morning is usually a pleasant experience – everything is closed, most people aren't even yet going to appropriate religious observances, and so you get an unimpeded view of the town, devoid of the hordes that can cause obstruction or aggravation. Unfortunately, Amsterdam is a somewhat popular destination for British ‘stag’ parties; and, indeed, as I was walking down Leidsestraat, two such parties, who did not seem to be the sort to be going either to an appropriate religious observance or to the ECLM, passed each other. Noise ensued.

On arriving at the Felix Meritis, at 08:30 on the dot, I saw a crowd of people, all seemingly with the same idea: to have the first coffee of the day. Fortunately, the doors opened shortly afterwards, and caffeination was allowed to take place. Conversation happened as well, though I can't remember very much of it, not yet having been fully invigorated by the coffee. Mostly I think I explained why I wasn't on the previous night's boat trip: a combination of somewhat tardy registration and a second mouth to feed...

And then the real action, the talks, got under way. Unlike many an academic conference, of of the ECLM's distinguishing features is that the talks are attended by the vast majority of the participants; there's much less of a feeling that people simply attend to chat to colleagues: probably partly because the talks are very much the ‘deliverable’ of the meeting (there are no proceedings or anything like that) but also, I think, because the talks cover interesting ground, and offer perspectives based on a solid amount of experience.

Jeremy Jones (from Clozure Associates) started the ball rolling, with a talk on the production of InspireData: an application built for data visualization in an (American, pre-University) educational context. Jeremy wisely started off with a demonstration of InspireData's features; it contains some very impressive-looking tools, and seems to present them to the user in a sensible way. I haven't tried it or gone beyond the demo, but it looks like a huge advance on, say, using an off-the-shelf spreadsheet program to do data analysis, even (dare I say it) for professionals – though whether it scales up to professional-sized data sets is another question. Some other take-home messages from Jeremy's talk: it is possible to sell shrink-wrapped software, even today, even written in Lisp; having a proper designer on the team (or as your client) can help enormously in producing a usable interface; and having a programmer as your client can be both a help and a hindrance – specify the acceptance process carefully.

Nicholas Neuss followed Jeremy, with a discussion of the FEMLISP framework for solving partial differential equations. Being in a somewhat darkened room, early in the morning at a weekend reminded me a little of my undergraduate days, where discussion of differential equations, fields and the like was par for the course – and the talk took me right back (in a good way). In particular, watching FEMLISP compute the eigenmodes of Lake Constance (surface waves, I think) was entertaining. One difference in kind between this talk and the previous is, I think, a function of the possible ‘market’ for the two tools; InspireData is sold in quantities of the order of tens of thousands at present, while, let's face it, FEMLISP is never going to have that kind of exposure: and so the resources aren't really available to make FEMLISP into a product usable by even other domain experts. Of course, the fact that Nicholas' boss makes a competing product might also have something to do with that...

There followed a talk about large Internet systems, from Stefan Richter. There was an interesting survey during the talk, asking people about the size of their userbase (assuming that they worked on web applications at all). An interesting distribution; certainly applications with millions of users are no longer rare – and Juho showed commendable restraint in not snorting something along the lines of “millions of users?” One other interesting moment was when Nick Levine stopped Stefan, to give him time to write down the long list of libraries that is already available to help build Lisp web applications, from the database to the front-end.

Kilian Sprotte gave the last presentation before lunch, talking about the GL-enabled Patchwork music visual programming system. The presentation unfortunately appeared to be a little bit unrehearsed, and so I'm not sure that Kilian got everything that he wanted to across to the general audience. From my perspective, though, it was sufficiently close to the day job that I could see the point (and catch some of the references; B-A-C-H and so on) – I'll be able to report back to people in the lab, who were asking about it, and maybe we can find some useful musical analysis algorithms in there. Also, it occurred to me that GSharp could usefully provide some import or export functionality for the chord and score editors, maybe through MusicXML, or maybe writing directly into the PWGL notations for notes (tighter couplings are likely to be hard, given PWGL's current non-Open-Source nature.)

Then we broke for lunch; Juho, Nikodemus and I ended up sitting on a table with Hannes Mehnert and Luke Gorrie; among other conversations, we had The Great SBCL Maintenance Debate, and we now have a proposed Plan. (I don't know if any other plans have been proposed, but hopefully soon there will be less uncertainty.) The rest of the lunch break was spent grilling Luke about OLPC and Kathmandu, and eating interesting interpretations of café food. (Pasta arrabiata with beans and squash? I don't think so.)

After lunch, a talk about cheap apartment design architect assistance software: Knowledge-Based Engineering, where in this case the knowledge base is about Norwegian building regulations. The idea of the House Designer product developed at Selvaag is to allow architects to experiment with designs, while tracking that building rules (both governmentally-imposed and the house style) can be accommodated, along with all the necessary pipes and electrics and so on. It was good to see a variety of techniques on display, and I liked the flashes of humour: for instance, that the user-interface group asking for XML descriptions was OK by the Lisp group, but that then the user-interface group wanted to send XML back, and that was not OK. It's good to know that Frode Fjeld has a Lisp job, too; he was solidly namechecked in the presentation.

Then Juan José García-Ripoll came to the stage to talk about ECL design and implementation. There were some wry moments for me there; starting with the observation that he wasn't really a computer scientist at all, but rather a physicist. Also, many of the motivations for the ECL design appear to be direct analogues of some of SBCL's PRINCIPLES: completeness, clean bootstrappability, and preferring maintainability over whizzy features, for instance (I hope I'm not mischaracterizing here; it's possible I'm just evaluating what he said through an SBCL-tinted lens). Given that, I think it's interesting how different the two systems look, maybe just from having different starting points?

After the final coffee break, yet more talks. Marc Battyani, of FractalConcept HPC Platform, talked about using Lisp to program FPGAs for custom high-performance solutions; in particular, applications in the financial world, such as derivative valuations and automatic share trading platforms. (A word of advice to Marc: it's not often that you get to say that you're 1000 times as fast as your nearest competitor, nor that you can process packets faster than the test network can send them to you – so don't let that message get hidden by the constant fumbling for a particular Microsoft Image and Fax viewer window!) The products he has look interesting; best of luck to him for finding a buyer in these more financially-challenged times...

And finally, the pièce de résistance: Kenny Tilton took the stage, to give “a rant on the state of Lisp and Lispniks touching on Algebra software, Lisp libraries, Open Source, Cello, Cells, and somewhere along the way introducing Triple-Cells, animated data modelling with persistence for free”. Unfortunately, it seemed that we weren't going to get this rant; instead, we got a small demo of the Algebra tutoring software, along with some discussion of the dataflow paradigm that Kenny believes is central to all simple applications, and a fair number of sound effects. Nice anecdote about Lisp and speeding tickets, though.

Then it was all over bar the dinner; I chatted to Pascal Costanza and Charlotte Herzeel about the busy workshop season, to Jeremy Jones and Marco Baringer about the (lack of) checkin policy to SBCL's CVS, to Edi about how much Heathrow Airport sucks, among many conversations. As we were about to call it a night, Nick mentioned that the Lambda Express had some spare tickets, and that we could probably hitch a lift back to London by train (rather than by plane to Terminal 5...), so we arranged to meet Dave Fox and his crew the following morning. While on the train, in a spirit of cross-implementation co-operation, we essentially finished the Araucaria from the Saturday Guardian – documentary evidence will be forthcoming. And then it really was all over.

Long time no blog. As ever, there are a number of reasons for this: complicated employment situation; too much work to do... but there's also the slightly addictive personality to consider. Since I've already been here once before, it should have come as no surprise to me that I could become engrossed in beating a roguelike game; in the end, I did beat Tales of Middle Earth (ToME) into submission, as I did with Nethack two years ago. (Two years? Well, consider that the only solid playing time is around Christmas, and then learn that last year I almost beat ToME into submission...)

“Complicated employment situation”, I hear you ask? The life of a junior academic researcher, while rewarding, often feels precarious: my main project research funding ran out at the end of May, and since then I have been employed on a succession of temporary research contracts. One thing this has led me to discover is that I am just not very good at having the next thing lined up before the current thing expires. Another is that I do not like the scary feeling of not knowing what I am doing next month – or perhaps more pertinently, where the next paycheque is coming from.

The good news is that this has now resolved itself; I have accepted a lectureship at Goldsmiths, a part of the University of London. (For readers not fully versed in the UK Higher Education terminology, this is a Real, Permanent Job involving a mixture of teaching, personal research and filling in lots and lots of forms.) I work in a group specializing in investigating sound and musical processes, from analysis and synthesis, through cognition, designing systems for cataloguing musical artifacts, all the way to automated Music Theory.

So what do I do there? My primary research responsibility is to the OMRAS2 project, which aims to do all sorts of clever things for recognizing and searching music: current fingerprinting services work on exact (but possibly degraded) copies of notionally entire tracks, whereas I have an interest in automatically deducing musical structures both from audio and from notation, and in working with approximate kinds of similarity (detecting that a track is a remix of another, or a cover version, and so on).

One of the reasons that I like this group so much, though, is that when I first met them, one of the PhD students (who is still with us, in a more advanced capacity) was working on his laptop using CMUCL and ILISP (this was late 2003, so SLIME was if not a glimmer in the mind of Luke Gorrie, at least not the clear choice for development on Free Common Lisps as it is today). This means that my interests in the engineering and language design aspects of SBCL development can go hand-in-hand with attempting to come up with protocols for interacting with musical corpora in many, many different formats: attempting to make it so that the same routines can operate both on representations of lute tablature and on a database of MIDI files designed for karaoke machines, being used to investigate memory for music. I finally have the excuse to use gsharp at work, too!

Speaking of gsharp, Brian Gruber handsomely completed his Summer of Code project, to add MusicXML import and export facilities. His work has been integrated into gsharp's CVS repository, and is being used and further developed here at Goldsmiths, and I got a T-shirt from Google. If you're reading: thanks, Brian.

It's not quite “all quiet on the SBCL front”, but all of the rest of my life leaves me with less time to spend on it than I would like. I did manage to restore support for the alpha architecture, which had bitrotted a little in the last year; this puts me in a reasonable position to finish up my work on improving the interaction between modular arithmetic and representation selection. In addition, I have been pushing forward a little bit the work that I reported on at ECOOP last year regarding user-defined subclasses of specializer; stay tuned for more details. (Maybe sooner than in six months' time!)

It's been a while. Lots of interesting[*] stuff to talk about. Since my previous diary entry was about my work for the European Lisp Workshop, why not start with my recollections and observations about that?

The workshop itself was one day long, and had one invited talk, five papers presented (one of which was mine) and a more informal demo / presentation of current advances in ContextL (a Lisp implementation of Context-Oriented Programming), given by Pascal Costanza.

The invited talk was given by Alexander Repenning, and was titled “Antiobjects” (and subtitled “Mapping Game AI to Massively Parallel Architectures using Collaborative Diffusion”, for what that's worth). The central idea, I think, was to convert a goal-oriented architecture – expressing behaviour of agents in the system in terms of what their goals are, in what is maybe the typical object-oriented programming style – into an optimization problem, viewing the system as a whole.

This change of perspective (the “Antiobjects” of the title allows for some interesting emergent properties. The compelling example Alexander gave was for ghosts chasing Pacman: the difficult bit in the traditional object-oriented style is in implementing collaborative strategies: if there are two ways to reach Pacman, and two ghosts, you (the evil games programmer) would like the two ghosts to take one way each. Generalising for large values of “two” is tricky. The Antiobject solution is instead to define a Pacman field, which diffuses away from the Pacman source, and have the ghosts be sinks of that field. Then the ghosts simply need to do hill-climbing of the Pacman field, and the desired behaviour of collaborative-Pacman-eating appears as if by magic.

Sébastien Mosser gave a talk about meta-models which, I have to confess, I didn't understand at all. I don't know what a “model” is, let along a “meta-model”, and unfortunately Sébastien didn't really give me an idea why I should care. I welcome enlightened discussion on this, if I should in fact care. The next presentation, reporting work by Pierre and Simon Thierry, was a general introduction to transparent persistence, which (as Pierre said) is almost a textbook example of using the Metaobject Protocol. The picture is slightly clouded because unfortunately update-instance-for-redefined-class is not quite able to cope with multiple class redefinitions in between accesses to instances. I believe I have a solution for this issue, but the margin of this blog is too small to contain it. (In practice, people work around the problem without too much pain: I discussed this at the workshop with not only Pierre, but also Arthur Lemmens of rucksack fame.

After lunch, I gave my talk about user-extensible specializers, details about their implementation, and examples of their use. In the course of giving examples, I inevitably made the mathematician's mistake of attempting to persuade the audience that simplifying (* _x 0) to _x was a good idea, but apart from that I don't think I embarrassed myself too much. What's interesting is that Jim Newton's talk, scheduled immediately after mine, was about an application of user-defined specializers, though in the SKILL (or maybe SKILL++, I forget) language of Cadence rather than in Common Lisp. (For the Common Lisper who is intrigued at seeing an application for user-defined specializer classes, as well as reading Jim Newton's paper I can recommend getting my implementation of the ideas by doing darcs get http://common-lisp.net/~crhodes/vclos, and playing around with the source files and examples therein.

After that, Nikodemus talked about his work on making the various caches in SBCL's implementation of CLOS thread-safe; we got to see some current directions in Context-Oriented Programming development (as well as news that a very few people are actually using it in the wild!), and then it was off to dinner, where I manage to surprise Edi by telling him that his webserver was too complicated, and David managed (not entirely) to surprise me by not talking at all about putative CLIM backends...

[*]In My Opinion.

People* like to say that Common Lisp is the programmable programming language, or the #1=(programmable . #1#) programming language: possibly because much of what Common Lisp provides is in some way malleable: the reader, the printer, the pretty-printer, the object system and the syntax are all to a large extent customizeable, through *readtable*, print-object, *print-pprint-dispatch*, the Metaobject Protocol and macros respectively.

Not everything in CL is in such a marvellous state of perfect adaptability – though perfect adaptability does not necessarily mean unlimited customizeability, as there are concerns about being able to compile to efficient code where that is desired as well as being able to mold the system to meet a set of needs. I've already discussed one case of (by default) non-extensibility, that of subclassing the sequence System Class; Common Lisp has a wide range of functionality that is applicable to sequences, but no way of defining new kinds of sequence, instead restricting the applicability of those sequence functions to just singly-linked lists and vectors. In this modern age, that's a little bit unambitious, and so I presented a paper at the International Lisp Conference this year about how to extend CL to allow the user to define his own sequence classes.

For the 4th European Lisp Workshop (in conjunction with ECOOP), I've looked at a similar example of user-extensibility, though maybe a little more esoteric: examining the ability of the user to define subclasses of the specializer metaobject class (and how to use those new specializers in the existing CLOS metaobject protocols). The first draft of the writeup around this issue is available; the document linked there is not final, but it is correct in the broad brushstroke summary, which is that in this area the Metaobject Protocol is poorly implemented (an even more executive summary might be: get SBCL 1.0.7 if you want to try it out...)

[*] some people, anyway. It's possible for people to get caught up in hype and to make claims that they cannot themselves justify, appealing instead to some higher authority when challenged; one common pattern with respect to Common Lisp could be called the brucio pattern.