Older blog entries for lkcl (starting at number 678)

multitask cooperative single-process http server

egads, mouthful for a subject let alone a title of a wiki page. for various reasons i needed to create a single-process cooperative multitasking HTTP server. it seems, bar one small tolerable bug, that i succeeded in using it to create a special kind of HTTP proxy.

the problem is illustrated by http://gnumed.org being a desktop-based application and using role-based access control on the PostgreSQL database. they want to convert it to a web application, by popular demand, and tried picking random web server frameworks. unfortunately, virtually all web server frameworks assume that the SQL database is to be used for anonymous session cookie management, assume a single (global) username/password and so on.

for gnumed, such assumptions are completely unacceptable: this is confidential patient data, and there's no way that gnumed are going to redesign the database middleware to REMOVE password security.

so i came up with something that will frighten the bejabbers out of any normal web developer: a proxy which is capable of keeping a remote connection permanently open to its upstream target, as long as that upstream HTTP server supports HTTP Connection "keep-alive".

it's… well, i've been yapping about it enough, and so have the gnumed developers: http://lists.gnu.org/archive/html/gnumed-devel/2010-07/msg00326.html

one of the problems with creating such a proxy is that you can't do it with a threaded or a forking architecture: you need to analyse the HTTP stream (check a session cookie and insert one if there isn't one) BEFORE you pass it on to the proxy, but most HTTP proxies are designed to fork first, think later.

the design also has several other uses, one of which is that it solves the issue that pyjamas-web faces. pyjamas-web is the version of pyjamas that actually executes a pyjamas app server-side and translates it into HTML. immediately you can see the problem, there, if you know that HTTP is stateless: how do you re-associate the actual in-memory running python application with the correct browser, when the application is sufficiently large that it is impractical to serialise it to disk?

Syndicated 2010-07-15 20:25:46 from blog/lkcl

mailing list transferred

hooray, another task done - sort-of. it appears that some messages were lost, leaving people wondering what the heck was going on, where's the "digest" mode (there isn't one in lamson, because zed always finds that people abuse it by replying "Re: Digest Volume 3.1415926")

had a bit of a fight with lamson: messages were bouncing from somewhere, causing multiple repeats i _cannot_ tell where from, so may end up fronting exim4 onto lamson despite my wishing not to do that. as i had to create a lists.pyjs.org, the problem with redirecting to a remote queue goes away (exim doesn't redirect to properly, even if you change the SMTP port: there's supposed to be an option you can set to say "yes do it anyway" but…)

i'm getting people still saying "why are you doing this? why waste time reinventing mailing list software? why reinvent it with software that's _worse_ than [proprietary and] much better offerings? why not just use google groups" and i have to gently remind them - again - that their questions hold the answers: the alternatives are proprietary, and they're not pyjamas applications.

i stated that i'm going to uphold a principle - to use free software and to do pyjamas front-end apps - and to stop doing what i'm doing, even if it's "not as good right now" would be breaking that principle. i'm absolutely going to stick to my free-software-driven principles, because i can, and because i choose to, and that's really all there is to say.

if anyone wants to _help_ improve the software, now, they can. whereas, with the source code to groups.google.com, they can't. but we know this already: i just think people find it a bit unusual for someone to jump train _and_ "rip up the tracks" on a running project and start pumping on a cheap but 100% Cooperative-owned hand-driven railway cart, because they don't like the fact that the train and the carriages are all run by a big corporation.

i think… perhaps it's worthwhile mentioning the phrases "bitkeeper vs git" and "linus vs tridge" - not entirely accurate but i think you get the picture.

Syndicated 2010-07-06 15:43:11 from blog/lkcl

new mailing list almost done

ouch. lots done today. managed to delete pyjs.org apache config, had to recreate it. had to create lists.pyjs.org due to conflicts with WSGI apache configs, i am _not_ going to try doing rewrite rules. almost got lists.pyjs.org done: only need to do an email enable/disable feature and then i'm done: bye bye googlegroups.

Syndicated 2010-07-04 23:36:17 from blog/lkcl

great old hat

Django Piston

I found a great django framework extension, by virtue of accidentally searching for "django oauth":


If i didn't know pyjamas, I would be terribly excited by this extension. as it is, it's pretty good anyway (because of the json emitter for example, and the bandwidth throttling decorators).

but i find myself feeling… terribly sad, and strangely looking in pity on these excellent pieces of work, because they just look like so much damn hard work to make use of. generating pages server-side that mix data with the display mechanism is so… old! no matter how exciting and fresh the framework and the ideas and thought that goes into the framework, the technique is still "old" and inconvenient.

i feel very strange for having to point this out.

Syndicated 2010-07-04 10:46:33 from blog/lkcl

pyjamas lamson replacement for google groups

Pyjamas and Lamson Mailing list viewer and poster

as part of the "eat dogfood" mantra, and the lovely lamson mailing list software having been chosen, i did an archive viewer a couple of days ago, which used the json-formatted archive created by lamson, because that's far easier to pick up (and less cpu-intensive) and turn into python than a Maildir.

what was missing was the means to post messages.

so, in order to do that, i've had to add openid support to the pyjamas app. then, having got that to work, it was no good on its own so i had to add a way for lamson to send out "is this person who they claim to be" messages, by which the openid could be associated with the email address.

haven't added that into the GUI yet - have to add a "preferences" box. hmm…

anyway: that was quite a lot of fun, working out how to create a state machine in lamson. it took me hours to work out what was going on, but i now should also be able to turn lamson into a "proper" email service, by supporting simple relaying of postmaster@pyjs.org and so on to me, which lamson *doesn't* do on its own.

but then, the next step is to allow posting of messages. that proved also challenging: in the end i chose to use the lamson.server.Relay, connecting over and faking up the "From" and "To".

there's still quite a bit to do, but once it gets to the "useable" state i'm simply going to chop pyjamas-dev googlegroups.com off at the knees. i'll add a reply button, i'll make sure there's a "compose" button… and… errr… attachments? nope. viewing attachments? nope. showing message "reply" threads? nnnope. well, ok, maybe. might have to see how that goes, see if it's easy to do.

about the only thing i migggght see if i can do is to add support for google oauth / openid, because most of the users duh being on googlegroups use google for authentication duh.

but - again, have to see how it goes: once a certain aaabsolute minimum threshold of functionality is achieved, it's bye bye google groups, and one less web site being used which is _not_ a pyjamas application goes out the window.

Syndicated 2010-07-03 22:46:46 from blog/lkcl

diary syndicated to advogato

The lovely Advogato

I like the fact that people can reply / discuss using ikiwiki, if they want to, whereas advogato has a single-stream of consciousness / diarrhoea. so i'm syndicating the pyjamas diary to there, let's see if it works…

Syndicated 2010-07-03 09:34:07 from blog/lkcl


Permissions are not set restrictively for the blog section of the wiki; Luke did not write this. Thank you for sharing, dave :) ikiwiki, we love you, we do. TODO: replace ikiwiki front-end with a pyjamas one and write some authentication in the process.

Syndicated 2010-07-02 17:09:28 from blog/lkcl

why convert everything to pyjamas

Why the heck convert the Pyjamas project management to Pyjamas?

i'm already getting questions about why i'm doing this. why not use github.com, why not use sourceforget, why not use ready-made, proven, tried-and-tested project management and development tools, designed for free software development? the answer is simple: when those sites convert their entire infrastructure to have a pyjamas front-end, and release the source code as free software so that people can either manage their projects via a desktop application or via the web front-end, i'll happily convert http://pyjs.org to use their facilities.

if, like 99.9% of the projects hosted on the above-mentioned free software hosting sites, pyjamas was not a web/desktop development platform, this would be a non-issue.

the purpose of pyjamas is to make life easier for both web and desktop development. i get a number of people telling me "but it would be great if you used {insert javascript toolkit}!" to which i have to again gently remind them that the toolkit they are recommending is javascript, not python, and so will work neither for "Eat Dogfood" reasons nor work with pyjamas-desktop (not interactively at the python level, anyway).

so in short: anything that's not using pyjamas is effectively a showcase for that other web technology, not for pyjamas. therefore, simple: to promote pyjamas, use it for everything!

Syndicated 2010-07-02 12:06:27 from blog/lkcl

lamson list archive viewer started

Eating Dogfood, one Web app at a time: Pyjamas List Archive

well, i took the Mail example and finally did something with it that i've been meaning to do for a while: actually got it to really read real email messages. the other application which does that is http://beammail.org which thomas did, a year ago, and i must update it. thomas actually did send, login and even read contacts and TODO information, so it's a worthwhile exercise to update it.

but the List Archive reader is slightly different. firstly, i decided to read the pre-processed json data, not the Maildir stuff. this will reduce the load on my happy server. if lamson has already gone to the trouble of splitting out the headers, splitting out the attachments and so on, why do it again?!


and the code is at:

    git clone gitolite@pyjs.org:pyjamaslamson

Syndicated 2010-07-02 12:01:39 from blog/lkcl

working pyjamas git jsonrpc service

Up and Running!


amazing - a JSONRPC-based git web service, along with a happily bare but functional front-end. it's… wow! i kinda love it. it doesn't look like much, but it's actually one of the first real functional development tools i've done with pyjamas, other than "demos".

a mailing list archive viewer is next, then i can drop groups.google.com.

if anyone wants the source code, it's at

   git clone gitolite@pyjs.org:pyjamasgitweb

Syndicated 2010-06-29 22:27:41 from blog/lkcl

669 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!