Older blog entries for Stevey (starting at number 625)

TAB completion is hard

I've spent the past few days overhauling the TAB-completion which is included in lumail.

Completing a single token is easy, if there is only one match, and you limit yourself to completing at the start of a line. But doing real completion is hard. Consider the case where you want to complete something like this:

unread_message_colour("re[TAB]

Clearly completing the first part "unread_[TAB]" is simple. But to complete "re" to "red" you need to split up your input line into tokens so that you can recognize a quote as a valid completion point.

Similarly you need to split on "(" to allow:

-- show the path to the editor
msg(edit[TAB]

To allow this to be changed/controlled by the user I defined completion_chars() which contains: SPACE, QUOTE, "(", etc.

I'm pleased with the user-callback for offering completion suggestions, my own is pretty basic and just includes all user-defined functions as well as an address book. The latter allows steve.org.uk[TAB] to complete to: "Steve Kemp" <steve@steve.org.uk> - because we allow matches anywhere in the completion string, rather than just prefix-matching.

I struggled with resolving ambiguities, but now that is handled correctly too. Press "TAB" when there are multiple choices available and you can graphically TAB-through the available choices, or press Esc to cancel.

In conclusion I've spent a few days fighting with user-interface stuff and now the mail-client is better, but I've still to tackle RFC 2047 header decoding because that is really hard!

Syndicated 2013-08-04 09:00:45 from Steve Kemp's Blog

International character sets and encodings are hard.

Today I've made the 0.15 release of lumail, which has several fixups and cleanups.

The previous release included a rewrite of the scrolling code, courtesy of kain88-de. This release fixes a few corner cases in that update which caused empty messages/Maildirs to be highlighted - operating on such ghost-entries would cause a segfault. Oops.

I've received several more great contributions from 7histle, and trou and I'm very happy with the state of the code and the usefulness of the application.

The biggest outstanding issue is RFC 2047 header decoding. Converting subject/to/from fields to readable versions of their encoded form:

Subject: =?utf-8?Q?Blipfoto=20=2D=20Introducing=20the=20all=2Dnew=20Bli

This is annoying because I'm using mimetic for handling all MIME-related code, and this doesn't seem to offer the facilities that I need.

The current plan is to use the RFC-2047 handling from vmime, but I've fought with that library unsucessfully for two days now - and a further complication is that the library is included in Squeeze/Sid, but not the stable release of Debian.

In conclusion I still regard the client as complete, because I'm using it exclusively and I rarely get "foreign" mails. But there is one more push required to fix all the outstanding bugs which generall boil down to:

  • Decode headers properly.
  • Ensure all our input/output is in UTF-8.

Randomly I'm wondering if I can call out to Lua to do the header decoding. Add "on_header_field()" and display the results. So today I'll be looking at how sensible that is, probably not very.

Syndicated 2013-07-26 09:25:06 from Steve Kemp's Blog

There must be a name for bugs you only find post-release

This week I made two releases of my mail client. Immediately after both releases I found bugs. Despite having been using the github source tree on my box for reading mail for days.

There must be a name for bugs that come up immediately after you've just made a release.

I'm torn between wanting to make a new release right now to fix the thing I spotted, or wait a few more days to fix a few other niggles.

Still I did write some cool code today:

  1. If a mail is received on the list debian.security-announce
  2. And the package in the Subject: is not installed on the current machine.
  3. The mail is marked as read.

Sure this means that a package on my webserver won't be visible to me, but my upgrade tool will see that. It just decreases the odds I read about an update that doesn't apply to me.

ObQuote: "don't pay heed to temptation
for his hands are so cold"

Syndicated 2013-07-18 19:49:25 from Steve Kemp's Blog

9 Jul 2013 (updated 9 Jul 2013 at 20:13 UTC) »

lumail is complete

So, writing an email client, how did that turn out? Pretty damn well as it happens.

My views continue to range from "email is easy" to "email is hard". But I'm now using my home-grown mail-client for 100% of my personal mail-handling.

Writing a mail client did seem a little crazy when I started, but the problem breaks down into a few distinct steps and individually they're not so hard:

  • Display a list of folders.
  • Display a list of messages.
  • Display a single message.

Once you've got that you're almost 80% complete. You're just missing things like "delete", "reply", "compose", and those things are pretty simple to implement.

I definitely made the right call in making it scriptable with Lua, because I've been able to write so many functions for working with mail. For example marking messages in all folders matching a regular expression.

The code is pretty well structured, and now I've got TAB-completion support on all primitives and user-additions I'm finding it a lot nicer to use.

The future? I'm not sure. But right now I'm very happy.

Syndicated 2013-07-09 12:31:35 (Updated 2013-07-09 20:13:40) from Steve Kemp's Blog

This weekend I will be mostly upgrading to wheezy

Having migrated my websites away from my ssh/mail box I'm going to upgrade that this weekend.

I've got my new mail client, lumail, working well enough to use exclusively, so the upgrade should be nice and simple.

I spent a few hours last night removing packages from my ssh/mail box to trim it down. Removing a bunch of Perl modules I used in my CGI coding, removing services such as nfs-common, portmapper, etc.

Today I'll have a stab at the upgrade. The only thing I have to be careful of is my backported/tweaked qpsmptd packages. I'll try the native wheezy version now it has caught up.

(I don't use any of the standard qpsmtpd plugins at all. Instead I have a separate tree of my own custom anti-spam and virtual-hosting aware plugins. They all work in a unified fashion. Using these plugins against a new version of qpsmtpd should be just fine. But obviously I need to test that.)

Work on Lumail is probably going to slow down now it is genuinely in use, but I'll keep an eye out for feature requests and missing primitives. Annoyingly I wasted 30 minutes just now implementing a plugin I'd already written: lumail issue #51.

I also need to step-back this weekend and reassess my hosting. When I was tweaking my slaughter setup I recently realized I have more hosts than I thought:

  • Cluster running for Debian-Administration.org:
    • 4 x web nodes
    • 2 x database nodes
    • 2 x "planet" nodes
    • 1 x "misc" node.
  • Personal stuff
    • 1 x ssh/mail host. (ssh.steve.org.uk/mail.steve.org.uk)
    • 1 x web host. (www.steve.org.uk, www.lumail.org, etc, etc.)
    • 1 x blogspam.net server (www.blogspam.net)
    • 1 x builder node - Runs buildd for producing my packages.

Total: 13 virtual machines. (+ one kvm-host)

Syndicated 2013-07-06 12:07:23 from Steve Kemp's Blog

So wheezy is fun..

Running a pristine operating system is fun. I keep going to run programs and finding they're not installed!

For example I'd planned to re-deploy the Debian Administration code-base this evening, but couldn't because fabric is not contained in Wheezy (#714421).

Thankfully backporting the package was trivial, but it was a minor stumbling block. I've hit a few of those recently.

Still in happier new my lumail mail client can now handle both adding attachments to outgoing mails and extracting them from incoming mails.

There are missing features which some people might expect and rely upon, such as:

  • Threading support.
  • GPG support.

But that said the scripting primitives allow interesting things to be done and I'm enjoying the experience of writing something so "major".

True I've got ~20 "stars" on github which isn't a great sign of popularity, but I have had some fun feedback and the client works for me.

I'm going to have to spend a few days working on TAB-completion code that plays-nice with curses, because that's a major irritation (and you can't mix/match curses & readline, annoyingly). But otherwise I think we're getting close to being complete enough I'll slow down.

Syndicated 2013-07-01 19:56:46 from Steve Kemp's Blog

So I have a new desktop..

So I have a new desktop computer. I installed Wheezy on it via a USB stick, and everything worked. All the hardware. Yay. I guess we take it for granted when things like sound, disks, and network cards just work these days. I remmeber fighting with distros in the past, where such things were not necessarily straightforward.

The only minor complication is the graphics card. I bought a cheap/random GeForce card for the new machine (£30):

$ lspci -nn | grep VGA
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF119 [GeForce GT 610] [10de:104a] (rev a1)

Booting up I get a working X.org and GNOME 3.x, but the open graphics driver is "too bad" so I get fallback GNOME; with "Applications" & "Places" menus.

Installing the proprietry driver gave me a full GNOME 3.x experience. But I didn't like it so for the moment I'm running:

  • GNOME fallback mode.
  • Bluetile.
  • Open (nvidia) drivers only.

The plan was to install awesome, or similar, but I'm just a creature of habit and I'm still cloning git/mercurial repos and selectively restoring backups.

My old desktop has been given to my partner to replace the EeeeeePC she's been using for the past year.

I'll fettle over the weekend until I'm back up and running properly; but for the moment I'm good. All my videos/music are ported across. I can print, and I have access to the repos I'm currently working on. (Mostly lumail which will have a new release over the weekend.)

Syndicated 2013-06-29 10:05:16 from Steve Kemp's Blog

So mid-summer is over.

So my partner and I spent last week in Helsinki, visiting for Midsummer.

The place was lovely, albeit "too damn hot"tm. (Despite my regularly shaved head I'm a pale-skinned redhead covered in freckles. In sustained sun I can be burned in the space of 10-20 minutes. Ouch.)

I ate cake, went to a sauna, and all the usual things. I also had personal interviews with all the mosquitos.

The only downside to the holiday was the timing:

  • Friday Worked.
  • Saturday woke up early and flew.
  • Time passes, Thorin sits down and starts singing about gold.
  • Sunday flew back.
  • Monday .. worked.
  • Tuesday worked ..

I'm a sysadmin and sometimes that involves out of hours work, so you can shuffle virtual machines around, upgrade processors, and do all the kind of "disruptive" work outside core business hours.

Yesterday I was awake for 22 hours and I was working for 14 of them. Today I'm doing nothing that involves a computer other than a token check-in or three to make sure everything is fine.

99% of the time I don't mind working late, starting early, or scheduling an event outside working hours (e.g. last night I worked from 10PM-3AM ). But with the holiday I've definitely been feeling burned. My sleeping is screwed up, and I'm just getting grumpy and stupid.

Still there were some highlights and I took some nice photos, met some good people, and learned some more Finnish.

I actually tried to find some Finnish instructor(s) here in Edinburgh, upon my return, and was amused to discover that there is an Edinburgh-based Finnish Society. Amused? They cancelled their midsummer event due to poor weather.

For the rest of the week I'm going to be very careful to count working hours and do nothing excessive. I've also got to get back into my gym-routine.

Still there is good news on the horizon. I get a new computer next Monday.

The new machine will run Wheezy, and Awesome. With 8GB RAM I'll stop hitting OOM conditions once I process many photographs, and I'm going to be very dedicated in using revision control for everything.

(I've noticed I've gotten lazy and have started storing bookmarks locally again, instead of under revision control which is a bad sign.)

Rambling. I woke up early (10:30AM) because "Microsoft Technical Support" called me and told me my computer had a virus ..

Syndicated 2013-06-26 09:53:23 from Steve Kemp's Blog

Debian is missing a tool, want to write it?

Seeing this piece in the news, about how Debian-Multimedia.org is now unsafe, I was reminded we don't have a tool to manipulate sources.lists entries.

For example:

$ apt-sources list
..
deb http://ftp.uk.debian.org/debian/ squeeze main non-free contrib
deb-src http://ftp.uk.debian.org/debian/ squeeze main

deb http://security.debian.org/ squeeze/updates main
deb-src http://security.debian.org/ squeeze/updates main
..

How about listing only my repos?

$ apt-sources list steve.org.uk
deb-src http://packages.steve.org.uk/firefox-wrapper/squeeze/ ./
deb     http://packages.steve.org.uk/firefox-wrapper/squeeze/ ./
deb     http://packages.steve.org.uk/meta/squeeze/ ./
deb-src http://packages.steve.org.uk/meta/squeeze/ ./
deb-src http://packages.steve.org.uk/minidlna/squeeze/ ./
deb     http://packages.steve.org.uk/minidlna/squeeze/ ./

Now add in a command to delete lines matching a given pattern:

# apt-sources delete debian-multimedia.org

Doesn't that seem like a tool that should exist?

I've added this quick hack to this repository which you can submit pull requests against, or use as a base.

TODO: Write the "add" handler. Neaten.

Ever felt jealous that Ubuntu users can add PPAs? Nows your chance to do something like this:

# apt-sources add "deb http://packages.steve.org.uk/lumail/wheezy/ ./"

Syndicated 2013-06-14 05:04:29 from Steve Kemp's Blog

Migrations and changes

So I'd previously talked about migrating machines. From having one virtual machine running "mail" + "web" + "stuff" I've now got three hosts:

ssh.steve.org.uk

This is supposed to be used solely for shell access, email reading, IRC.

Sadly it still hosts one website, the web interface to my Mercurial repositories. This can't be moved without moving the repositories which is a step too far. Although I don't particularly want people browsing my code/changes I do want them to be able to clone them. If I could get anonymous-SSH checkouts working, sanely, then I'd be happy, but I don't see how to do that.

skx-web

This hosts all my websites except for two.

The two that are excluded are my mercurial repositories, which still lives on ssh.steve.org.uk, and my blogspam service.

blogspam

This runs my blogspam.net service.

I wish I could retired this, since it uses cruftly XML::RPC. I'd rather see a RESTful application sending/receiving JSON.

Sadly I can't kill it without annoying a lot of people. So it must remain.

I chose to add the blogspam guest, because that service does really take over an IP, and it just seemed simplest to move it to one machine. As a quick hack both http://repository.steve.org.uk & http://blogspam.net/ run under Apache. Although the other websites run under their own UID with thttpd + my proxy.

Now time to change the subject entirely. I've recently joined gym. Which isn't as horrible as I thought it might be, though as a matter of policy I refuse to go on any of those fancy running/jogging machines.

The first three weeks I just alternated between the cycling machines and the rowing machines. Initially ten minutes on each, then twenty, then thirty.

Now I'm being all brave and adventurous, using new machines and pieces of equipment.

Writing this I've got a dull ache in both my arms, after doing seated-dips with 100Lbs. So I guess I'm starting to make progress.

No specific goals in mind, but I've been paying slightly more attention to my diet over the past month and I think if I'm a "little fitter" and have "slightly nicer arms" then I'll be happy enough.

I've no desire to go all anal and count calories, or give up chocolate and beer. So it is almost hard to explain why I'm going, but .. it is fun, and watching the numbers change is fun too.

I'll probably post more about this in the future.

Syndicated 2013-06-11 13:44:47 from Steve Kemp's Blog

616 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!