Older blog entries for Stevey (starting at number 616)

Migrations and changes

So I'd previously talked about migrating machines. From having one virtual machine running "mail" + "web" + "stuff" I've now got three hosts:

ssh.steve.org.uk

This is supposed to be used solely for shell access, email reading, IRC.

Sadly it still hosts one website, the web interface to my Mercurial repositories. This can't be moved without moving the repositories which is a step too far. Although I don't particularly want people browsing my code/changes I do want them to be able to clone them. If I could get anonymous-SSH checkouts working, sanely, then I'd be happy, but I don't see how to do that.

skx-web

This hosts all my websites except for two.

The two that are excluded are my mercurial repositories, which still lives on ssh.steve.org.uk, and my blogspam service.

blogspam

This runs my blogspam.net service.

I wish I could retired this, since it uses cruftly XML::RPC. I'd rather see a RESTful application sending/receiving JSON.

Sadly I can't kill it without annoying a lot of people. So it must remain.

I chose to add the blogspam guest, because that service does really take over an IP, and it just seemed simplest to move it to one machine. As a quick hack both http://repository.steve.org.uk & http://blogspam.net/ run under Apache. Although the other websites run under their own UID with thttpd + my proxy.

Now time to change the subject entirely. I've recently joined gym. Which isn't as horrible as I thought it might be, though as a matter of policy I refuse to go on any of those fancy running/jogging machines.

The first three weeks I just alternated between the cycling machines and the rowing machines. Initially ten minutes on each, then twenty, then thirty.

Now I'm being all brave and adventurous, using new machines and pieces of equipment.

Writing this I've got a dull ache in both my arms, after doing seated-dips with 100Lbs. So I guess I'm starting to make progress.

No specific goals in mind, but I've been paying slightly more attention to my diet over the past month and I think if I'm a "little fitter" and have "slightly nicer arms" then I'll be happy enough.

I've no desire to go all anal and count calories, or give up chocolate and beer. So it is almost hard to explain why I'm going, but .. it is fun, and watching the numbers change is fun too.

I'll probably post more about this in the future.

Syndicated 2013-06-11 13:44:47 from Steve Kemp's Blog

Migrations and movements

Recently I wanted to cleanup my "main" remote machine. It is a system I've had for many years, which started off as a i386 KVM-guest and was later migrated in place to an AMD64 installation.

These days the host runs:

• Mail for many domains, via QPSMTPD and ms-lite.
• Website hosting for many domains. Each site running under a dedicated per-UID thttpd instance, behind a node reverse proxy.
• IRC for kirsi.

The plan was originally to move the "mail stuff" to a new (wheezy) guest. I aborted that after discovering that the mutt-patched package has (IMHO) regressed.

So today I spun up a new virtual machine, and configured it to host websites.

Thus far I've migrated steve.org.uk, and lumail.org to the new host. Both simple sites that are built via my static-site-generator. Migration mostly involved configuring the proxy and the thttpd instances - then using rsync to migrate the content.

I've renamed my current host, which was previously www.steve.org.uk and is now ssh.steve.org.uk, and pushed DNS changes.

If all is smooth and happy I'll slowly migrate the rest of the sites. Fingers crossed this will be painless and I'll have a clean split between "login + mail" and "websites".

Syndicated 2013-06-08 14:11:34 from Steve Kemp's Blog

How do people deal with email?

As part of writing a new mail client I'm wondering about how to change my email-life, and how other people process/handle their incoming email.

I sort my incoming email into folders at delivery-time using procmail. Mail is generally filtered into mailboxes on the basis of the company that sent it, the person that sent it, or the machine which generated it.

Because I manage a lot of machines personally I've split things up so that I have a folder per host. So on a morning I might have unread mail in the following folders:

machines.steve.org.uk/
machines.da-db1/
machines.da-db2/
machines.da-web1/
machines.da-web2/
machines.da-web3/
machines.da-web4/

The per-machine mailboxes usually contain a single mail every day from LogWatch, along with output from any cron-jobs. For example today I received the mail:

From: Cron Daemon
To: steve@steve.org.uk
Subject: Cron steve@steve.org.uk /home/steve/bin/download-check

URL http://nodejs.org/ - no longer matches v0.10.9

Generally speaking I don't need to read the per-machine messages. I'll keep the most recent 100 for reference, but only need to look if something seems "off" on a machine. But if I don't look I'd not see the node upgrade notice, so find that I do read them after all.

This suggest to me that email isn't the right way to handle this kind of thing. Instead I should use a notification system - at work we have a central service called MauveAlert (yes, Red Dwarf reference). Mauve receives "alerts" of various kinds, via UDP. The alerts are then fanned out to appropriate people via XMPP, Email, or SMSs.

I have a similarly-inspired system I use on my Debian Administration cluster. A (node) service runs non-stop collecting UDP messages and showing them on a dashboard. I look at it throughout the day to see when slaughter runs, etc.

Anyway in conslusion I get a lot of mail. Some of it is related to random projects, and all ends up in the steve.org.uk/ mailbox, some of it relates to machines, and gets filed away, and I have regular conversions with folk so I have a .people.kirsi/ folder which receives a lot of attention, for example.

ObRandom:daily() - Mark ~/Maildir/.machines.*, etc, read.

Syndicated 2013-06-05 10:58:58 from Steve Kemp's Blog

Minimalism still works out

When people ask me why I chose to embed Lua in my mail-client I'll point to my on_idle() documentation.

Moving from a callback which runs once every second, or so, to allowing the user to schedule tasks on arbitrary boundaries is pretty cool - and obviously requires no explicit support from myself.

Now I've fixed a couple of bugs which went unspotted/unreported in the first release I'm ready for a new one "soon".

In the meantime I'm running the client exclusively, and loving the ability to view all unread mail, only, regardless of the parent folder.

Syndicated 2013-06-04 05:12:26 (Updated 2013-06-04 06:21:37) from Steve Kemp's Blog

First binary release of lumail

Give me a few days and I'll stop writing about Lumail, but tomorrow I intend to make the first stable source & binary release. From that point onwards you'll not need to track the github repository to getu pdates.

The website has had an overhaul in advance of the release, but could still benefit from a logo. As usual I've written the website using my templer static-site generator. I hacked up a couple of plugins to make it easy to generate the pages of Lua primitive documentation, and handle cross-links suitably. (The source is available for reference.)

The first binary release of the mail-client is obviously something of a big deal. I've been using the client daily for the past week or so, as a read-only mail-viewer. But now that the compose() and reply() primitives are present it is usable "for real". Having real scripting present is also allowing me to do interesting things, which are kinda/sorta demonstrated on the examples page.

Now its a case of fixing up a couple of display-related glitches, and implementing things that are both missing and desirable. Happily the list of missing things is actually surprisingly small.

I think the biggest outstanding issue is that the defaults are Steve-friendly, for example the colour-setup should probably be configurable.

Beyond the personal-defaults I think the next biggest issue is the lack of threading support. Messages are displayed in oldest->newest order. Always.

The other omission is that it is impossible to tag-things, in the mutt-sense, so you can't reply to two messages at once. That's a design decision I might have to revisit. The balance of course is that you can open multiple folders at once, and that rocks!

Happy days.

Syndicated 2013-05-30 22:07:29 from Steve Kemp's Blog

Lumail continues to progress

Although I've still not got the ability to reply to messages, and composing new ones is ugly, my toy mail client is working nicely.

I've received a couple of patches, and given commit access to the repository to one other user.

Currently I'm still juggling primitives around and working out what is missing. The big exceptions are the obvious:

• Cannot reply to a message.
• Cannot move a message to a new folder.
• When composing a mail to be sent no copy is saved in "sent-mail", or similar.
• Thread-view is absent. Indefinitely.

But on the plus side the lua scripting is lovely:

precious ~/git/lumail $ rm /tmp/unread.log
precious ~/git/lumail $ ./lumail  --rcfile ./lumail.lua --eval "dump_unread();"
precious ~/git/lumail $ head /tmp/unread.log
Selected folder /home/skx/Maildir/.Automated.backups
	Folder has 10 unread messages
Selected folder /home/skx/Maildir/.Automated.bounces
	Folder has 3 unread messages
Selected folder /home/skx/Maildir/.CRM.Spam
	Folder has 7 unread messages
Selected folder /home/skx/Maildir/.facebook.com
	Folder has 4 unread messages
..

The website needs some love, most notably a logo. And there are several reported bugs/todo-items I need to work through.

Still for a toy program I'm using it daily. (Though still using mutt to reply to messages & view/save attachments.)

Syndicated 2013-05-23 12:22:37 from Steve Kemp's Blog

Some good, some bad

Today my main machine was down for about 8 hours. Oops.

That meant when I got home, after a long and dull train journey, I received a bunch of mails from various hosts each saying:

• Failed to fetch slaughter policies from rsync://www.steve.org.uk/slaughter

Slaughter is my sysadmin utility which pulls policies/recipies from a central location and applies them to the local host.

Slaughter has a bunch of different transports, which are the means by which policies and files are transferred from the remote "central host" to the local machine. Since git is supported I've now switched my policies to be fetched from the master github repository.

This means:

• All my servers need git installed. Which was already the case.
• I can run one less service on my main box.
• We now have a contest: Is my box more reliable than github?

In other news I've fettled with lumail a bit this week, but I'm basically doing nothing until I've pondered my way out of the hole I've dug myself into.

Like mutt lumail has the notion of "limiting" the display of things:

• Show all maildirs.
• Show all maildirs with new mail in them.
• Show all maildirs that match a pattern.
• Show all messages in the currently selected folder(s)
  • More than one folder may be selected :)
• Shall all unread messages in the currently selected folder(s).

Unfortunately the latter has caused an annoying, and anticipated, failure case. If you open a folder and cause it to only show unread messages all looks good. Until you read a message. At which point it is no longer allowed to be displayed, so it disappears. Since you were reading a message the next one is opened instead. WHich then becomes marked as read, and no longer should be displayed, because we've said "show me new/unread-only messages please".

The net result is if you show only unread messages and make the mistake of reading one .. you quickly cycle through reading all of them, and are left with an empty display. As each message in turn is opened, read, and marked as non-new.

There are solutions, one of which I documented on the issue. But this has a bad side-effect that message navigation is suddenly complicated in ways that are annoying.

For the moment I'm mulling the problem over and I will only make trivial cleanup changes until I've got my head back in the game and a good solution that won't cause me more pain.

Syndicated 2013-05-14 20:23:08 from Steve Kemp's Blog

The rain in Scotland mainly makes me code

Lumail <http://lumail.org> received two patches today, one to build on Debian Unstable, and one to build on OpenBSD.

The documentation of the lua primitives is almost 100% complete, and the repository has now got a public list of issues which I'm slowly working on.

Even though I can't reply to messages I'm cheerfully running it on my mail box as a mail-viewer. Faster than mutt. Oddly enough. Or maybe I'm just biased.

Syndicated 2013-05-11 14:08:15 (Updated 2013-05-11 23:12:59) from Steve Kemp's Blog

So progress is going well on lumail

A massive marathon has resulted in my lumail mail client working well.

Functionally the application looks little different to the previous C-client, but it is a lot cleaner, neater, and nicer internally.

The configuration file luamail.lua gives a good flavour of the code, and the github repository has brief instructions.

Initially I decied that the navigation/index stuff was easy and the rest of the program would be hard; dealing with GPG-signatures, MIME-parts, etc.

But I'm stubborn enough to keep going.

If I can get as far as reading messages, with MIME handled properly, and replying then I can switch to using it immediately which will spur further development.

I'm really pleased with the keybinding code, and implementing the built-in REPL-like prompt was a real revelation. Worht it for that alone.

The domain name lumail.org was available. So I figured why not?

Syndicated 2013-05-07 18:40:27 from Steve Kemp's Blog

After you've started it seems like a bad idea?

To recap: given the absence of other credible alternatives I had two options:

• Re-hack mutt to give me a sidebar that will show only folders containing new messages.
• Look at writing a "simple mail client". Haha. Ha. Hah.

I think there is room for a new console client, because mutt is showing its age and does feel like it should have a real extension language - be it guile, lisp, javascript(!), Lua, or something else.

So I distilled what I thought I wanted into three sections:

• mode-ful. There would be a "folder-browsing mode", a "message-browsing mode" and a "read-a-single-message" mode.
• There would be scripting. Real scripting. I chose Lua.
• You give it ~/Maildir as the configuration. Nothing else. If the damn computer cannot find your mailboxes something is wrong.

So how did I do? I wrote a ncurses-based client which has Lua backed into it. You can fully explore the sidebar-mode - which lets you select multiple folders.

From there you can view the messages in a list.

What you can't do is anything "real":

• Update a messages flags. new -> read, etc.
• GPG-validation.
• MIME-handling.
• Attachment viewing.

For a two-day hack it is remarkably robust, and allowing scripting shows awesomeness. Consider this:

--
-- show all folders in the Maildir-list.
--
function all()
   -- ensure that the sidebar displays all folders
   sidebar_mode = "all";
   -- we're going to be in "maildir browsing mode"
   cmail_mode = "sidebar";
   reset_sidebar();
   refresh_screen();
end

--
-- Test code, show that the pattern-searching works.
--
-- To use this press ":" to enter the prompt, then enter "livejournal".
--
-- OR press "l" when in the sidebar-mode.
--
function livejournal()
   sidebar_pattern = "/.livejournal.2";
   sidebar_mode = "pattern";
   reset_sidebar();
   refresh_screen();
end

--
-- There is a different table for each mode.
--
keymap = {}
keymap['sidebar'] = {}
keymap['index']   = {}
keymap['message'] = {}

--
-- In the sidebar-mode "b" toggles the sidebar <-> index.
--
-- ":" invokes the evaluator.
-- "q" quits the browser and goes to the index-mode.
-- "Q" quits the program entirely.
--
keymap['sidebar'][':'] = "prompt-eval"
keymap['sidebar']['b'] = "toggle"
keymap['sidebar']['q'] = "toggle"
keymap['sidebar']['Q'] = "exit"

-- show all/unread/livejournal folders
keymap['sidebar']['a'] = "all"
keymap['sidebar']['u'] = "unread"
keymap['sidebar']['l'] = "livejournal"

Neat, huh? See the cmail.lua file on github for more details.

My decision hasn't really progressed any further, though I can see that if this client were complete I'd love to use it. Its just that the remaining parts are the fiddly ones.

I guess I'll re-hack mutt, and keep this on the back-burner.

The code is ropey in places, but should you wish to view:

• https://github.com/skx/cmail

And damn C is kicking my ass.

Syndicated 2013-04-30 15:03:35 from Steve Kemp's Blog