Older blog entries for Stevey (starting at number 595)

More competition for server management and automation is good

It was interesting to read recently from Martin F. Krafft a botnet-like configuration management proposal.

Professionally I've used CFEngine, which in version 2.x, supported a bare minimum of primitives, along with a distribution systme to control access to a central server. Using thse minimal primitives you could do almost anything:

  • Copy files, and restart services that depend upon them.
  • Make minor edits to files. (Appending lines not present, replacing lines you no longer wanted, etc)
  • Installing / Removing packages.
  • More ..

Now I have my mini cluster (and even before that when I had 3-5 machines) it was time to look around for something for myself.

I didn't like the overhead of puppet, and many of the other systems. Similarly I didn't want to mess around with weird configuration systems. From CFEngine I'd learned that using only a few simple primitives would be sufficient to manage many machines provided you could wrap them in a real language - for control flow, loops, conditionals, etc. What more natural choice was there than perl, the sysadmin army-knife?

To that end slaughter was born:

  • Download polices (i.e. rules) to apply from a central machine using nothing more complex than HTTP.
  • Entirely client-driven, and scheduled via cron.

Over time it evolved so that HTTP wasn't the only transport. Now you can fetch your policies, and the files you might serve, via git, hg, rsync, http, and more.

Today I've added one final addition, and now it is possible to distribute "modules" alongside policies and files. Modules are nothing more than perl modules, so they can be as portable as you are careful.

I envisage writing a couple of sample modules; for example one allowing you to list available sites in Apache, disable the live ones, enable/disable mod_rewrite, etc.

These modules will be decoupled from the policies, and will thus be shareable.

Anyway , I'm always curious to learn about configuration management systems but I think that even though I've reinvented the wheel I've done so usefully. The DSL that other systems use can be fiddly and annoying - using a real language at the core of the system seems like a good win.

There are systems layered upon SSH, such as fabric, ansible, etc, and that was almost a route I went down - but ultimately I prefer the notion of client-pull to server-push, although it is possible in the future we'll launche a mini-daemon to allow a central host/hosts to initial a run.

Syndicated 2013-02-02 21:27:09 from Steve Kemp's Blog

Shame there isn't more competition for self-hosted analytics

Today I've been mostly replanting spider-plants, aloe-vera plants, and shuffling trees around inside my flat.

Beyond that I've been updating my trivial dashboard skeleton, which was put together as part of this simple introduction article. (So there is a standalone redis&sinatra-using visualization server)

After working on the display I was suddenly reminded that I run a cluster now. That means I have four servers each writing a local Apache logfile, and no central way of viewing all my visitor-data.

There are several open source analytic packages such as piwik and openwebanalytics - but they require MySQL & PHP at the back-end.

Given that node.js is "teh new shiny" it is a surprise there isn't something out there using that, and web sockets perhaps, to collect visitor data.

I found a few toy projects, but nothing that seemed to be a clear winner. Adding some javascript to webpages to submit:

  • Browser version
  • Referer
  • Screen Size.
  • window.location
  • etc

Is trivial. The hard part is storing that and visualizing it in a neat way. Making data pretty is something I'm notoriously bad at - unless it is turning numbers into graphs using a good library I'm out of luck most of the time.

Anyway I will keep digging. Ideally I'll have a scalable node service that'll receive submissions, bung them in redis, and then show real-time activity in a sexy fashion. I can dream?

Syndicated 2013-01-25 19:28:46 from Steve Kemp's Blog

Something, something, dark side.

I want to like LDAP. Every so often I do interesting things with it, and I start to think I like it, then some software that claims to support LDAP fails to do so properly and I remember I hate it again.

I guess the problem with LDAP is that most people are scared by it, unless you reach a certain level of scale you don't need it. That makes installing it out of the blue a scary prospect, and that means that lots of toy-software applications don't even consider using it until they're mature and large.

When you bolt-on support for LDAP to an existing project you have to make compromises; do you create local entries in your system for these scary-remote-LDAP-users? Do you map group members from LDAP into your own group system? ANd so on.

To be fair to the application developers if the requirements for installation were "Install LDAP" they'd probably have a damn smaller userbase, and so we cannot blame OpenLDAP, or the other servers.

All the same it is a shame.

The very next piece of software I ever write that needs to handle logins will use LDAP and only LDAP. How hard can it be?

In happier news I re-deployed http://www.debian-administration.org/ over the weekend. It now uses the Bytemark BigV platform which rocks.

The migration was supposed to be a "Christmas Project", but took longer than expected due to the number of changes I need to make to the software, and my deployment plan. Still I'm very happy with the way things are running now, and don't expect I'll need to move or make significant changes for the next nine years. I just hope there is still interest in such things then.

ObQuote: "Would you like a treatment? " - Dollhouse

Syndicated 2013-01-22 19:16:06 from Steve Kemp's Blog

Debian-Administration.org almost migrated

The new version of the Debian Administration is almost ready now. I'm just waiting on some back-end changes to happen on the excellent BigV hosting product.

I was hoping that the migration would be a fun "Christmas Project", but I had to wait for outside help once or twice and that pushed things back a little. Still it is hard to be anything other than grateful to folk who volunteer time, energy, and enthusiasm.

Otherwise this week has largely consisted of sleeping, planting baby spider-plants, shuffling other plants around (Aloe Vera, Cacti, etc), and enjoying my new moving plant (video isn't my specific plant).

I've spent too long reworking templer such that is now written in a modular fashion and supports plugins. The documentation is overhauled.

The only feedback I received was that it should support inline perl - so I added that as a plugin this morning via a new formatter plugin:

Title: This is my page title
Format: perl
Name: Steve
This is my page.  It has inline perl:

   The sum of 1 + 5 is { 1 + 5 }

This page was written by { $name }

ObQuote: "She even attacked a mime. Just found out about it. Seems the mime had been reluctant to talk. " - Hexed

Syndicated 2013-01-06 10:32:42 from Steve Kemp's Blog

Template systems redux.

People seemed interested in my mini-reviews of static-site generators.

I promised to review more in the future, and so to shame myself into doing so I present:

As you can see I've listed my requirements, and I've included a project for each of the tools I've tested.

I will continue to update as I go through more testing. As previously mentioned symlink-handling is the thing that kills a lot of tools.

Syndicated 2012-12-30 10:58:19 from Steve Kemp's Blog

More polish for slaughter

A couple of days ago I made a new release of slaughter, to add a new primitive I was sorely missing:

if ( 1 != IdenticalContents( File1 => "/etc/foo" ,
                             File2 => "/etc/bar" ) )
   # do something because the file contents differ

This allows me to stop blindly over-writing files if they are identical already.

As part of that work I figured I should be more "visible", so on that basis I've done two things:

After sanity-checking my policies I'm confident I'm not leaking anything I wish to keep private - but there is some news being disclosed ;)

Now that is done I think there shouldn't be any major slaughter-changes for the forseeable future; I'm managing about ten hosts with it now, and being perl it suits my needs. The transport system is flexible enough to suit most folk, and there are adequate facilities for making local additions without touching the core so if people do want to do new things they don't need me to make changes - hopefully.

ObQuote: "Yippee-ki-yay" - Die Hard, the ultimate Christmas film.

Syndicated 2012-12-29 11:08:53 from Steve Kemp's Blog

Is there a ACL system for "all" revision control systems?

Once upon a time a company started using distributed version control, and setup several project repositories using darcs.

Over time people became more sane and new projects were created in mercurial.

Later still Git became available, and was used by a few of the brave.

Sadly each of these projects is hosted on the same host, and in the home directory of the same user. This means these two commands work:

hg clone ssh://projects@dev.host/foo

git clone ssh://projects@dev.host/bar

I'm now wanting to setup per-repository ACLs and have hit a problem...

There are several git-wrappers such as gitolite and gitosis. There is also the excellent hg-gateway and mercurial-server for dealing with mercurial.

However I've yet to find a wrapper which will handle both git & mercurial repositories, under the same UID. (+ Darcs too, of course).

So my question - is there such a beast out there, or do we need to write it? I expect such a thing would be useful for many people, so I'm surprised I've not yet found it.

Syndicated 2012-12-16 10:38:57 from Steve Kemp's Blog

December 2012 Software Updates

Some brief software updates:


This is the monitoring tool I wrote for Bytemark. It still rocks, and has run over 10 million tests without failure.

I'd love more outside feedback, even if just to say "documentation needs work".


This is my sysadmin tool for multiple hosts - consider it cfengine-lite, or cfengine-trivial more likely.

The 2.x release is finally out, and features pluggable transports. That means your central server can be running HTTP, RSYNC, FTP, or anything you like.

90% of the changes came from or were inspired by Csillag Tamas, to whom I owe a debt of thanks.


A static-site generator, written in Perl.

I use this to generate blogspam.net, and other sites from simple layouts. Tutorial available online.


A trivial hack which allows using Redis as a schema-less document storage system.

Assuming you never delete documents it is simple, transparent, and already in live use for Debian Administration

Random Comment on Templer:

Although I've made extensive notes on common static site generators, and they will be discussed at length in the near future, I do want to highlight one problem common to 90% of them: Symbolic links.

For example webgen fails my simple test:

~/hg/websites$ webgen create test.example.com
~/hg/websites$ cd test.example.com/src/

~/hg/websites/test.example.com/src$ mkdir jquery-1.2.3
~/hg/websites/test.example.com/src$ touch jquery-1.2.3/jquery.js
~/hg/websites/test.example.com/src$ ln -s jquery-1.2.3 jquery

~/hg/websites/test.example.com$ webgen
Starting webgen...
~/hg/websites/test.example.com$ ls out/  | grep jq

Here we see creating a symlink to a directory has not produced a matching symlink in the output. Something I use frequently. for example.

Some tools mangled symlinked directories, or files, some ignore them completely. Neither is acceptible.

Syndicated 2012-12-06 14:42:04 from Steve Kemp's Blog

Reassessing what I want from a simple website creation tool

Thanks very much to everybody who commented, both publicly and privately, on my previous entry.

To recap: I have three sites that were each generated by slightly different templating software I'd built and tweaked over the years. I was frustrated that the three copies of the generation tool had all drifted and diverged from each other, and was looking to setup a new (static) site.

The obvious conclusion was either:

  • Unify the creation tool I used, such that all four sites could be generated by a single tool.
  • Avoid the pain of doing that, and suffer through a process of using a well-maintained tool maintained by somebody else. ("Suffer" because tags would be different, and the layout/template syntax would change.)

I'd cheerfully decided to go down the second route, because life is short. But after having quick reads, then spending several hours investigating likely contenders I kept finding reasons why they weren't suitable.

Today I reworked my tool to succesfully generate each of the three sites. That was less annoying than expected, after I'd decided "I'll have to change my templates anyway, when I switch to a real tool".

So in the interests of sharing I placed my tool online, and wrote documentation:

This is not a U-turn. This is not a commitment to avoid the investigation of real replacements. This is just something I had to do as a cleanup and to make sure I fully understood exactly what my requirements were.

In conclusion: My requirements are now absolutely known, fixed, and understood. I still firmly intend and expect to migrate to something by the end of the year. Ideally something that will make tag pages, RSS feeds, and other clever things easy.

(Getting rid of literal shell usage in my templates, and unifying the way I auto-generate galleries via file globs was a useful change in its own regard - I always felt slightly dirty..)

Lookout for more summaries and reviews of specific tools when I've had the chance to relax and start looking again.

Syndicated 2012-12-01 16:03:03 from Steve Kemp's Blog

Simple website creation tools

I host a number of websites which are mostly static in nature, these are often hand-crafted, but three of them us a slightly hacked up creation of my own.

Given a master "template" the file foo.skx gets massaged into foo.html.

Sadly I added features randomly, and now I have three template-driven sites which are handled slightly differently. This put me in the position of having to choose between two options:

  • Unify my template-handling.
  • Use something else.

Simplifying my life is good. So I examined a list of static site generators, and a few more found by searching github.

Other than doing clever things by knowing which page is "current" I needed to do only minimal magic:

  • Conditionally include files.
  • Setup per-page CSS files.
    • e.g. This page and this page differ by the stylesheet used. (And the text content too, clearly!)
  • Setup per-page templates.

Webgen looked like a good fit, but I couldn't get per-page templates to work out. Either they would work, or I would get weird errors about blocks not being known.

Webby worked, but I didn't like it.

Poole was the next one that I got far down the road with, but it allowed only a single site-wide template. Shame because otherwise I loved its flexibility enought to tolerate writing "macros" in Python.

I've still got to test more out, but it is a fun process. I fully intend to adopt an existing tool, and not keep working on my own.

Tonight I'm going to look at a few more.

Syndicated 2012-11-29 18:24:07 from Steve Kemp's Blog

586 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!