Older blog entries for Stevey (starting at number 679)

That is it, I'm going to do it

That's it, I'm going to do it: I have now committed myself to writing a scalable, caching, reverse HTTP proxy.

The biggest question right now is implementation language; obviously "threading" of some kind is required so it is a choice between Perl's anyevent, Python's twisted, Rubys event machine, or node.js.

I'm absolutely, definitely, not going to use C, or C++.

Writing a a reverse proxy in node.js is almost trivial, the hard part will be working out which language to express the caching behaviour, on a per type, and per-resource basis.

I will ponder.

Syndicated 2014-03-22 14:49:49 from Steve Kemp's Blog

Time to mix things up again

I'm currently a contractor, working for/with Dyn, until April the 11th.

I need to decide what I'm doing next, if anything. In the meantime here are some diversions:

Some trivial security issues

I noticed and reported two more temporary-file issues insecure temporary file usage in apt-extracttemplates (apt), and libreadline6: Insecure use of temporary files - in _rl_trace.

Neither of those are particularly serious, but looking for them took a little time. I recently started re-auditing code, and decided to do three things:

• Download the source code to every package installed upon this system.
• Download the source code to all packages matching the pattern ^libpam-, and ^libruby-*.

I've not yet finished slogging through the code, but my expectation will be a few more issues. I'll guess 5-10, given my cynical nature.

NFS-work

I've been tasked with the job of setting up a small cluster running from a shared and writeable NFS-root.

This is a fun project which I've done before, PXE-booting a machine and telling it to mount a root filesystem over NFS is pretty straight-forward. The hard part is making that system writeable, such that you can boot and run "apt-get install XX". I've done it in the past using magic filesystems, or tmpfs. Either will work here, so I'm not going to dwell on it.

Another year

I had another birthday, so that was nice.

My wife took me to a water-park where we swam like fisheseses, and that tied in nicely with a recent visit to Deep Sea World, where we got to walk through a glass tunnel, beneath a pool FULL OF SHARKS, and other beasties.

Beyond that I received another Global Knife, which has now been bloodied, since I managed to slice my finger open chopping mushrooms on Friday. Oops. Currently I'm in that annoying state where I'm slowly getting used to typing with a plaster around the tip of my finger, but knowing that it'll have to come off again and I'll get confused again.

Linux Distribution

I absolutely did not start working on a "linux distribution", because that would be crazy. Do I look like a crazy-person?

All I did was play around with GNU Stow, and ponder the idea of using a minimal LibC and GNU Stow to organize things.

It went well, but the devil is always in the details.

I like the idea of a master-distribution which installs pam, ssh, etc, but then has derivitives for "This is a webserver", "This is a Ruby server", and "This is a database server".

Consider it like task-selection, but with higher ambition.

There's probably more I could say; a new kitchen sink (literally) and a new tap have made our kitchen nicer, I've made it past six months of regular gym-based workouts, and I didn't die when I went to the beach in the dark the other night, so that was nice.

Umm? Stuff?

Have a nice day. Thanks.

Syndicated 2014-03-20 11:19:57 from Steve Kemp's Blog

Sending commands to your running mail-client

So I wrote a mail client, and this morning I added the ability for it to receive input from a Unix domain socket.

In one terminal I have my email client open. In another I run:

lumailctl /tmp/foo.sock "open('/home/skx/Maildir/.livejournal.2014/');"

That opens the unix domain socket, and pipes the following command to it:

open('/home/skx/Maildir/.livejournal.2014/');

The mail client has already got the socket open, and the end result is that my mail client suddenly opens the specified mail folder, and redraws itself.

Neat.

The "open" function is obviously a lua function, which builds upon the lua primitives the client understands:

function open( folder )
   clear_selected_folders()
   set_selected_folder(folder)
   index()
end

Obviously this would be woefully insecure if it were released like this. Later I'll wire up some lua function to establish the socket, such that the user specifies where the socket is created (their home directory, ideally), and it doesn't run by default.

Syndicated 2014-03-17 11:10:20 from Steve Kemp's Blog

So load-balancers are awesome

When I was recently talking about load-balancers, and automatically adding back-ends, not just removing bad ones that go offline, I obviously spent a while looking over some.

There are several dedicated load-balancers packaged for Debian GNU/Linux, including:

• fair
• pen
• haproxy
• pound

In addition to actual dedicated load-balancers there are things that can be coerced into running in that way: apache2, varnish, squid, nginx, & etc.

Of the load-balancers I was immediately drawn to both pen and pound, because they have command line tools ("penctl" and "poundctl" respectively) for adding/removing/updating the running configuration.

Pen I've been using for a couple of days now, and although it suffers from some security issues I'm confident they will be resolved in the near future. (#741370)

My only outstanding task is to juggle some hosts around and stress-test the pair of them a little more before deciding on a winner.

In other news I kinda regret the whole blogspam.net API. I'd have had a far simpler life if I'd just ran the damn thing as a DNSBL in the first place. (That's essentially how it operates on the whole anyway. Submit spammy comments for long enough and you're just blacklisted, thereafter.)

Syndicated 2014-03-14 13:49:16 from Steve Kemp's Blog

Discovering back-end servers automatically?

Recently I've been pondering how to do service discovery.

Pretend you have a load balancer which accepts traffic and routes incoming requests to different back-ends. The loadbalancer might be pound, varnish, haproxy, nginx, or similar. The back-ends might be node applications, apache, or similar.

The typical configuration of the load-balancer will read:

# forward

# backends
backend web1  { .host = "10.0.0.4"; }
backend web2  { .host = "10.0.0.6"; }
backend web3  { .host = "10.0.0.5"; }

#  afterword

I've seen this same setup in many situations, and while it can easily be imagined that there might be "random HTTP servers" on your (V)LAN which shouldn't receive connections it seems like a pain to keep updating the backends.

Using UDP/multicast broadcasts it is trivial to announce "Hey I'm a HTTP-server with the name 'foo'", and it seems to me that this should allow seamless HTTP load-balancing.

To be more explicit - this is normal:

• The load-balancer listens for HTTP requests, and forwards them to back-ends.
• When back-ends go away they stop receiving traffic.

What I'd like to propose is another step:

• When a new back-end advertises itself with the tag "foo" it should be automatically added and start to receive traffic.

i.e. This allows backends to be removed from service when they go offline but also to be added when they come online. Without the load-balancer needing its configuration to be updated.

This means you'd not give a static list of back-ends to your load-balancer, instead you'd say "Route traffic to any service that adfvertises itself with the tag 'foo'.".

VLANS, firewalls, multicast, udp, all come into play, but in theory this strikes me as being useful, obvious, and simple.

(Failure cases? Well if the "announcer" dies then the backend won't get traffic routed to it. Just like if the backend were offline. And clearly if a backend is announced, but not receiving HTTP-requests it would be dropped as normal.)

If I get the time this evening I'll sit down and look at some load-balancer source code to see if any are written in such a way that I could add this "broadcast discovery" as a plugin/minor change.

Syndicated 2014-03-11 13:28:36 from Steve Kemp's Blog

Time to get back to my roots: Perl

Today I wrote a perl Test::RemoteServer module:

#!/usr/bin/perl -w -I.

use strict;
use warnings;

use Test::More tests => 4;
use Test::RemoteServer;

#
#  Ping Tests
#
ping_ok( "192.168.0.1",       "Website host is up: IPv4" );
ping6_ok( "www.steve.org.uk", "Website host is up: IPv6" );

#
#  Socket tests
#
socket_open( "ipv4.steve.org.uk", "2222", "OpenSSH is running" );
socket_closed( "ipv4.steve.org.uk", "22", "OpenSSH is not available on :22" );

I can see a lot of value in defining tests that are carried out against remote hosts - even if they're more basic than the kind of comprehensive testing you'd get via Custodian, Nagios, etc.

Being able to run "make test" and remotely probe services is cool.

Unfortunately I suspect the new-hotness is to couple the testing with your Chef, Puppet, CFengine, Slaughter, Ansible, etc, policies. That way you have two things:

• A consistent way to define system-state.
• A consistent way to test that the damn thing worked.

Coming to CPAN in the near future anyway, I can throw it up on Github in advance if there is any interest..

Syndicated 2014-03-07 21:46:03 from Steve Kemp's Blog

So I bought some new hardware, for audio purposes.

This week I received a logitech squeezebox radio, which is basically an expensive toy that allows you to listen to either "internet radio", or music streamed from your own PC via a portable device that accesses the network wirelessly.

The main goal of this purchase was to allow us to listen to media stored on a local computer in the bedroom, or living-room.

The hardware scans your network looking for a media server, so the first step is to install that:

• The slimdevices wiki has decent information.
  • Or you can find the code on github.com.

The media-server has a couple of open ports; one for streaming the media, and one for a user-browsable HTML interface. Interestingly the radio-device shows up in the web-interface, so you can mess around with the currently loaded playlist from your office, while your wife is casually listening to music in the bedroom. (I'm not sure if that's a feature or not yet ;)

Although I didn't find any alternative server-implementations I did find a software-client which you can use to play music from the central server - slimp3slave - and again you can push playlists, media, etc, to this.

My impressions are pretty positive; the device was too expensive, certainly I wouldn't buy two, but it is functional. The user-interface is decent, and the software being available and open is a big win.

Downsides? No remote-control for the player, because paying an additional £70 is never going to happen, but otherwise I can't think of anything.

(Shame the squeezebox product line seems to have been cancelled (?))

Although I did start hacking a C & Lua alternative, it looks like there are enough implementations out there that I don't feel so strongly any more.

I'm working in a different way to most people, rather than sort mails at delivery time I'm going to write a trivial daemon that will just watch ~/Maildir/.Incoming, and move mails out of there. That means that no errors will cause mail to be lost at SMTP/delivery time.

I'm going to base my work on Email::Filter since it offers 90% of the primitives I want. The only missing thing is the ability to filter mails via external commands which has now been reported as a bug/omission.

Syndicated 2014-03-06 15:48:14 from Steve Kemp's Blog

Replacing ugly things would save the world many hours

There are some tools that we use daily, whether we realize it or not, that are unduly ugly. Over time you learn to use them and you forget just how hard they are to learn, and you take it for granted.

Today I had to guide somebody through using procmail, and I'd forgotten how annoying it is.

In brief I use procmail in three ways, each of which I had to document:

• Run a command, given a new email, and replace the original email with the output of that command.
• Run a command, silently. Just for fun.
• Match a regular expression on a header-field, and file accordingly.
  • Later extended to matching regexps on multiple headers. ("AND" + "OR" )

There are some projects that are too entrenched to ever be replaced ("make", I'm looking at you), but procmail? I reckon there's a chance a replacement would be useful, quickly.

Then again, maybe I'm biased.

Syndicated 2014-03-05 18:56:11 from Steve Kemp's Blog

Anti-social coding

I get all excited when I load up Github's front-page and see something like:

"robyn has forked skx/xxx to robyn/xxx"

I wonder what they will do, what changes do they have in mind?

Days pass, and no commits happen.

Anti-social coding: Cloning the code, I guess in case I delete my repository, but not intending to make any changes.

Syndicated 2014-03-03 10:11:50 (Updated 2014-03-03 11:17:35) from Steve Kemp's Blog

Some direction, some distraction

It seems that several people replied to the effect that they would pay people to take care of applying security updates, or even configuring adhoc things such as wikis, graphite, and MySQL.

Not enough people to rely upon, but perhaps there is scope for remote stuff being done in exchange for folding-money. (Of course some of those that replied are in foreign countries which makes receiving payment an annoyance, that's a separate problem though.)

Food for thought.

In the meantime I've settled into my use of lighttpd, which I've recently migrated to.

One interesting thing is that you can set your own "Server Name" directive:

# Set server name/version
server.tag = "lighttpd/(steve)"

This value is used by mod_dirlisting, so for example if you examine a directory which doesn't contain an index.html file you see the server-name. Cute.

Well cute unless, or until, somebody sets:

# Set server name/version
server.tag = "<script>alert(3)</script>"

That does indeed show javascript to all your visitors. Not a security problem itself, as you need to be root on the remote site. If you're root in the remote server you could just modify the actual HTML pages being served to include your javascript. That said it's a little icky.

The following patch avoids the issue:

--- mod_dirlisting.c.org	2014-02-26 00:14:43.296373275 +0000
+++ mod_dirlisting.c	2014-02-26 00:16:28.332371547 +0000
@@ -618,7 +618,7 @@
 		} else if (buffer_is_empty(con->conf.server_tag)) {
 			buffer_append_string_len(out, CONST_STR_LEN(PACKAGE_DESC));
 		} else {
-			buffer_append_string_buffer(out, con->conf.server_tag);
+                        buffer_append_string_encoded(out, CONST_BUF_LEN(con->conf.server_tag), ENCODING_HTML);
 		}

 		buffer_append_string_len(out, CONST_STR_LEN(

Syndicated 2014-02-27 13:21:01 from Steve Kemp's Blog