Older blog entries for Stevey (starting at number 559)

Happy birthday to me

Recently I accidentally flooded Planet Debian with my blog feed. This was an accident caused by some of my older blog entries not having valid "Date:" headers. (I use chronicle which parses simple text files to build a blog, and if there is no Date: header present in entries it uses the CTIME of the file(s).)

So why did my CTIMEs get lost? Short version I had a drive failure and a PSU failure which lead to me rebuilding a few things and cloning a fresh copy of my blog to ~/hg/blog/.

My host is now once again OK, but during the pain the on-board sound started to die. Horribly crackly and sounding bad. I figure the PSU might have caused some collateral damage, but so far thats the only sign I see.

I disabled the on-board sound and ordered a cheap USB sound device which now provides me with perfect sound under the Squeeze release of Debian GNU/Linux.

In the past I've ranted about GNU/Linux sound. So I think it is only fair to say this time things worked perfectly - I plugged in the device, it was visible in the output of dmesg, and /proc/asound/cards and suddenly everything just worked. Playing music (mpd + sonata) worked immediately, and when I decided to try playing a movie with xine just for fun sound was mixed appropriately - such that I could hear both "song" + "movie" at the same time. Woo.

(I'm not sure if I should try using pulse-audio, or similar black magic. Right now I've just got ALSA running.)

Anyway as part of the re-deployment of my desktop I generated and pass-phrased a new SSH key, and then deployed that with my slaughter tool. My various websites all run under their own UID on my remote host, and a reverse-proxy redirects connections. So far example I have a Unix "s-stolen" user for the site stolen-souls.com, a s-tasteful user for the site tasteful.xxx, etc. (Right now I cannot remember why I gave each "webserver user" an "s-" prefix, but it made sense at the time!)

Anyway once I'd fixed up SSH keys I went on a spree of tidying up and built a bunch of meta-packages to make it a little more straightforward to re-deploy hosts in the future. I'm quite pleased with the way those turned out to be useful.

Finally I decided to do something radical. I installed the bluetile window manager, which allows you to toggle between "tiling" and "normal" modes. This is my first foray into tiling window managers, but it seems to be going well. I've got the hang of resizing via the keyboard and tweaked a couple of virtual desktops so I can work well both at home and on my work machine. (I suspect I will eventually migrate to awesome, or similar, this is very much a deliberate "ease myself into it" step.)

ObQuote: "Being Swedish, the walk from the bathroom to her room didn't need to be a modest one. " - Cashback.

Syndicated 2012-03-10 12:54:21 from Steve Kemp's Blog

5 Mar 2012 (updated 7 Mar 2012 at 14:06 UTC) »

Today I migrated from 32-bit to 64-bit, in-place

This evening I sat down and migrated my personal virtual machine from a 32-bit installation of Debian GNU/Linux to a 64-bit installation.

I've been meaning to make this change for a good few months, but it took me until this evening until I decided it was as good a time as any.

Mostly the process is painless:

  • Ensure you have a 64-bit kernel, with support for 32-bit binaries too.
  • Install the 32-bit compatibility libraries, such that your old binaries work.
  • Overwrite your binaries and libraries in-place so you have a 64-bit base system.
  • Patch it up afterwards.

I overwrote a lot of the libraries and binaries on the system such that I had a working 64-bit apt-get, dpkg, sash, etc, and associated libraries. Then once I had that I could use those tools to pull the resto of the system up to date.

One thing I hadn't counted on is that I needed to have a 64-bit version of bzip such that "apt-get update" didn't complain about errors. I suspect I could have fixed that by re-configuring my system to disable compression. Still it was easily solved.

Along the way I also shot myself in the foot by having a local caching DNS resolver, listening on 127.0.0.1, which broke. With no DNS I couldn't use apt-get - but once the problem was identified it was trivial to fix.

Anyway all seems OK now. My websites are up, email is flowing and I guess anything else can wait until the morning.

ObQuote: "Somebody's coming up. Somebody serious." - Leon

Syndicated 2012-03-05 01:12:44 (Updated 2012-03-07 14:06:59) from Steve Kemp's Blog

23 Feb 2012 (updated 7 Mar 2012 at 01:08 UTC) »

Symbiosis is wonderful

Symbiosis

Symbiosis is the collective name given to a group of Debian GNU/Linux packages which implement simple virtual hosting. It is developed by my employers Bytemark.

Symbiosis is basically a collection of configuration snippets, code, and libraries which works to offer virtual hosting in a reliable consistent and easy to understand fashion.

You implement hosting for a new domain by merely creating a directory tree. So for example you might configure the hosting for the domain example.com by running:

mkdir -p /srv/example.com/public/htdocs
echo "hello, world" >> /srv/example.com/public/htdocs/index.html

mkdir -p /srv/example.com/mailboxes/webmaster
echo "super-secret" > /srv/example.com/mailboxes/webmaster/password

mkdir -p /srv/example.com/config
echo "3l33t" > /srv/example.com/mailboxes/config/ftp-password

There you are, now http://www.example.com/ and http://example.com/ will work, and you may login to check mail with the email address webmaster@example.com via POP3, IMAP, IMAPS, or POP3S. Finally you can FTP with username example.com and be dropped into the public directory.

The mail handling is very flexible, and the webhosting supports wonderful things.

I don't generally talk about work-stuff explicitly, but we've just made a major new release of the Symbiosis system such that it works upon Squeeze and has lots of IPv6 support out of the box. (Email, DNS, HTTP, Firewalling, FTP etc.)

All in all it is simple, well-documented, and open-source with a reasonably large user-base. More external testers, users, and developers would be a wonderful thing..

Mutt Mailboxes & Idle Hooks?

Mutt is wonderful but I'm starting to get annoyed by its lack of auto-mailbox discovery.

Assuming you use procmail you might deliver mail to ~/Maildir/.foo/ and mutt won't notice that if the directory is created once it starts.

(This is because generally mailboxes are defined via "mailboxes =one =two ..", even if you use a shell snippet it won't get updated unless you re-read configuration, or re-exec mutt).

I wish it were possible to use inotify/dnotify/something magic such that everything beneath ~/Maildir would just work.

(Re-reading mailboxes manually is one solution but it is .. nasty?)

I'm thinking that of all the possible solutions one of the most potentially interesting would be to define a new hook: "idle-hook command .."

That way "command" would be executed every time the client is idle. (This is a distinct state unrelated to IMAP IDLE times.)

Nopte: There is already "mail_check" & "timeout" options. Even running a defined command immediately following the code for mail_check would be reasonable.

Reverse Proxy

I continue to use, love, and enjoy my node.js-based reverse HTTP proxy, and pub discussions seemed to suggest it is a great idea (due to flexibility) but it will never take on because people don't trust node.

I'm almost tempted to re-code it in LUA & C. But I can't help but think that would be a waste of time which would not increase adoption - after all most people use "simple" reverse proxies, and they are well suited by Apache, nginx, or even varnish.

Still no rush I suppose.

In more personal news after living in this flat for 7 years, or so, I'm getting a new bathroom designed and deployed. Good times.

In the meantime I've been steadily watching Stargate SG-1 having recently purchased a box-set of series 1-10. I've just started series six this evening, and I'm enjoying it a lot.

ObQuote: "You have been recruited by the Star League to defend the frontier against Xur and the Ko-Dan armada. " - The Last Starfighter (1984). First film I ever saw at a cinema as a child.

Syndicated 2012-02-23 19:24:21 (Updated 2012-03-07 01:08:53) from Steve Kemp's Blog

Some domains just don't learn

For the past few years the anti-spam system I run has been based on a simplified version of something I previously ran commercially.

Although the code is similar in intent there were both explicit feature removals, and simplifications made.

Last month I re-implimented domain-blacklisting - because a single company keeps ignoring requests to remove me.

So LinkedIn.com if you're reading this:

  • I've never had an account on your servers.
  • I find your junk mail annoying.
  • I suspect I'll join your site/service when hell freezes over.

I've also implemented TLD-blacklisting which has been useful.

TLD-blacklisting in my world is not about blocking mail from foo@bar.ph (whether in the envelope sender, or the from: header), instead it is about matching the reverse DNS of the connecting client.

If I recieve a connection from 1.2.3.4 and the reverse DNS of that IP address matches, say, /\.sa$/i then I default to denying it.

My real list is longer, and handled via files:

steve@steve:~$ ls /srv/_global_/blacklisted/tld/ -C
ar  br  cn  eg  hr  in  kr  lv  mn  np  ph  ro  sg  tg  ua  ve  zw
aw  cc  cy  gm  hu  is  kz  ma  my  nu  pk  rs  sk  th  ug  vn
be  ch  cz  gr  id  it  lk  md  mz  nz  pl  ru  su  tr  uy  ws
bg  cl  ec  hk  il  ke  lt  mk  no  om  pt  sa  sy  tw  uz  za

On average I'm rejecting about 2500 messagse a day at SMTP-time, and 30 messages, or so, hit my SPAM folder after being filtered with CRM114 after being accepted for delivery. (They are largely from @hotmail and @yahoo, along with random compromised machines. The amount of times I see a single mail from a host with RDNS mysql.example.org is staggering.).

(Still looking forward to the development of Haraka, a node.js version of qpsmtpd.)

ObQuote: "Mr. Mystery Guest? Are you still there? " - Die Hard

Syndicated 2012-02-05 13:24:44 from Steve Kemp's Blog

So mega-upload is gone

So the site http://megaupload.com/ has been taken offline, amidst allegations of knowingly conducting in piracy.

There are probably a lot of legitimate users who have lost access to their uploaded files, even if they were offsite backups you can imagine a user owning a website which now has a million dead-links.

This reminds me of a conversation I overheard on Jon Dowlands blog - the summary is that he'd written a (useful) tool to extract attachments from Maildir folders and was wondering how to store and access those attachments. The upshot seemed to be magical URLs of the form:

  • https://file.example.com/sha1/509c2fe2eba509e93987c3024a74d74583c274bd

The comments covered an alternative which was hash:///sha1/xxxxxxxxxxxxxxxx, which then becomes close to the magnet:// schema.

I've not yet thought things through, but I can't help thinking that with the redundency already present in the internet we should be looking at non-server-specific links. Yes there are times right now when you might want to address a specific file on a specific server - but otherwise? Wouldn't it be nice if you could just access a file from "anywhere" which happened to have the right contents?

Already my nonporn-but-definitely-adult-site makes its images available as /img/$md5sum.jpg - and similarly the storage at the back-end of my random image upload site uses SHA1 hashes to store the actual files.

To make this more complete what we need is something that crawls the internet to find files by hash; then add support in browsers. Obviously this must be async and could introduce timing issues, but fundamentally it seems like a reasonable approach to the problem of a single host going offline.

(Consider what happens if imgur.com disappears. All those links would die, yet 99% of the images would still be available somewhere.)

I'm tempted to suggest microformat format but I need to consider the matter. Right now I'm going to immediately update my current image hosts to use, at the very least:

 <a href="/foo" rel="sha1:xxxxx md5sum:xxxx">
  <img src="foo.jpg" alt="img name">
 </a>

The unfortunate thing is you cannot have a 'rel="xx"' attribute for an image. So you either have to encode it in the parent link, or add it to the alt attribute which is suboptimal.

ObQuote: "Now, they tell me I paid my debt to society." - Oceans Eleven (2001)

Syndicated 2012-01-21 12:42:37 from Steve Kemp's Blog

Some misc. updates

Security

Today I made available a 3.2.0 kernel for my KVM guest which has a bastardised version of the PID hiding patch configured:

So now on my guest, as myself, I can only see this:

steve@steve:~$ ls -l /proc/ | egrep ' [0-9]+$'
dr-xr-xr-x  7 steve users          0 Jan 13 17:22 15150
dr-xr-xr-x  7 steve users          0 Jan 13 17:29 15739
dr-xr-xr-x  7 steve users          0 Jan 13 17:29 15740
lrwxrwxrwx  1 root  root          64 Jan 13 17:20 self -> 15739

Running as root I see the full tree:

steve:~#  ls -l /proc/ | egrep ' [0-9]+$'
total 0
dr-xr-xr-x  7 root        root                 0 Jan 13 17:20 1
dr-xr-xr-x  7 root        root                 0 Jan 13 17:20 1052
dr-xr-xr-x  7 root        root                 0 Jan 13 17:20 1086
dr-xr-xr-x  7 root        root                 0 Jan 13 17:20 1101
dr-xr-xr-x  7 root        root                 0 Jan 13 17:20 1104
dr-xr-xr-x  7 root        root                 0 Jan 13 17:21 1331
dr-xr-xr-x  7 pdnsd       proxy                0 Jan 13 17:21 14409
dr-xr-xr-x  7 root        root                 0 Jan 13 17:21 14519
..

This (obviously) affects output from top etc too. It is a neat feature which I think is worth having, but time will tell..

mod_ifier

A long time ago I put together an Apache module which allowed the evaluation of security rules against incoming HTTP requests. mod_ifier was largely ignored by the world. But this week it did receive a little attention.

The recent rash of Hash Collision attacks inspired inspired a fork with parameter filtering. Neat.

Otherwise nothing too much to report - though I guess I didn't actually share the link to the RESTful file store I mentioned previously. Should you care you can find it here:

ObQuote: "I saw a man, he danced with his wife" - Chicago, Frank Sinatra

Syndicated 2012-01-13 17:33:46 from Steve Kemp's Blog

Review of the Panasonic Lumix FS-16 camera

Recently I've been wanting to replace my old point and shoot camera, a Canon PowerShot A620. I've got a pair of DLSR cameras and I do frequently carry one of them out with me, but there are undoubtedly occasions where I'd rather not bother, or where I find myself wanting to take a picture without having one to hand.

Unfortunately the PowerShot is pretty large itself, although significantly less so than the DSLRS I possess. (I cannot remember the last time I used the PowerShot outside my flat, that is how rarely it goes outdoors).

The PowerShot has been a good camera to me for many years and the three features I liked the most:

  • A real view-finder.
  • It runs on 4x AA batteries; easy to find.
  • Shoots (smallish) movies.

Picking a replacement camera, even with the help of fine comparison websites like snapsort.com is hard. Cameras have moved on and "improved" a lot over the last few years - to the extent that finding one with a built-in viewfinder is hard. Finding one with a built-in viewfinder and running on easily replaceable batteries was virtually impossible.

Eventually I settled on the Panasonic Lumix FS16, which omits both:

  • Integrated rechargeable battery.
  • LCD-only viewfinder.

The way that you use the LCD or viewfinder differs pretty significantly, but the LCD wasn't as bad as I'd feared:

ViewFinder

You hold the camera to your eye, and press the appropriate buttons.

LCD

You typically hold the camera at arms length, which means you're prone to shaking your hands/arms and getting blurry shots.

Because you're holding the camera relatively far away from your eyes if you have the sun at your back you're liable to need to squint.

The LCD on the Lumix FS-16 isn't amazing, but neither is it horrific and it is better than expected in dark locations.

So after a week what do I think? On the whole it is a fine camera, better than the PowerShot in many ways, and while it has draw-backs none are deal-breakers:

Size

The best camera is one you have with you; on that basis this camera is a clear win being smaller, lighter, and more compact than the Canon.

I've taken this camera with me, randomly, to several places and returned with useful and interesting images.

Low Light

Low light performance is pretty poor. With only one manual control you see noise if you're shooting in gloomy pubs, and outdoors. With the flash you can get acceptable pictures if you're careful - but its a tricky thing to get right.

(To update this a little: Outdoors at night? No. In a pub with poor lighting you'll be alright.)

Manual Controls

The camera features precisely two manual controls:

  • "Flash on" vs. "Flash off".
  • ISO can be changed from: Auto, 100, 200, 400, 800, 1200 and 1600.

There is no notion of shutter speed, nor is there any ability to change the aperture size. (Though both these values are displayed on the screen as you take a picture I wonder why? As you can't change anything you can't use the information in any useful fashion, and presumably a non-camera-person wouldn't understand what these numbers represent.)

The lack of these two controls is a little galling, but pretty common for the low-end P&S cameras.

Video Recording

There is no external MIC so sounds aren't great, but they're not horrible either.

Video recordings are limited to the smaller of 8 minutes or 2Gb. So no long films, but short ones look fine. Just be aware that once you start recording focus won't change, nor will zooming work.

Compared to the canon the quality is improved; but the Canon allowed you to (optically) zoom whilst recording. Here you can only zoom with your feet.

Recharging Time

When I received the camera it took about an hour to charge. The battery life seems reasonable - the level is 2/3 a week later and I've been shooting, reviewing, and deleting regularly.

(Note: I never use USB to transfer pictures, I always remove the card and plug it into my PC. Whether this makes a difference to battery life I don't know.)

Controls

Physical controls are reasonable. There is a mechanical slide-switch to turn on/off. I like that, as it is less prone to being knocked by keys, change, etc.

There is also a physical slide-switch to change from "shoot" to "review current images/videos". (Same as my Canon) I think this is a mistake, and don't see why it can't be a soft-button.

Full Auto

There are several modes available in the camera (remember the caveat about lack of aperture/shutter speed) I've been using both full-auto and manual modes, and both are good. Full auto would suit most people - it has clever face-tracking.

Focusing Speed

As expected this is not stellar. Walking to the corner shop the other lunchtime I found a cat in the road, I talked to her and she rubbed herself against my ankles. Could I focus fast enough to catch her looking up at me? No.

For static scenes, and candid shots of people it'll suffice. For fast action and moving children probably not a chance.

On balance, the upgrade was worthwhile.

ObQuote: "I don't mean to lecture and I don't mean to preach. And I know I'm not your father..." - Spider-Man

Syndicated 2012-01-07 15:08:49 from Steve Kemp's Blog

6 Jan 2012 (updated 7 Jan 2012 at 00:27 UTC) »

Tonight I've mostly been using Sinatra

This evening I've mostly been using Sinatra to build a little file storage service which uses a REST API.

That means I can upload a file:

skx@birthday:~/hg/sinatra$ curl -X PUT -F file=@/etc/fstab http://localhost:4567/
{"id":"dbd1bdc11b5a1a8e80588a135648b4c2edffb49a","path":"/"}

Download that same file:

skx@birthday:~/hg/sinatra$ curl -X GET -F id=dbd1bdc11b5a1a8e80588a135648b4c2edffb49a  \
   http://localhost:4567/
# /etc/fstab: static file system information.
..
/dev/cdrom        /media/cdrom0   udf,iso9660 user,noauto     0       0

Get an index of files:

skx@birthday:~/hg/sinatra$ curl http://localhost:4567/
[{"id":"dbd1bdc11b5a1a8e80588a135648b4c2edffb49a","type":"file"}]

And finally we can delete a file:

skx@birthday:~/hg/sinatra$ curl -X DELETE -F "id=dbd1bdc11b5a1a8e80588a135648b4c2edffb49a" \
  http://localhost:4567/
Removed

We can also upload to different paths so we can replicate a file-system if we wanted to. (I added in "type" to hold either "file" or "directory", though I guess if we were to code up a FUSE client we'd want to store things like ctime, UID, GID, etc. THe list operation will show both files and sub-directories)

The code was trivial once I got the hang of Sinatra, and I'm pretty pleased with it so far. I don't yet need to use it for anything, but I'm thinking of unifying the way that I store images on a couple of sites - and fetching them via JSON and Javascript might be an option this was an experiment in that direction. (Though I'd probably want to hook in rsync so we replicated the eventual upload location for safety.)

In other news I've been all organized and upgraded the kernel on my guest:

steve@steve:~$ uptime
 22:00:28 up  4:18,  1 user,  load average: 0.14, 0.05, 0.05
steve@steve:~$ uname -r
3.2.0-kvm-hosting.org-i386-20120106

So for once I'm up to date with a cutting edge kernel. Happy times.

ObQuote: "How you expect to run with the wolves come night when you spend all day sparring with the puppies? " - The Wire (Omar)

Syndicated 2012-01-06 22:04:06 (Updated 2012-01-07 00:27:37) from Steve Kemp's Blog

The final updates of 2011

I've been informed by a couple of people that the Debian Administration site is down. Sadly it is; at the moment the host isn't showing anything on the serial console and remotely power-cycling it isn't showing any signs of life.

At this time of year I don't want to drag anybody in to take care of it, so ETA on recovery/replacement hardware is Monday/Tuesday.

In other news I've made it to year five of the KVM hosting sub-project/thing. Originally started as a Xen host its been running happily for quite some time. I suspect next year, or the year after that the price/specification ratio will end up losing out and we'll cancel the whole thing - but there are no immediate reasons to make any change.

Finally I knocked up a simple tool to validate my TinyDNS records prior to uploading them. It is simplistic, but adequate to catch the kind of mistakes I make:

Honestly it probably wants to be rationalised a little more - and check records more carefully. e.g. Ensure that the host a CNAME refers to itself exists, and making sure that the nameservers specified are valid.

I just wanted to make something quick after accidentally uploading a zonefile where I'd managed to fat-finger several important records. le sigh.

Oddly enough asking on serverfault.com showed no real suggestions - other than actually running tinydns locally and doing a zone-xfer to validate records. Overkill and harder than I'd like.

Happy New year if you care about such things..

"I finished growing up, Léon. I just get older. " - Leon

Syndicated 2011-12-31 20:29:45 from Steve Kemp's Blog

So I'm a peddler of smut nowadays.

Over the Christmas period I've not been doing too much.

December was a month that started out pretty well, in the first 10 days I had five models/friends/random-folk come to pose for me. (I have a few folk lined up for mid-January, but things tailed off quickly this month due to people having little free time).

I spent a while mulling over what I was doing with images, as I've recently been doing a fair number of more NSFW images - In December this and this were my two favourite shots.

I've posted NSFW images in various places over the past year (with permission; some volunteers/victims/friends don't want me to share anything, so I don't. They just hang on my walls.) but I was never consistent.

Anyway I realised that .xxx domains are now available, so I figured I'd snarf one up and use that. That lead to tasteful.xxx - which is mostly full of images I didn't actually take, but that will change. (Sadly "artistic" was gone!)

In more on-topic news I reported #651896 - a trivial security issue in another setgid(games) binary. I've got a couple more of those to tidy up and report in the near future.

ObQuote: "Bright light. Bright light." - Gremlins

Syndicated 2011-12-26 18:08:05 from Steve Kemp's Blog

550 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!