Stevey is currently certified at Master level.

Name: Steve Kemp
Member since: N/A
Last Login: 2013-01-05 09:46:29

FOAF RDF Share This

Homepage: http://www.steve.org.uk/

Notes:

For the curious I live in Edinburgh, Scotland.

I'm a big believer in the benefits of the open source
software. I've written, or contributed, to a large
number of Open Source projects including GNU Emacs,
GNUTella, GoGo, GNUMP3d, MP3Blaster.

Nowadays I guess the most visible thing I do in my spare
time is run a site I created for Debian GNU/Linux system administration:

* http://www.debian-administration.org/

Projects

Articles Posted by Stevey

Recent blog entries by Stevey

Syndication: RSS 2.0

On de-duplicating uploaded file-content.

This evening I've been mostly playing with removing duplicate content. I've had this idea for the past few days about object-storage, and obviously in that context if you can handle duplicate content cleanly that's a big win.

The naive implementation of object-storage involves splitting uploaded files into chunks, storing them separately, and writing database-entries such that you can reassemble the appropriate chunks when the object is retrieved.

If you store chunks on-disk, by the hash of their contents, then things are nice and simple.

The end result is that you might upload the file /etc/passwd, split that into four-byte chunks, and then hash each chunk using SHA256.

This leaves you with some database-entries, and a bunch of files on-disk:

/tmp/hashed/ef267892ee080862c96a8d2d05de62f48e20f0875f27379e7d58c73ea4455bf1
/tmp/hashed/a378977155fb42bb006496321cbe31f74cbda803c3f6ca590f30e76d1afad921
..
/tmp/hashed/3805b0245bc8375be7125ae228eef711552ac082ffb9bf8756e2964a2393a9de

In my toy-code I wrote out the data in 4-byte chunks, which is grossly ineffeciant. But the value of using such small pieces is that there is liable to be a lot of collisions, and that means we save-space. It is a trade-off.

So the main thing I was experimenting with was the size of the chunks. If you make them too small you lose I/O due to the overhead of writing out so many small files, but you gain because collisions are common.

The rough testing I did involved using chunks of 16, 32, 128, 255, 512, 1024, 2048, and 4096 bytes. As sizes went up the overhead shrank, but also so did the collisions.

Unless you could handle the case of users uploading a lot of files like /bin/ls which are going to collide 100% of the time with prior uploads using larger chunks just didn't win as much as I thought they would.

I wrote a toy server using Sinatra & Ruby, which handles the splitting/hashing/and stored block-IDs in SQLite. It's not so novel given that it took only an hour or so to write.

The downside of my approach is also immediately apparent. All the data must live on a single machine - so that reassmbly works in the simple fashion. That's possible, even with lots of content if you use GlusterFS, or similar, but it's probably not a great approach in general. If you have large capacity storage avilable locally then this might would well enough for storing backups, etc, but .. yeah.

Syndicated 2015-05-07 00:00:00 from Steve Kemp's Blog

A weekend of migrations

This weekend has been all about migrations:

Host Migrations

I've migrated several more systems to the Jessie release of Debian GNU/Linux. No major surprises, and now I'm in a good state.

I have 18 hosts, and now 16 of them are running Jessie. One of them I won't touch for a while, and the other is a KVM-host which runs about 8 guests - so I won't upgraded that for a while (because I want to schedule the shutdown of the guests for the host-reboot).

Password Migrations

I've started migrating my passwords to pass, which is a simple shell wrapper around GPG. I generated a new password-managing key, and started migrating the passwords.

I dislike that account-names are stored in plaintext, but that seems known and unlikely to be fixed.

I've "solved" the problem by dividing all my accounts into "Those that I wish to disclose post-death" (i.e. "banking", "amazon", "facebook", etc, etc), and those that are "never to be shared". The former are migrating, the latter are not.

(Yeah I'm thinking about estates at the moment, near-death things have that effect!)

Syndicated 2015-05-04 00:00:00 from Steve Kemp's Blog

Validating puppet manifests via git hooks.

It looks like I'll be spending a lot of time working with puppet over the coming weeks.

I've setup some toy deployments on virtual machines, and have converted several of my own hosts to using it, rather than my own slaughter system.

When it comes to puppet some things are good, and some things are bad, as exected, and as any similar tool (even my own). At the moment I'm just aiming for consistency and making sure I can control all the systems - BSD, Debian GNU/Linux, Ubuntu, Microsoft Windows, etc.

Little changes are making me happy though - rather than using a local git pre-commit hook to validate puppet manifests I'm now doing that checking on the server-side via a git pre-receive hook.

Doing it on the server-side means that I can never forget to add the local hook and future-colleagues can similarly never make this mistake, and commit malformed puppetry.

It is almost a shame there isn't a decent collection of example git-hooks, for doing things like this puppet-validation. Maybe there is and I've missed it.

It only crossed my mind because I've had to write several of these recently - a hook to rebuild a static website when the repository has a new markdown file pushed to it, a hook to validate syntax when pushes are attempted, and another hook to deny updates if the C-code fails to compile.

Syndicated 2015-04-27 00:00:00 from Steve Kemp's Blog

skx-www upgraded to jessie

Today I upgraded my main web-host to the Jessie release of Debian GNU/Linux.

I performed the upgraded by changing wheezy to jessie in the sources.list file, then ran:

apt-get update
apt-get dist-upgrade

For some reason this didn't upgrade my kernel, which remained the 3.2.x version. That failed to boot, due to some udev/systemd issues (lots of "waiting for job: udev /dev/vda", etc, etc). To fix this I logged into my KVM-host, chrooted into the disk image (which I mounted via the use of kpartx), and installed the 3.16.x kernel, before rebooting into that.

All my websites seemed to be OK, but I made some changes regardless. (This was mostly for "neatness", using Debian packages instead of gems, and installing the attic package rather than keeping the source-install I'd made to /opt/attic.)

The only surprise was the significant upgrade of the Net::DNS perl-module. Nothing that a few minutes work didn't fix.

Now that I've upgraded the SSL-issue I had with redirections is no longer present. So it was a worthwhile thing to do.

Syndicated 2015-04-18 00:00:00 from Steve Kemp's Blog

Subject - Verb Agreement

There's pretty much no way that I can describe the act of cutting a live, 240V mains-voltage, wire in half with a pair of scissors which doesn't make me look like an idiot.

Yet yesterday evening that is exactly what I did.

There were mitigating circumstances, but trying to explain them would make little sense unless you could see the scene.

In conclusion: I'm alive, although I almost wasn't.

My scissors? They have a hole in them.

Syndicated 2015-04-14 00:00:00 from Steve Kemp's Blog

759 older entries...

 

Stevey certified others as follows:

  • Stevey certified skx as Master
  • Stevey certified zx80user as Journeyer
  • Stevey certified faw as Apprentice
  • Stevey certified Liedra as Journeyer
  • Stevey certified ladypine as Journeyer
  • Stevey certified Sarah as Apprentice
  • Stevey certified Ward as Master
  • Stevey certified chipx86 as Journeyer
  • Stevey certified johnnyb as Journeyer
  • Stevey certified perlpimp as Journeyer
  • Stevey certified CaptainNemo as Journeyer
  • Stevey certified mobius as Apprentice
  • Stevey certified tjansen as Journeyer
  • Stevey certified auspex as Apprentice
  • Stevey certified laymusic as Journeyer
  • Stevey certified apeiro as Journeyer
  • Stevey certified dneighbors as Master
  • Stevey certified alejandro as Journeyer
  • Stevey certified scandal as Master
  • Stevey certified moray as Journeyer
  • Stevey certified sacha as Apprentice
  • Stevey certified Malkin as Journeyer
  • Stevey certified dria as Journeyer
  • Stevey certified Jordi as Journeyer
  • Stevey certified Mysidia as Journeyer
  • Stevey certified dirtyrat as Journeyer
  • Stevey certified bdelacretaz as Apprentice
  • Stevey certified braden as Journeyer
  • Stevey certified pencechp as Apprentice
  • Stevey certified brouhaha as Journeyer
  • Stevey certified fejj as Journeyer
  • Stevey certified hanna as Apprentice
  • Stevey certified aero6dof as Apprentice
  • Stevey certified gman as Journeyer
  • Stevey certified dlecorfec as Apprentice
  • Stevey certified jpr as Journeyer
  • Stevey certified Pizza as Journeyer
  • Stevey certified kroah as Master
  • Stevey certified wingo as Journeyer
  • Stevey certified jwz as Master
  • Stevey certified ploppy as Master
  • Stevey certified nosinut as Journeyer
  • Stevey certified Radagast as Journeyer
  • Stevey certified Jody as Master
  • Stevey certified ariya as Apprentice
  • Stevey certified nausicaa as Journeyer
  • Stevey certified dan as Master
  • Stevey certified kappa as Apprentice
  • Stevey certified slamb as Apprentice
  • Stevey certified stevebaker as Journeyer
  • Stevey certified gregor as Journeyer
  • Stevey certified maelstorm as Apprentice
  • Stevey certified palm as Apprentice
  • Stevey certified Artimage as Journeyer
  • Stevey certified bluets as Apprentice
  • Stevey certified jds as Journeyer
  • Stevey certified steve as Apprentice
  • Stevey certified elanthis as Apprentice
  • Stevey certified afayolle as Journeyer
  • Stevey certified bonzini as Journeyer
  • Stevey certified jml as Apprentice
  • Stevey certified ciphergoth as Journeyer
  • Stevey certified Fefe as Master
  • Stevey certified Denny as Journeyer
  • Stevey certified sethcohn as Journeyer
  • Stevey certified bjf as Apprentice
  • Stevey certified sdodji as Journeyer
  • Stevey certified raph as Master
  • Stevey certified jarod as Apprentice
  • Stevey certified StevenRainwater as Journeyer
  • Stevey certified Surfr as Apprentice
  • Stevey certified rlevin as Journeyer
  • Stevey certified ike as Apprentice
  • Stevey certified ebf as Journeyer
  • Stevey certified chakie as Journeyer
  • Stevey certified gstein as Master
  • Stevey certified dtype as Apprentice
  • Stevey certified pompeiisneaks as Journeyer
  • Stevey certified acme as Master
  • Stevey certified lsdrocha as Apprentice
  • Stevey certified mjg59 as Journeyer
  • Stevey certified squrl as Journeyer
  • Stevey certified physos as Apprentice
  • Stevey certified rasmus as Master
  • Stevey certified jelle as Journeyer
  • Stevey certified chrisime as Journeyer
  • Stevey certified julian as Master
  • Stevey certified eliot as Journeyer
  • Stevey certified sh as Journeyer
  • Stevey certified mascot as Apprentice
  • Stevey certified SyntaxPolice as Journeyer
  • Stevey certified bytesplit as Apprentice
  • Stevey certified nymia as Journeyer
  • Stevey certified thomasvs as Master
  • Stevey certified Bram as Journeyer
  • Stevey certified pfremy as Apprentice
  • Stevey certified todd as Master
  • Stevey certified bma as Journeyer
  • Stevey certified coolvibe as Journeyer
  • Stevey certified deekayen as Journeyer
  • Stevey certified fxn as Journeyer
  • Stevey certified bgeiger as Apprentice
  • Stevey certified angelsun as Journeyer
  • Stevey certified andrelop as Apprentice
  • Stevey certified jono as Master
  • Stevey certified groom as Apprentice
  • Stevey certified axboe as Master
  • Stevey certified jennv as Journeyer
  • Stevey certified tseaver as Journeyer
  • Stevey certified duncanm as Apprentice
  • Stevey certified jc as Apprentice

Others have certified Stevey as follows:

  • fxn certified Stevey as Journeyer
  • wingo certified Stevey as Master
  • faw certified Stevey as Master
  • Liedra certified Stevey as Journeyer
  • ladypine certified Stevey as Journeyer
  • CaptainNemo certified Stevey as Journeyer
  • apeiro certified Stevey as Journeyer
  • dneighbors certified Stevey as Journeyer
  • Jordi certified Stevey as Journeyer
  • Mysidia certified Stevey as Master
  • braden certified Stevey as Journeyer
  • Chicago certified Stevey as Master
  • pencechp certified Stevey as Master
  • brouhaha certified Stevey as Master
  • wardv certified Stevey as Journeyer
  • Pizza certified Stevey as Master
  • jrf certified Stevey as Journeyer
  • ariya certified Stevey as Master
  • maelstorm certified Stevey as Journeyer
  • palm certified Stevey as Journeyer
  • alfie certified Stevey as Master
  • donscarletti certified Stevey as Master
  • afayolle certified Stevey as Journeyer
  • bonzini certified Stevey as Journeyer
  • ciphergoth certified Stevey as Journeyer
  • domi certified Stevey as Master
  • sdodji certified Stevey as Master
  • jarashi certified Stevey as Master
  • jarod certified Stevey as Master
  • ebf certified Stevey as Journeyer
  • lsdrocha certified Stevey as Master
  • alexm certified Stevey as Journeyer
  • richdawe certified Stevey as Journeyer
  • sh certified Stevey as Journeyer
  • mascot certified Stevey as Master
  • realblades certified Stevey as Apprentice
  • lerdsuwa certified Stevey as Master
  • bytesplit certified Stevey as Apprentice
  • thom certified Stevey as Master
  • coolvibe certified Stevey as Master
  • speeder certified Stevey as Master
  • broonie certified Stevey as Journeyer
  • angelsun certified Stevey as Journeyer
  • MAK certified Stevey as Master

[ Certification disabled because you're not logged in. ]

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!

X
Share this page