Advogato blog for Stevey

The problem with the English language is all those pesky words

Sun, 29 Jun 2008 12:21:11 GMT

There exist many bayasian/statistical spam filters, ranging from products such as spambayes, and spamassassin, to crm114. Each of them works in their own way. Having used and tested almost all of them I've noticed a common flaw.

The vast majority of spam-filters struggle to correctly classify "419 scam" mails, lottery fraud, and similar mails.

Why is that? In general, having read hundreds of these mails, I can see several things that are common in these kind of the mails:

Mention of currency in both numeric and word forms. ($1,000,000 + 1 million US dollars)
Mention of a country / nationality (Sierra Lione, Nigerian)
Mention of a reference/claim number and often "official address".
Christian references.
Greetings such as "dear friend", and mentions of discretion/secrecy.
Size. (A scam mail is typically greater in length than an average spam mail).

Whilst none of these individually are indicative of a scam mail it is interesting to count their combined occurance.

I've written a toy program to count these things, and so far the success rate is >60% which is a reasonable start - providing this kind of detection occurs after normal filtering.

I may experiment further, but I figured a public query on scam detection might be appropriate.

Whilst the detecting a scam mail is a subset of detecting a spam email there are probably simplifications that may be made, and exploring those wouldn't be a bad thing.

ObQuote: Buffy.

Sorry I'm late. Work was murder.

Tue, 24 Jun 2008 16:11:22 GMT

I've spent a few hours recently looking at building RPM packages of GNU/Linux kernels, which has been a frustrating process.

There are many many online guides which give the impression that this is actually a pretty complex process. For example How To Compile A Kernel - The CentOS Way guide. (Did I mention how bad most of the howtoforge guides are recently?)

So, after fiddling around for an afternoon and getting lost I decided to abandon the process.

Here is a tested process for building a binary RPM kernel package:

cd linux-2.6.24.7/
make rpm

Yes this works just fine upon a Centos 5.x machine - I'm used to using make-kpkg to make a Debian kernel package, but it seems that if you just visit kernel.org and download the latest version you can build a RPM without any extra effort thanks to native support. Cool.

Now I need to work out how to create, host, and update a YUM repository. That looks fiddly and annoying too. XML. Eww. Any guides are most welcome - ultimately I need to package and host a "recent" kernel for Centos 4.x, Centos 5.x and Fedora Core 6-9 - each for i386 + amd64.

ObQuote: Spiderman

There is something evil there

Mon, 23 Jun 2008 18:09:05 GMT

So I've had a hectic few days, and I'm getting close to having caught up with the things that I've been sitting on whilst I've been away.

ObRandom: Several people, independantly, have told me within the past few days that "whilst" is not a real word. it is. End of ..

Some interesting things I've been working upon recently include a fun little firewall tool. Once upon a time I wrote a firewall script which worked like this:

firewall/
`-- incoming.d
    |-- smtp
    |-- ssh
    `-- www
`-- outgoing.d
    |-- ssh
    |-- smtp
    |-- dns
    `-- icmp

When you executed the magic firewall script it would scan the incoming.d directory, and for each file it found lookup the relevant port in /etc/services. These port numbers would then be opened. And at the end you'd just have a "-j DROP".

After a long phone conversation to a colleague on Thursday/Friday of last week I've now reworked this idea anew. There is still the notion of filenames referring to what is allowed for a pair of directories (incoming.d/ + outgoing.d/) but even more flexability and no hardwired use of /etc/servvices.

I guess some ideas are just too simple to give up ..?

Anyway there are a plethora of different firewall applications of varying sophistication and complexity in the world. I don't really want to go out of my way to promote this one - but at the same time it might be a useful idea for somebody?

The next (work) job I have is determining how to make a "kernel" + "kernel-dev" RPM package based on Debian sources. Joy. Actually the more I look around the more fiddly, annoying, and troublesome I suspect this is going to be. Sigh.

ObQuote: The Grudgy

Manni - you're not dead yet.

Wed, 18 Jun 2008 21:08:16 GMT

Well I'm back and ready to do some fun work.

In the meantime it seems that at least one of my crash-fixes, from the prior public bugfixing, has been uploaded:

yelp: segfault while loading info documentation

I'm still a little bit frustrated that some of the other patches I made (to different packages) were ignored, but I guess I shouldn't be too worried. They'll get fixed sooner or later regardless of whether it was "my" fix.

In other news I've been stalling a little on the Debian Administration website.

There are a couple of reasonable articles in the submissions queue - but nothing really special. I can't help thinking that the next article being a nice round number of 600 deserves something good/special/unique? hard to quantify, but definitely something I'm thinking. I guess I leave it while the weekend and if nothing presents itself I'll just go dequeue the pending pieces.

In other news I've managed to migrate the mail scanning service into a nicely split architecture - with minimal downtime.

I'm pleased that:

The architecture was changed massively from a single-machine orientated service to a trivially scalable one - and that this was essentially seamless.
My test cases really worked.
I've switched from being "toy" to being "small".
I've even pulled in a couple of new users.

Probably more changes to come once I've had a rest (but I guess I write about that elsewhere; because otherwise people get bored!).

The most obvious change to consider is to allow almost "instant-activation". I dislike having to manually approve and setup new domains, even if it does boil down to clicking a button on a webpage - so I'm thinking I should have a system in place such that you can sign up, add your domain, and be good to go without manual involvement. (Once DNS has propogated, obviously!)

Anyway enough writing. Ice-cream calls, and then I must see if more bugs have been reported against my packages...

ObQuote: Run Lola Run.

Alcohol's illegal this month

Sat, 31 May 2008 13:14:38 GMT

Busy times, despite being on holiday.

Mostly this has been doing "business" work, and fiddling with self-promotion. But despite this I managed to find time to write some extremely useful new Lisp:

Sensible (new) window placement for GNU Emacs

Anyway very little time over the coming week will be spent online. All being well. Still enjoying playing with my (loaned) Nokia 770 - maybe I'll get another one of my own eventually.

ObQuote: 30 Days Of Night

You get the dwarf. I get the girl.

Wed, 21 May 2008 23:13:59 GMT

Recently I have mostly been "behind". I've caught up a little on what I wanted to do though over the past couple of days, so I won't feel too bad.

I've:

made a new release of the chronicle blog compiler, after recieving more great feedback from MJ Ray.

un-stalled the Planet Debian.

updated the weblogs hosted by Debian Administration, after help and suggestions from Daniel Kahn Gillmor.

stripped, cleaned, and tested a new steam engine. Nearly dying in the process.

discovered a beautiful XSS attack against a popular social networking site, then exploited that en masse to collect hundreds of username/password pairs - all because the site admins said "Prove it" when I reported the hole. Decisions decisions .. what to do with the list...

released a couple of woefully late DSAs.

started learning British Sign Language.

Anyway I've been bad and not writing much recently on the Debian Administration site, partly because I'm just sick of the trolling comments that have been building up, and partly due to general lack of time. I know I should ignore them, and I guess by mentioning them here I've kinda already lost, but I find it hard to care when random folk are being snipy.

Still I've remembed that some people are just great to hear from. I know if I see mail from XX they will offer an incisive, valid, criticism or a fully tested and working patch. Sometimes both at the same time.

In conclusion I need my pending holiday in the worst way; and I must find time to write another letter...

ObQuote: Dungeons & Dragons

I still don't know why I'm here

Thu, 15 May 2008 03:19:06 GMT

I wasn't going to comment on the recent openssl security update, because too many people have already done so.

Personally I thought that Aigars Mahinovs made the best writeup I've seen so far.

However I would like to say that having 20+ people all mailing security[at]debian.org to say the webpage we referenced in the security advisory is currently blank is not useful, or ask for details already released in the advisory they replied to, or ask for even more details is not so much fun.

Having people immediately start mailing questions like "Huh? What can I do" is only natural, but you can't expect a response when things are as hectic as they have been recently. Ideally people would sit on their hands and bite their tongues. Realistically that isn't going to happen, and realistically this post will make no difference either...

Had the issue not leaked to unstable so quickly (and inappropriately IMHO) then we'd have had a little more time. But once an issue is reported you need to coordinate with other distributions, and etc. Handling something as severe as this is not fun, and random mails from users are a distraction, and a resource-hog.

I should say I was not in any way involved in the discovery, the reporting, the preparation of the fix(es), or the releasing of the update. I knew it was coming, but everybody else seemed to have it well in hand. When there are mails going back and forth for 5+ days with ever-growing Cc: lists, and mailing lists being involved I figure one more cook wouldn't be useful.

So in conclusion:

a. Bad hole.

b. Fixing this will take years, probably.

c. 50+ mails to the security team within an hour of the advisory going public complaining of missing information is not helpful, not useful, and quite irritating. (Albeit understandable).

d. People who don't know the details of an attack, or issue, shouldn't speculate and start panic, fear, and confusion. Esp. when details are a little vague.

e. I still like pies.

Once again thanks to everybody who was involved and put in an insane amount of work. Yes this is only the start - our users have to suffer the pain of regenerating everything - but we did good.

Really. Debian did good.

It might not look like it right now, but it could have been so much worse, and Debian did do good.

ObQuote: X-Men: The Last Stand

Yea, just look at all the passion on that wall.

Sun, 11 May 2008 02:14:40 GMT

There should be a website to coordinate cinema-dates.

I don't like going to the cinema alone and have, in the past, frequently missed viewing films rather than go alone.

This is a habit I'm growing out of, but I still think it is better to go with a friend or two.

In the near future I'm going to view the last Indianna Jones movie, and the Sex & The City film. I have partners for both of those.

But after that? There are a few films which I can't immediately think of who I'm going to lure away with me. I could either :

Go alone, regardless.
Randomly ask people to come

If there were a site that had list of upcoming films, and allowed you to express interest in going to see them that would be a fantastic idea. (Obviously location based).

I'd not even assume "dating", because I think in my life I've had a first-date at a cinema once. When I was about 14. Because it just doesn't work - you can't talk during, (and back then we couldn't go to the pub afterward to discuss the film. I think we did anyway ;)

For bonus points you could allow people to rate the films, or even each other. Hmm.

Somebody write it for me? I've got too much on my plate ..probably

ObQuote: Se7en

You're not too technical, just ugly, gross ugly

Wed, 7 May 2008 12:11:00 GMT

Well a brief post about what I've been up to over the past few days.

An alioth project was created for the maintainance of the bash-completion package. I spent about 40 minutes yesterday committing fixes to some of the low-lying fruit.

I suspect I'll do a little more of that, and then back off. I only started looking at the package because there was a request-for-help bug filed against it. It works well enough for me with some small local additions

The big decision for the bash-completion project is how to go forwards from the current situation where the project is basically a large monolithic script. Ideally the openssh-client package should contain the completion for ssh, scp, etc..

Making that transition will be hard. But interesting.

In other news I submitted a couple of "make-work" patches to the QPSMTPD SMTP proxy - just tidying up a minor cosmetic issues. I'm starting to get to the point where I understand the internals pretty well now, which is a good thing!

I love working on QPSMTPD. It rocks. It is basically the core of my antispam service and a real delight to code for. I cannot overemphasise that enough - some projects are just so obviously coded properly. Hard to replicate, easy to recognise...

I've been working on my own pre-connection system which is a little more specialied; making use of the Class::Pluggable library - packaged for Debian by Sarah.

(The world -> Pre-Connection/Load-Balancing Proxy -> QPSMTPD -> Exim4. No fragility there then ;)

Finally I made a tweak to the Debian Planet configuration. If you have Javascript disabled you'll no longer see the "Show Author"/"Hide Author" links. This is great for people who use Lynx, Links, or other minimal browsers.

TODO:

I'm still waiting for the creation of the javascript project to be setup so that I can work on importing my jQuery package.

I still need to sit down and work through the Apache2 bugs I identified as being simple to fix. I've got it building from SVN now though; so progress is being made!

Finally this weekend I need to sit down and find the time to answer Steve's "Team Questionnaire". Leave it any longer and it'll never get answered. Sigh.

ObQuote: Shooting Fish

Only after disaster can we be resurrected

Tue, 6 May 2008 19:13:22 GMT

I leave my main desktop logged in for months a time; as demonstrated by my previous bug with the keyboard transition for xorg.

The screen is setup to lock after 5 minutes of idle, so there's no real security issue, and it is extremely convenient.

Every few weeks though my desktop gets into a funny state where no new windows may be opened.. Existing applications continue running without any problems, but no new windows/shells/whatever may be opened.

Tonight it happened again.

And the lightbulb went on in my head: My flat uses CFEngine to manage itself. (Two physical servers here, with 5-10 Xen guests, and a number of remote servers.)

One of the things that CFengine is configued to do is to tidy directories of files which are older than 30 days. Including /tmp.

So that explains that.

Every month the magic cookie in $TMP would be nuked, and X would disallow new connections.

I guess the next time this happens I should look at using Xauth to fix the issue, but generally I just logout, make coffee, smoke a cigarette, and login again.

In conclusion: I'm a stupid-head.

ObQuote: Fight Club