RhysJones is currently certified at Apprentice level.

Name: Rhys Jones
Member since: 2002-08-15 19:48:28
Last Login: N/A

FOAF RDF Share This

Homepage: http://www.wiblog.com/backburner/

Notes:

A Welsh-speaker living in Swansea, who's very happy there. Has now rediscovered the passphrase to his proper Advogato page, and is even happier.

Recent blog entries by RhysJones

Syndication: RSS 2.0
24 Aug 2002 (updated 24 Aug 2002 at 16:30 UTC) »
He who obeys, obeys Bayes?

Yup, another set of possibly ill-conceived thoughts on spam filtering. There was a good round-up of what's currently going on in yesterday's NTK, incidentally. (Search for Tracking in that page).

What the current crop of Bayesian spam filters are trying to do is detect whether an email is written in 'normal English' or 'spam English'. That language identification requirement is exactly what this this text-based language identifier is trying to work out.

So, can anyone tell me why an n-gram (where 2 <= n <= about 4-5) based system, working on the word level, hasn't been tried on this spam/non-spam classification problem? After all, n-gram statistics are used by virtually every single speech recognition software package out there. That's because n-grams are what let the speech recogniser determine which word has a reasonable chance of following the word it thinks you just said, and this information is used to improve the likelihood of what you said being a 'valid' sentence.

There are problems with this, of course: recognising 'spam English' as opposed to 'proper English' is much more difficult than recognising English as opposed to, say, French. But there are subtleties of vocabulary ('free' can often be followed by 'porn', for instance), that may make this a workable method.

Well, those are my thoughts anyway. Someone's probably tried this already and will tell me why it doesn't work as well as other methods, but even in that case I really would be interested to know why.

[edited for formatting]

23 Aug 2002 (updated 23 Aug 2002 at 09:07 UTC) »
thomasvs wrote:

...when I look at scripts I wrote five years ago, or code for my thesis, I'm ashamed ;) So the big question is : What will I think five years from now about stuff I did today ?

I could've written that, but, being me, I didn't. It's an incredibly sobering thought though. The mantra of 'Do stuff. Repeat.', and the goal of continuous improvement, is still as pertinent now as it ever was. Maybe programmers, localisers and other free software advocates really are made, and not born.

Festival is great. Not only it is GPLed ('course), but it's also truly multi-lingual, thanks to a Welsh speech researcher (Briony Williams). Unfortunately, the letter-to-sound rules for Welsh are now written in Lisp. Goal, then, is to attempt to gain a passing acquaintance with Lisp. Never thought I'd see myself typing that. Mind, never thought I'd have to use Fortran either; that's another story.
15 Aug 2002 (updated 15 Aug 2002 at 19:55 UTC) »

Oops. Created an account as Rhys a while back, but completely forgot the password. Decided that a new account was the better option (sorry to those who've already certified me once...)

This week, I have been mostly looking for Welsh-speaking open source advocates. In my view, and that of a few others, GNU/Linux Must Be Translated. Mail me if you fit that bill. Diolch yn dalpau.

 

RhysJones certified others as follows:

  • RhysJones certified chipx86 as Journeyer
  • RhysJones certified gregorsamsa as Master
  • RhysJones certified Telsa as Master
  • RhysJones certified Bryce as Master
  • RhysJones certified alan as Master
  • RhysJones certified dick as Master
  • RhysJones certified helcio as Journeyer
  • RhysJones certified andrewb as Journeyer
  • RhysJones certified salmoni as Journeyer
  • RhysJones certified dgatwood as Journeyer

Others have certified RhysJones as follows:

  • ade certified RhysJones as Apprentice
  • Penix certified RhysJones as Apprentice
  • mascot certified RhysJones as Apprentice
  • dltxprt certified RhysJones as Apprentice

[ Certification disabled because you're not logged in. ]

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!

X
Share this page