Advogato blog for pap

5 Apr 2001

Thu, 5 Apr 2001 13:09:02 GMT

Did some work on Sotonians last night. Finally dusted down the old schema and began work on a lower-level API for it. Basically, for each table I need the classic insert, modify, get and delete behaviour modelled. The biggest dilemma, however, is where to perform the error checking.

If I wanted, I could just pass stuff to the DB verbatim, and get it to report errors to me if values don't fit into the attributes they are assigned to. That can be costly, though. I don't really want the overhead of creating a DB connection if the end result is going to be fruitless.

Which leads us into option 2 - which is to provide program logic which will use the Perl regexp checking to make sure values fit before placing them inside. Makes for a much bigger API, but you can be reasonably sure that DB connections are worthwhile.

I did actually knock up a seventy-line Perl script to analyze my DB schema and provide template methods for insertion, modification, deletion and retrieval. This had type checking built-in, so if something was NOT NULL, it'd produce a check to make sure that particular attribute has been supplied as a parameter. It also spawned little regexp and length checks to ensure that INTEGERs were correct, and that VARCHARs had a sensible number of characters in them.

The problem with the generated code stems from the schema. The DB is heavily normalised, which results in the same attribute being present in a lot of other tables. In each table method, the same code was being used to check the same attribute again and again. The generated API weighed in at something like 1200 lines, which is a maintenance bitch. So, whilst my DB API generator was a nice idea ( and a hairy coding experience - nice lot of escape sequences ) I'm now thinking about taking an alternative approach.

I plan to wrap all the attribute checks into a single method, which can be called from the insertion, modify, retrieval and delete functions. I'm also thinking of parameterising those functions to enable me to have just one routine for insertion, one for modification, etc.

Still, a damn useful prototyping session, and something that ought to bear fruit regardless of it's apparent failure.

22 Mar 2001

Thu, 22 Mar 2001 13:22:17 GMT

A while since my last entry. I've been inundated with work of all kinds since my last post - most of it a total waste of time under better development circumstances.

Been writing this file moving system that collects files from an arbitrary number of locations and delivers them to an arbitrary number of locations. The collection connection is quite limited in terms of bandwidth, so the sensible thing to do is to maintain a database of what you've downloaded and what you haven't - to ensure that files aren't needlessly transferred.

A job for something like Postgres, you might assume. However, due to some crap company policy the only DB I could get my hands on would be ORACLE, which is clearly overkill. I haven't got root anywhere on our dev or live boxes, so I can't really install stuff myself - even if I could IT Services would put a contract out on my life.

So, the bulk of my experience over the last few months has been writing a Perl-hash-based DB in order to meet the system's basic requirements. A lot of fun.

Got the system into a nicely working state, but I have to ask why big corporates think that if something is free, it is unworthy of consideration. It's a miracle that we even have Perl here. And haven't they heard of third-party support contracts? Re-inventing the wheel is not my idea of fun. The sooner more companies embrace open-source software, the better.

20 Dec 2000

Wed, 20 Dec 2000 14:57:45 GMT

Sotonians Actually done some open source work since my last diary entry. Been working on the database structure for Sotonians - a locally-themed site for the city of Southampton, England.

I know what you're thinking - it's another of those sad sites which is replete with local (flower shops|restaurants|brothels) advertising their (chrysanthenums|reconstituted dog|cheap Thai imports) in wholly unappropriate colours. Nothing could be further from the truth.

The main idea of the site is to give a voice to the disenfranchised denizens of our fair city. The site is frequently rude, covers all areas of conversation from monkey-spanking to primate-punishing - and is something we're looking to scale up.

The current implementation is somewhat dodgy. Most of the stuff is statically delivered, making maintenance a bit of a bitch. TheCorruptor and I are fed up with this headache, so we're doing something about it.

Essentially, we're moving almost all of the content into a PostgreSQL database. The DB design is now complete - I've just got to nip it on over. The API for DB updates will be coming next. Incidentally, we chose postgres because I have a personal thing for clean databases. It has stuff like referential integrity and transactions (not sure we'll need transactions for a lot of our stuff). postgres will help us keep the DB clean without excessive logic, hence its adoption as the DB of choice.

TheCorruptor is going to use the APIs to yank stuff outta the database, transform it using XSLT and deliver it to end users.

The DB design and API are actually very generic, which will allow it to be used for a wide range of dynamic content websites.

The new version of Sotonians will launch in early 2001.

27 Nov 2000

Mon, 27 Nov 2000 13:36:52 GMT

Been awhile since my last diary entry. Haven't had a great deal of time to be involved with stuff, due to the slings and arrows of outrageous relationship fortune. Need a kick up the arse at the moment to get my developer head back on.

Projects are slowly starting to kick in. The main thing I'm involved with at the moment is the complete re-write of our fair web-site, Sotonians. I'm working on this with TheCorru ptor - he's handling client-side delivery while I'm doing all of the backend stuff I know and (love|hate).

Been a reasonably interesting couple of months work-wise. I've become the custodian of a Perl-based GPS project, which isn't open source, but is interesting all the same. I've done the analysis on the entire codebase and sussed out where and why things are going wrong. Did some real SE for a change and weighed the project in at about 100 days. Only problem is, I am the team. Not having a long- haired associate to (rubbish|validate) my ideas is really starting to bake my brains.

We've procured a dedicated server for Sotonians and a number of other sites that we run, including southampton.pm.org - so I've been busy working on that. Unfortunately, access at my place of work is a bit crap - but there are always HTTP tunnels for that sort of thing.

TheCorru ptor and I are going into business together very shortly. I'm already contracting - he will be too, and we're looking to develop even more projects for the good of the business. Not exactly Marxism - but hey - we all have to eat and pay for the nights out. It should be pretty neat - we're quite up for it now, and figure that our vast collection of bullshit coupled with our sporadic sampling of talent should enable us to at least put Linux dev boxes on the table for awhile.

So, to summarise then. A lot of projects, a lot of business and life as we know it isn't going to be the same in a months time.

Situation normal, then.

21 Jul 2000

Fri, 21 Jul 2000 11:01:29 GMT

Another first post from a newbie muppet!

Where are we at? Been looking at possible projects to get involved with but am frustrated when I find that someone has already done the very thing I thought would be useful.

I was recently mortified to see someone charging a $10 registration fee for a Windows-based file-splitter, so my first project will be a simple Windows-app to do precisely that, but for nowt, and make it open-source. If you know of someone who has already done this, let me know.

The app is mostly intended to help the diminishing population of Amiga owners transfer large files from their PCs to Amiga on several 720K disks.