Older blog entries for apenwarr (starting at number 210)

2006-12-01: Novel, yes; widespread, no; Old news, new acronyms: REST vs. SOAP

Novel, yes; widespread, no

That's right, kids, November is over and while my novel isn't "done," it did indeed pass 50000 words. I have a tiny bit of work left to completely wrap up the story, and a little bit more tweaking before it's up to my usual questionable quality standards (as opposed to its current "unquestionable" quality). That should only take a few more days.

And after that, you, yes, you, the one and only, will still not have access to read it, because I'm not currently planning to post it publicly anywhere, lest Google reveal to all the world that I'm an idiot. But I'd probably send you a private copy if you asked.

Old news, new acronyms: REST vs. SOAP

I've been seeing news in random places around the web lately announcing the official death of Web Services and SOAP in favour of REST, or was it XML Schemas in favour of RelaxNG, or whatever bad things the people in question don't like in favour of whatever good things the people in question do like. You can find many of these links by following links from Tim Bray's Ongoing.

And then comes the inevitable reply, They Can't Hear You, explaining that these so-called "deaths" are rather overstated, because all of the supposedly "bad" technologies are still very much in use in big companies everywhere, and moreover, most people in those companies haven't even heard of the so-called "good" technologies.

Now, of course I'm always totally in favour of a good flamewar, but if the flamewar is good enough, then I prefer to confuse the issue rather than resolving it. Thus, I bring you my considered opinion:

Yes, it seems those bad things really are dead. But they're not. Except where they are.

You see, declaring SOAP dead in favour of REST is equivalent to declaring Windows dead in favour of Unix. No, no, not "just as stupid." I mean equivalent. If one was true, the other would probably also be true. And you may have noticed that Unix has totally prevailed. But it hasn't. Except in the places where it has.

A while ago at NITI I did a presentation about functionality vs. elegance. Basically, I argued that people often treat them as a single continuum, sacrificing features for beauty or vice versa according to their priorities. But really they're two independent dimensions, and a small few things are functional and elegant.

In my example at the time, I showed how Windows isn't elegant (well duh) but it is highly functional; you can make it do anything. Unix, on the other hand, was an amazing achievement in its heyday because it was highly functional and highly elegant. Because this was originally true, people experience the beauty of Unix and then tend to believe that "the Unix way" is the best way to solve all their problems.

Unfortunately, original Unix was designed for processing text on a command line. Its elegance came from clever, well-executed simplifications like the unified filesystem, devices-as-files, pipelines, and everything-as-text. Unfortunately, none of those things have anything to do with graphical user interfaces. X11 is anything but elegant, and none of the various Unix widget toolkits are earth-shatteringly great like Unix was. Furthermore, trying to map elegant command-line stuff into GUIs has been a consistent, horrible failure; compare a Windows-based IDE debugger to any GUI wrapper on top of gdb, and just try to tell me I'm wrong.

To make something both highly functional and highly elegant, you need what we call a simplifying assumption. In the command-line world, the credit typically goes to the invention of Unix pipes, which let you easily link small tools together to accomplish a big job. Windows has no such magic (well it does, but they forgot to support it in the shell), and so it predictably sucks at text processing. But Unix has no simplifying assumption in its GUI. So they have to trade off between functionality and elegance. In typical Unix fashion, this means there are a zillion half-baked alternatives, each at different points along the tradeoff continuum.

But there are places where Unix's traditional elegance is still a win, and that happens to be the world of Internet servers. The web works because it mostly just paraphrases Unix's cleverness. It has a unified filesystem (er, URI space), devices-as-files (er, CGI scripts generating content), and everything-as-text (html). Sadly, its UI mostly sucks, except for the AJAX stuff where we basically hack the hell out of Javascript and CSS to mangle it into a decent UI, at the cost of almost all the operational elegance. That's because what's missing from the web model (and the Unix GUI, for that matter) is an analogue for Unix's pipelines - I can't easily connect one little thing to another little thing to generate a high-quality big thing.

The fancy modern name for that is a component system. Unix pipelines are one kind of component system, although people who talk about Component Systems as if they had Capital Letters would probably gasp at my saying so. That's because those people design the second kind of component system: things like COM. You know: the way Windows does it.

So here's the thing. In the world of the web, there are two competing ideas for component systems: REST and SOAP. And these correspond almost exactly to Unix and Windows, or in other words, pipelines and COM. Pipelines contain squishy (nowadays Web 2.0 people would say "mashable") text in which the format is flexible and implicit and probably being parsed only 99% correctly. That sounds to me exactly like REST and microformats and RSS and friends.

Meanwhile, there's SOAP and SOA and WS-Whatever and XML Schemas. They're just like COM: useless unless your syntax and semantics are exactly right, semi-easily checkable for valid syntax, and purportedly self-describing in an essentially useless way (except for syntax checking purposes). Self-describing or not, Google will never be able to index your SOAP service without a special plugin, just like Google Desktop can't index any of your binary document formats without special COM plugins, one per format.

Phew. Okay, so that was the background information. Here's my point: yes, the whole Internet runs on Unix philosophy. But businesses sure don't. The big problem comes up in my description of pipelines up above: they only parse about 99% correctly, which is fine for your idiotic comments about YouTube videos, but pretty nasty when you mangle critical business data. And when the business dudes get involved, they'd rather do anything than mangle their critical business data. You hear me? Anything!

If it keeps the data from getting mangled, they'd happily sacrifice searchability. Or developer hours. Or the ability to use off-the-shelf software. Or millions of dollars in licensing fees (because at least they'll accurately know how many millions).

Pay attention, because that attitude is exactly why Windows is so strong, why the vast majority of developers prefer to develop on Windows, and why the vast majority of users prefer to use Windows. It may be gross, and adding new features or libraries may be a lot like stabbing yourself repeatedly with a fork (which, tragically, is not itself included in Win32), but Windows works consistently. So does SOAP. But not Unix or microformats or REST.

It would be awesome if someone could find a way to satisfy both camps (the Internet people and the Enterprise people) at once. Then maybe one or the other set of technologies could finally die. But I'm not counting on it. Until then, one or the other technology is effectively dead, but which one it is depends who you are.

Syndicated 2006-12-02 03:55:27 from apenwarr's log

2006-11-24: I made the Fido lady cry

I made the Fido lady cry

This evening I got a "customer satisfaction survey" call from Fido. She assured me that the call wasn't going to use any of my cell phone minutes, and then asked a simple leading question:

"On a scale of 1 to 10, how would you rank your current satisfaction with Fido?"

"Current? As in, not historically?"

"Right. How do you feel about the service you're getting right now?"

"Well, about a 6."

"Err... well, that's pretty low. Can I ask you why?"

So I got it all off my chest. The ever-increasing fees (increasing? What kind of technology is this?). The annoying people who phone me when I'm a few days late on a payment because I've been travelling and haven't picked up my bill. (I mean, charge me a late fee if you must, but *phoning* me for being three days behind? Yeesh.) The lately incessant calls to sign onto a multi-year contract, which shows they no longer care about "real" customer loyalty (I've been a customer for about 10 years, no contract required). The way their ads aren't so innocent anymore: they used to be just cute dogs while everybody else would use salamanders (what?) or cute girls; now they're cute dogs with cute girls, and it's only a matter of time until the dogs won't even get equal billing. And of course, my actual problem, which is that I'm out of town so much lately that I pay long distance charges for almost every incoming call minute.

In short, I used to recommend Fido to all my friends because of its awesomeness compared to other cell phone companies. But since Rogers bought them, they've been in steady decline. The lady on the phone rightly pointed out that they're still "better and less expensive than all the other companies!" Well, fine. It's not like I'm saying I'm planning to switch because I found a better deal elsewhere, because I didn't. I just don't care about you anymore. So if I did find a better deal elsewhere, then I suppose I'd probably switch.

And this is where the whole conversation got kind of weird, because she was totally horrified by this attitude. She asked if she could put me on hold to check with everyone else to see if there's a plan I could be on to fix my long distance problems (there wasn't). She checked to see if signing onto a multi-year contract would help me somehow (it wouldn't). She checked all over to see if I could adjust anything to save money given my admittedly weird calling and travel patterns (there wasn't). She discovered that I was currently entitled to a free new handset (I don't need one).

But here's the thing. She kept me on the phone for 20+ minutes as she went through everything she could think of to try to solve my problems, even after I assured her I wasn't planning to switch or cancel my service. That can't have been part of her job description. She was taking her "customer satisfaction" job really personally, to the point where she thought of it as a personal failure if I wasn't happy with my Fido service.

Which is silly, of course, because it's not her fault Fido's recent policy changes suck. It's Rogers' fault, and perhaps the fault of the almighty dollar (rumour has it that Fido was losing money at the time they were bought; I don't know if that's true).

It seems to me I've talked to this particular lady before when I've called Fido support. I think she might have been working for Fido almost as long as I've been a customer. That means that she's been through the days of rave reviews, successful word of mouth marketing, and sensible management. The management was indeed sensible to hire all those great, loyal, dedicated people in customer service. Strangely, those loyal people are still there, and still really care just as much as they used to. They just don't have the power to make the customers happy anymore.

I didn't really make her cry. Only almost. But I think she will if I ever cancel.

Syndicated 2006-11-25 02:40:07 from apenwarr's log

2006-11-13: Novelling vs. My Sanity

Novelling vs. My Sanity

"I need to attract more customers so I can get promoted or at least demoted so I don't have to wear this squid anymore. It's a long story."

Final score: Novelling:1, My sanity:0

Syndicated 2006-11-14 06:15:18 from apenwarr's log

2006-11-06: The Brave New World of Schedulation; The Google Factor

The Brave New World of Schedulation

Schedulator is the software project management system we originally developed at NITI to solve a simple problem: I'm a programmer, and I ended up as a manager, and I didn't want to spend all my time doing management.

It's essentially a fancy automated implementation of Joel Spolsky's Painless Software Schedules, where each developer can manage his/her own schedule. By combining the results generated that way, you can produce an accurate schedule for the whole team.

Schedulator isn't very many lines of code, but it's the result of years of tweaking of the Schedulation process, which actually works pretty well, and has a lot of advantages over "normal" software scheduling. As part of my new job, I've been redoing major portions of the Schedulator to make it more flexible: it now supports bug tracking systems other than FogBugz, for example (although that one's still my favourite :)), and no longer requires GracefulTavi (as much as I like wikis, doing it that way led to a pretty inflexible design).

The new Schedulator isn't packaged up nicely, and may never be because it has a pretty limited audience. But it should be much easier to get up and running with the new version than the old one, because it has far fewer prerequisites (and the parts that have prerequisites are now optional plug-ins).

I did, however, write some extensive documents explaining the theory and practice of Schedulation. My favourite is the first one, Schedulator Philosophy. Read it, and maybe find out if Schedulator is right for you.

The Google Factor

Not long ago, pcolijn wrote about Google's development process and compared it with Schedulator. I generally agree with his analysis. The problem is that most of the problems Peter identified were not problems with Schedulator, per se: they were problems with the general team atmosphere and management. (I'm allowed to say that, because I was one of the managers.) Schedulator is very compatible with the "Agile development style" done at places like Google. Okay, so Schedulator doesn't come with any index cards included, but it does do a lot of the grunt work of arranging those index cards - particularly when there are hundreds of them in your bug tracking system and they all seem important - for you.

As much as Google is indeed awesome, hires a lot of great people, produces a lot of great things, and makes an awful lot of money in the process, I sometimes caution people against taking their experiences at Google too seriously. People in highly successful companies are prone to an attribution bias, much like people are in unsuccessful companies. That is to say, no matter what your team does at Google, Google is still going to be making billions of dollars. And no matter what your team does at some unsuccessful or dying company, you probably can't pull it out of the toilet. In both places, the consequences of your actions are disconnected from reality. (Bill Gates called this the honeymoon phase.) At Google, raises and bonuses for everyone. At the unsuccessful company, even great people get laid off sometimes because there's no money to keep paying them.

For example, the article that Peter was responding to had this very important paragraph:

    Another incentive is that every quarter, without fail, they have a long all-hands in which they show every single project that launched to everyone, and put up the names and faces of the teams (always small) who launched each one, and everyone applauds. Gives me a tingle just to think about it. Google takes launching very seriously, and I think that being recognized for launching something cool might be the strongest incentive across the company. At least it feels that way to me.

Sure, this is a great incentive for people to work harder, and Google's very wise to do it. But do the math: every single project that launched? Always small teams? But Google employs thousands and thousands of programmers! They can't all be on the big screen. In fact, most of them can't. But Google's still successful.

Which kind of team are you on? How can you tell? Statistically speaking, you're probably on the unsuccessful kind. And beware of logic that sounds like, "my team did this, and we weren't too late" or "their company did it this way, and went three years overtime." There's a correlation there, but there could easily be no causality. Super-ultra-smart people could do their project plan on stone tablets and still get their projects done an order of magnitude faster than complete boneheads.

That said, I used (and still use) Schedulator, and it works great for me :)

Syndicated 2006-11-06 22:05:56 from apenwarr's log

2006-10-30: Programmer Replacement

Programmer Replacement

As a programmer, there are two main ways to make yourself irreplaceable. One is to write code that nobody else can possibly understand, so they can't figure out how to get rid of you. One is to write code so great that nobody could ever want to get rid of you.

The best way to be replaceable is to be somewhere in between.

Syndicated 2006-10-30 15:38:12 from apenwarr's log

2006-10-27: Six Words

Six Words

apm linked to a series of "six word short stories" by famous people in Wired Magazine. Many are quite excellent.

I'm not famous, but this sounded like fun, so here I go:

    Who's that bioengineered lapdog eating *now*?

(This is a work of pure fiction. Any resemblance to real CorporateDogs is purely coincidental.)

Woo hoo! Only 49,994 words left!

Syndicated 2006-10-27 13:20:50 from apenwarr's log

2006-10-24: A day in the life of power management

A day in the life of power management

Hmm, it seems the expected flamewar is flaming even better than expected. hub chimes in to say that power management has been working just fine on Linux since 1999, thank you very much. Um, yes, well... sit back and relax, because I have a story to tell. Like all my favourite stories, it involves a lot of stupid people doing stupid things. It doesn't have a happy ending. If you're easily depressed, you should stop reading right now. That means you, Pierre.

Once upon a time there was APM, or "Advanced Power Management." As with most fancy-named specifications, its name certainly doesn't imply that there was an earlier "Totally Simple Power Management" that it replaced. The way APM worked was it used a kind of "super supervisor mode" in the x86-series CPUs to actually run some BIOS code above the operating system layer in an invisible way so the kernel couldn't detect it (except for the bad BIOS coding that would result in lost clock ticks and random interrupt latency). This BIOS code would do things like detect presses of the laptop's suspend button and forcibly suspend the system whether the operating system liked it or not. If your operating system supported APM, which was by no means necessary in order for suspend/resume to work, it could get a notification from the BIOS that it was about to suspend. In later APM specifications, they added a way for your software to reject the suspend operation in progress. However, because of BIOS bugs, this didn't really work so well. Also, if you took too long to service the notification, the BIOS would just give up on you and suspend anyway. Ah, those were the days.

Oh, also, because the BIOS writers for a laptop always knew exactly which hardware devices were in the particular laptop, the APM BIOS could always know exactly how to suspend and resume all the devices in your laptop transparently to the kernel. And yes, it actually worked just fine. Even the video mode was saved and restored correctly on most systems, all behind the scenes.

But that was a long time ago. To give you an idea of how obsolete APM is, my apmd page seems to be the fourth hit in a Google search for APM - and it's the first one about power management. And let me tell you, that's not because I'm such a popular guy.

There were two perceived problems with APM. First of all, it was highly x86-specific, which annoyed the people at Intel who were trying to make and sell a non-Intel-compatible processor. (Remember that failed experiment? Me neither.) Secondly, operating systems programmers correctly noted that all BIOS programmers are crackheaded morons who can't implement an API correctly to save their lives. The way things tended to work was this: Windows didn't come with native APM support at the time, and the BIOS programmers would screw up the APM implementation horribly, but that was okay! Because every motherboard had to include a special APM driver for Windows anyway, and this APM driver would just implement the broken APM calls that the BIOS required, and so nobody would know the difference. Sure enough, nobody (er, well, no Windows users) did, and the world of power management was a fine place. For Windows users.

Linux users had a bit of trouble because they had to independently discover all the stupid BIOS bugs in various laptop APM implementations. But they mostly sorted this out eventually. In any case, one popular way to make your problems go away and have suspend/resume work mostly right was to simply disable the Linux APM altogether and just have your BIOS do it transparently.

So anyway, back to those people who wanted to fix what wasn't particularly broken. They invented ACPI, and I want to kill them. Oops, I'm getting ahead of myself.

ACPI stands for Advanced Control and Power Interface. Now, I have two things to say about that. First of all, there was no "Delightfully Simple and Straightforward Control and Power Interface," although ACPI actually makes APM seem that way in retrospect. And secondly, ACPI has nothing at all to do with APIC, the Advanced Programmable Interrupt Controller. The only comparable thing between the two is that they both have Linux kernel boot-time options to disable them because they both have buggy Linux drivers that cause your computer to crash a lot.

Now where was I? Oh, right, ACPI. So, the idea of ACPI was to get the BIOS developers out of the way on a normally-running system by turning around the power interface: instead of the BIOS running things and just occasionally notifying the kernel when something happened, the kernel would run things and just ask the BIOS to do stuff occasionally, like power down various hardware and blink the lights and so on. That would mean BIOS bugs wouldn't be so harmful. Oh! And while we're here, because we're insane, why not implement the whole thing using Forth-like bytecode instead of real assembly language, so it can also run on that new (doomed) 64-bit processor we've been working on? Forth-like bytecode is super simple and can be implemented in a couple of kbytes, so it won't cause much overhead, and suddenly everything will be portable. It'll be great!

Because I like foreshadowing, I'll give you the quick version of what I'm about to say. To my total amazement, they managed to fail totally on all counts. How's that for consistency?

First of all, ACPI completely and utterly fails to remove the BIOS from the picture: in fact, you're calling back and forth to it far more than you ever did with APM. And because you're calling it from your context instead of it having its well-known super-supervisor context, it's more likely to get confused by the funny way you do your stack or registers or memory protection modes. And there are so many ways to call into ACPI, because they broke it into tiny pieces so your kernel can control exactly what it wants to when it wants to... except that the BIOS manufacturers didn't actually test what happens when you call their stuff in random order, so actually you have to reverse engineer exactly what order is safe to call things in, or your system crashes horribly.

Oh, also, the bytecode thing went awry somewhere, because the Linux ACPI implementation runs to hundreds of kbytes and is filled with all kinds of weird and very complicated special case drivers for obviously totally dissimilar APIs like fans (it goes fast! it goes slow!) and CPUs (it goes fast! it goes slow!) and LCD backlights (it gets bright! it gets dark!). And naturally, ACPI, being a big horrible pile of crap, was never adopted on any non-x86 platforms, so its CPU independence doesn't help.

(Around the time all this garbage was being invented, people were trying to make non-x86 platforms run PCI video cards, which was tough because the video card initialization code was written in x86 assembly. The XFree86 group and other groups solved this problem unilaterally in a less elegant-sounding but actually working way. To this day, video BIOSes are still in x86 machine code, not bytecode.)

But that's not all!

Remember, the OS developers wanted to get the BIOS developers out of the picture, because BIOS developers are indeed crackheaded morons - I think we can all agree on that. Unfortunately, while they completely failed to do this - and in fact, ACPI makes things much worse - they also made it so the BIOS developers can happily just disclaim any responsibility for whatever parts of power management they don't want. Once upon a time, in the golden age of APM, the BIOS had to support all your devices because it was the BIOS whose @#$! responsibility it was to suspend and resume everything. Now, however, the OS is expected to pick up wherever the BIOS leaves off - which is almost everywhere. That means most ACPI BIOS implementations don't actually handle any parts of the suspend/resume process properly, often including even the CPU speed. Certainly they're not smart enough to suspend your ethernet chip. And heaven help you if you want your video mode to be restored! My now-stolen last laptop, a Sony Vaio, actually had an ACPI interface to control the LCD backlight... but it didn't do anything. There was a totally different non-ACPI backlight control elsewhere in the system, and the BIOS developers simply didn't bother to take out the ACPI one leftover from a previous laptop model. The OS has to know this, based on the laptop model number, and deal with it.

But that's okay! Because the Windows driver programmers, sitting right next to the BIOS programmers or maybe the hardware designers, can simply compensate for all this stuff. The CD that comes with every laptop contains modified drivers for all the broken stuff the hardware and BIOS designers did wrong when building your system in the first place, so everything is fine! For Windows users.

Now, Linux certainly didn't have it all easy in the days of APM, but things had a pretty good chance of working because they were relatively simple. With ACPI, everything is just a total disaster. You have to implement suspend-to-disk all by yourself, and it's doomed to suck because the stupid BIOS does its time-consuming and useless initialization before you're even allowed to start. You have to implement power saving features in every single driver, where with APM you didn't have to do it in any driver. Linux developers are notoriously bad at handling exception conditions, which power management is, so the power management code for most drivers is almost-untested barely working garbage. And of course, you do still have to call into the mostly-but-unfortunately-not-totally-useless ACPI BIOS, which for some reason takes hundreds of kbytes of source code to do and requires talking to a horrendously buggy BIOS. That means you need an exception table listing every laptop anyway to tell the kernel which bugs you need to work around at which time.

And if you do all that stuff correctly, your laptop will suspend and resume properly!

And you know what? Even then, it'll still suck, because that's just what you have to do to make it barely work at all. If you want to, say, bypass the stupid BIOS POST phase to make it boot faster, or do what Apple does and actually save to disk and memory at suspend time, then resume from memory whenever possible, or have the system suspend to memory and then suspend to disk if you stay suspended for a long time, or any of that other complicated stuff: that's all extra. Meanwhile, most hardware developers are a bunch of slackers and even when you do suspend the bloody thing properly, the battery dies in a few hours anyhow.

So kudos to the Linux developers for making it almost work. I'm sure that was really hard. Yay team.

Compared to that, Apple cheated like crazy. But I still like my Mac, because it actually works.

(And I have Ubuntu running constantly in a virtual machine because I mostly hate Darwin, but that's another story.)

Syndicated 2006-10-24 01:39:02 from apenwarr's log

2006-10-23: In my new role as Mac zealot...

In my new role as Mac zealot...

Far be it from me to get into any sort of flamewar-causing religious-type discussion (ha ha), but pcolijn asked how any sensible developer could use a Mac for their work, given a few of its flaws. In order, and without any actual reference to the questions:

  • I don't watch movies while computing, because I can only do one thing at a time. I know, I'm an inferior life form. Sorry.

  • I don't play movies on my computer at all, actually, because computers are stupid. (It took me years to find the root cause of my endless video-playing problems on every operating system, but that's definitely it.)

  • I prefer to listen to my CDs rather than looking at them.
And last but certainly not least...

  • I use ion for my window management, of course! It's just as excellent as ever, plus power management works. Actually I had to upgrade to ion2 (I was probably the last person in the universe to do so) in order for it to make sense. In ion1 the results when you press F9 are hilarious (all the frames are separate windows and swoosh to random locations) but rather useless. Anyone who wants a copy of my ion2 config files, just ask.
Also, because you didn't ask, I have no patience for the actual OS X "imitation Linux" crap (any OS released post-y2k where just typing "find" doesn't list all your files is stupid, plus "ps axr" doesn't work, and fink seems to mostly consist of non-working packages) and installed Ubuntu in a virtual machine. So I'm "remote displaying" on my MacOS X Xserver from my virtual machine. It's a bit silly, but due the magic of modern virtualization and disturbingly fast CPUs, it works fine.

(How fine? Well, pphaneuf asked me to time how long it takes to compile Quadra on native OSX vs. my virtual Ubuntu, but unfortunately I'm not smart enough to compile it on either one successfully. Anyway, it's fast enough for me.)

Syndicated 2006-10-23 13:54:18 from apenwarr's log

2006-10-21: Delirium 3: Specs and Constraints

(No, I still haven't finished transcribing my handwritten notes from when I was laptopless in early September. Here's another entry. Note to self: I know it's where the term "copy and paste" came from, but yeesh, handwritten notes are a really inferior medium for actually doing it.)

Delirium 3: Specs and Constraints

So let's say you managed to assemble a team of obsessive, maniacal geniuses who are all "signed up" and ready to get the job done. Great! So, uh... what job, exactly?

There's more than one reason people don't run teams like this too often. The most basic reason is obvious enough: the rules are so strange, undocumented, counterintuitive, and completely opposite from normal (successful!) engineering practice that people just don't try to assemble the "right" conditions very often. But if, after takeoff, the results tended to work out spectacularly... well, people would get over all that other stuff, right? ... Right? ... Actually, yes, I think so.

But life isn't so simple. There's a much more serious problem with the "obsessive genius" technique. The problem is that the results are entirely out of control. You'll almost certainly get something cool - probably more cool stuff than you know what to do with. But you won't know in advance exactly what it'll be. And that, my friends, makes it hard to run a business.

Some companies do it anyway. They start a "research lab" or "skunkworks," put the wheels of obsession in motion, and pray like crazy that something good comes out.

Well, research labs aren't for me. I'm not going to beg "the establishment" for their precious resources, with nothing more to offer than a random splash from my really, really big shiny innovation firehose. When all I have to choose from is total randomness in large quantities or total determinism in disappointing quantities, I'm in a classic compromise situation. The non-compromise condition is massive, sustainable, directed ingenious output. And my theory is that it's a lot easier to direct output that's already massive, sustainable, and ingenious than it is to take small deterministic output and make it bigger and more ingenious. It's easier to learn to aim a big firehose than to put out a big fire with a really accurate squirt gun. Mind you, no matter how well you aim, some tangential stuff is still going to get a little wet. But it'll dry off eventually.

Which all finally brings us around to my point. It is possible to direct the output of a team of obsessives - sort of. You just have to be careful about what exactly you direct.

Determinists love specs - clear descriptions of the ins and outs, which must be captured exactly. Then you leave your geniuses to do any design they want, as long as it implements the spec precisely.

I tried that method. I really did. I sliced and diced specs vs. designs in every way I could think of. UI design isn't specification, it's design!, I declared once.

But with each and every attempt at this technique, I failed in one of two ways. Either the spec was too dumb and restrictive - or the result was too crazy and random. Either the product was overspecified or it was underspecified. And now I realize that it always will be. Trying to "right-size" your spec is just a tradeoff - a compromise. The right answer doesn't even lie on that axis.

But even though I've never really managed to separate the spec from the design (for management purposes at least - it works for post-documentation, or what we sometimes call a "retrospec"), I have successfully produced non-random, ingenious results - and I've worked with other teams, like NVS and GourD, that produced them. What's the secret?

The secret is "end-to-end." The problem with our spec-design separation is actually very simple: it draws a line between the real, deterministic world - the spec - and the world of genius - the design. But that's nonsense. How can your product be ingenious if its workings were defined by someone deterministic? How can it be useful if the workings are defined by someone crazy? The spec, too, must be written by someone in genius mode. Those were the real successes I've seen - where the spec, the design, and the product were all done in genius mode, "proper processes" be darned. And I've been fooled like everyone else: GourD had a Requirements doc, then a Spec, then a Design, then an implementation. Deterministic, right? The model of good engineering!

Yeah, sure. The author of all those parts was the same person, and while they were written down in sequence - which is a very fair, respectable, methodical approach applicable to geniuses as well as anyone else - I know for sure that the documents were "non-causal." The contents of the design changed the requirements long before anyone wrote down the design. I was there. You can't fool me. Just try to tell me the requirements would have been exactly the same if Mediawiki hadn't existed as a model. It was all a good exercise in in thinking clearly. But it wasn't a "deterministic engineering process." It was a sham. It was an ingenious bunch of work that couldn't help but come out because all the right pieces, including the right people with the right motivations, were in place.

So the question is, then, how did it get to be the right product? Where did the genius stop and the real world begin?

The answer to that question is the key to everything.

Syndicated 2006-10-21 18:29:48 from apenwarr's log

2006-10-17: ISO-9001; Writing stuff down; High-level intellectual discussion


My apartment in Montreal is near an ISO-9001 certified parking lot. I didn't even know that was possible. Or necessary. Or even something that would have occurred to anybody in a million years. But now... I'm just not sure anymore. Is it really safe to deal with those other, unprofessional, "fly by the seat of your pants" parking people? I mean, what if I'm parked there and they just go out of business, or the pavement falls off, or worse? Will they have all the requisite forms?

Writing stuff down

    Before you read this, please understand that I never wanted to write this document, being grossly under-qualified, but I always wanted to read it, and this was the only way.
    -- Paul "Rusty" Russell, Kernel Hacker

apm wonders if an expressed idea is more valuable than an unexpressed one. You can spin in circles for hours wondering what "valuable" means and what's the meaning of life and whether there's a God, but let's think of it this way instead: why exactly is it so hard to write things down?

Well, it is.

I think the reason I write stuff down, at least, is because writing it down forces me to make it better. It's so easy, once something is written down, to see that it completely doesn't make sense and you'd better start over. (Perhaps this is why salespeople don't write stuff down.) So even if you don't care whether writing your story will improve the lives of anyone else, maybe you can justify all that work by thinking that maybe you'll be improved in the telling.

Speaking of which, several people have asked me why exactly I write this journal, since it's pretty obvious that I mostly blather on about nothing important, there's no particular unifying theme, most people don't understand what the heck I'm talking about most of the time, I'm certainly never going to get rich doing it, and so on. I've made up several answers in the past. But here's another possibility: maybe writing it down helps me to clarify things in my own mind. And maybe pretending to write for a "real" audience keeps my standards a bit higher, so that I'm forced to clarify things even more. But in the end, maybe I just write because I enjoy reading what I write. And yes, I always laugh at my own jokes. Always. Even if it's just out of sympathy.

Err, my condolences if you actually enjoy reading this crap. Feel free to continue. But you'll be sorry:

High-level intellectual discussion

Okay, I just have to get this out of my system. The other day I was cleaning up a spill - as a purely scientific experiment done in the most hygienic way imaginable (given the circumstances) - with my tongue. And I noticed, to my amazement, just how effective it was. Now, we all know that smearing around a spill with your finger just smears it around and makes it worse. And paper towels sometimes help, particularly the extra-absorbant kind.

But as far as I know, my tongue isn't extra-absorbant. It's not like I lick stuff up, then pull it back in and squeeze it out so I can lick up some more. You just lick stuff up, and it goes in, and then you can lick more stuff up. Why can't my fingers do that? Isn't it amazing? Wow.

Syndicated 2006-10-18 13:02:56 from apenwarr's log

201 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!