Older blog entries for apenwarr (starting at number 12)

What have I done?

According to http://net-itech.com/america/news/story_00025.htm, I said this about Guadec:

    "Net Integration produces and contributes to several Open Source projects in the process of making our Net Integrator and other products, and we're happy to be able to give back to the community by assisting at GU4DEC," says Avery Pennarun, Vice President Development and Senior Systems Architect, Net Integration Technologies. "UniConf, our advanced object-relational web-enabled remote management console with cross-platform transaction-oriented XML content delivery, will totally change the way people do business and revolutionize the world of Open Source."

So here's the bad news: I did, in fact, say that.

And the less bad news: I was kidding when I said it, I swear!

I honestly expected the people in marketing to see my joke, especially that last part about "totally change the way people do business" and "revolutionize the world of Open Source." Unfortunately, it's now a press release linked off our front page. I think I have just learned some kind of important life lesson.

I'll let you know when I figure out what it is.

Okay, so I lied somewhat in my UniConf paper for GUADEC. The paper admits that I lied, but doesn't say about what exactly. I'm going to spend some time this weekend and over the next week trying to make as many of those lies as possible come true, thus making myself retroactively more honest.

The hardest part to get right will be the sample monikers. I probably should have tried those before I included them in the paper.

Visited a Buddhist temple on Thursday. I saw an hour-long ceremony (more than 50% of which was silent meditation) and then they gave me some cookies, literature, bottles of blessed water, and a sacred tomato.

(Actually, I was later assured that the tomato was not sacred at all, but just any old tomato. But it sounds better if I call it a sacred tomato.)

4 May 2003 (updated 4 May 2003 at 05:35 UTC) »
Real-life Buffer Underruns

Okay, this is about a week old but I'm so impressed by it that I'll post it anyway. While I was visiting my family last week, I finally learned how water pressure tanks work.

Background info: my family lives outside of town, and gets their water from a well. Wells are in the ground, and typically the water would rather be in the ground than come up to your house and shoot out of your taps. In a city, they make the water come out by pressurizing the water at the Central Water Pressurizing Authority (or whatever), but out in the country, you're left to your own devices.

What you *can* do, and some people do this but it's bad, is to just turn on your pump whenever you turn on the tap. The pump works like an airplane propeller, sucking the water from the ground (or a holding tank, or whatever) and pushing it through your pipes into your tap. The problem: this means your pump has to turn on and off an awful lot, and response to water requests is rather sluggish.

The alternative is a pressure tank, in which you pump water into the tank, increasing the pressure so that the water wants to get out. The more water you add, the more it wants to get out. Now here's the catch: why does adding water make it want to get out more? What is "water pressure" exactly? Since liquids can't be compressed, how can adding more liquid increase the pressure?

My dad finally told me the answer to this last week. (It turns out he's always known this, but I never asked. Go figure.) The answer is: the pressure tank is actually a sealed tank full of air, and the water goes into an expandable baggie *inside* that. An empty tank has some amount of air pressure (with one valve for adding/removing air) and an empty baggie attached to the in/out water pipes.

Adding water to the tank decreases the volume of air, but not the amount of air. Thus, because of everyone's favourite chemistry formula, PV=nRT, the air pressure increases. The air wants to expand and push the water out of the tank, and the more water you add, the more the air pressure increases - and thus, the more the water pressure increases. The "water pressure" that you can measure on the water outflow pipe is exactly equal to the air pressure in the tank.

Why do we care? Well, this means that in fact most water pressure tanks are more than 50% empty even when they're "full." They're mostly air. And if you tune them wrong, it's even worse. So don't expect that your 65 gallon water pressure tank will ever have 65 gallons of water in it, or you'll seriously confuse yourself.

And now we get into my area of interest, namely networking. What's the point of the tank again? To make it so the pump doesn't have to instantly respond to changes in water requests. Also to make it so that, even if the well runs dry temporarily, there'll still be water available to service our requests. But standard tank tuning algorithms (which my dad also knew, because he read the instructions for his tank) involve setting up the tank to activate the pump when the tank is almost empty and deactivate the pump when the tank is almost full (where "full" is using our new definition, meaning still more than half empty). In programming, we call these two points the "low water mark" (LWM) and the "high water mark" (HWM), respectively (snicker).

Problem is, if the well runs dry while you're nearing the low water mark, you have almost nothing left in your buffer, er, tank, to service the requests. So, for maximum recoverability, you want the low water mark to be as high as possible. But this decreases the "block size" of demands on the service provider, er, pump. Where with the tank almost empty you could say "send me 30 gallons of water", now you have to say "send me 2 gallons of water" much more often. In the typical heavily optimized case, your tank doesn't buy you anything because the LWM=HWM, just like with no tank at all. You'll only notice the improvement if the service disappears altogether for a while.

Of course, you're also receiving water that's 30 gallons old - your tank has increased the staleness of the data, er, water, that you receive.

Okay, the other problem, and this is the point of the story: the reason my dad bought the tank in the first place. The well was running dry pretty often this winter, since it was a pretty dry year. To improve the situation, my dad thought it would make sense to buy a larger tank (buffer), so that if there was a temporary high demand for water followed by a long period of low demand, the tank could fill up and not stress the well too much. Now, this didn't really work, mostly because of the HWM/LWM tuning problem above, and partly because no buffer will save you if the long-term average bandwidth of the network is less than average demand. (Water caching is gross.)

In fact, the larger tank made things worse once spring came and the water was abundant again: because the transaction size (HWM minus LWM) was much larger than with the old tank, it would run the pump for longer in one shot. Unfortunately, HWM-LWM for the tank is more water than would fit in the entire well, so you're guaranteed to underrun your buffer every time you go to the network for more service, thus causing TCP to start limiting your bandwidth and... oh wait, I'm mixing my metaphors again. Anyway, the tank made it appear that there still wasn't enough bandwidth, even when the congestion went away.

The Moral of the Story

Adding buffers doesn't fix anything unless you tune them properly. And they sometimes make your system unusable.

More Mozilla

dan: I really did try to find something that the average person would find more useful in Mozilla than IE, but I stand by my opinion. The average person doesn't benefit. If my grandma could figure out what a popup was or how to configure blocking for it, that would be great; if IE didn't do perfectly good password management already (and even better with RoboForms) that would be wonderful.

I was honestly hoping (expecting?) that I'd find something in the list of 101 items, since I'm fully aware that Mozilla (or its variants) are the best non-IE browsers available. I use it myself, since I don't have a Windows desktop. But unfortunately Microsoft has us beat. If one of the things in the list had been "loads faster" or "renders faster" or "renders more pages that people actually visit" or "integrates better with your desktop" or "fits in with your existing desktop theme" or "isn't ugly", then you would have had me. Sadly, IE does all of those things better than Mozilla, and those are the things that the average person cares about.

I was even personally pleased to see that Mozilla is now claiming pipelined HTTP, which is one of my big concerns - sadly, average people don't care about that, either.

22 Apr 2003 (updated 22 Apr 2003 at 04:10 UTC) »
Mozilla Still Sucks

dyork: I was distressed to read the entire "101 things that Mozilla can do that IE cannot" article and not find a single thing that an average person would find useful. Looks like browsers are officially "mature" now, kinda like word processors. (Translation: only really really smart people can think of something useful to add.)

Election Algorithms

Bram: The election analysis stuff is really interesting. I finally understand what's wrong with the Canadian/U.S. election systems (which are the only ones I care enough about to have watched, sorry). Now to convince the Powers That Be that they should switch to the condorcet system. Good luck.

There is also something here about document correlation, but I'm not yet sure exactly what.

Opening the NIT

After more fiddling, we're at last almost ready to replace the old open.nit.ca with the new new OpenNit quasi-wiki. And the anonymous cvs access is all set now using cvsd, pending one DNS registration.

Branch Constraint Theory

If I can just tear myself away from random browsing and email for a few hours, I'll be able to sanitize my (as previously mentioned) Branch Constraint Theory paper. Maybe tonight. Should I post it as an advogato article? Hmm.

(I must be on vacation, because I actually feel like I'm catching up for a change. I'm sure I'll get over it soon.)

14 Apr 2003 (updated 14 Apr 2003 at 03:21 UTC) »

Woo hoo, my UniConf and Sharvil's ExchangeIT talks got accepted for Guadec. In half-hour timeslots instead of hour ones, which is unexpected but not really a problem. (Were you supposed to request something, or something?)

Now all I need is a passport, a plane ticket, and a place to stay. Eek!

Amazing Discovery

Today I learned that according to Statistics Canada, there are more married women in Canada than married men. (Statscan is truly awesome, so I suppose there's some explanation for this.)

Document Correlation

This weekend I wrote a page correlator using ideas I blatantly stole (badly) from Paul Graham's essay on spam detection using Bayesian Filtering. His math makes way more sense than my cheesier version, but it mine just a hack so that's okay for now.

Quick summary of the theory: the interesting aspects of a document are characterized by the locally most common words among the set of globally least common words. That is, if I say the word splahooie a lot, but nobody else does, that makes "splahooie" an interesting characteristic of my document. But if everyone else says splahooie, *or* I don't say splahooie very much, it's not a keyword.

So anyway, that works pretty well with some refinement. But using this technique and a cheesy "keyword correlation" algorithm in perl+mysql, and using our internal company wiki (900+ pages) as a data source, I made it so you can get a list of "interesting pages" related to your current page.

The results were... interesting. The algorithm, though simple, is surprisingly good. What it does, though, is a bit weird, because of the way we define "interesting." We only care about globally uncommon stuff. The result is a system that tells you not exactly, "What's related to this page?" but rather, "What's unusual about this page?" If I ask about pphaneuf, it mentions XPLC but not Net Integrator, even though I (as an evil manager) make sure he works more on the latter than the former. But other people do too, so XPLC is more interesting as far as the algorithm is concerned. It's kind of like the anti-google.

Hmm, an abnormality detector. I bet I could sell this to school boards in Kansas.

Cascading Failures

Well, I promised not to discuss my life in here, but since I'm about to use it as an example of generalized system failure modes, I figure it's okay.

The goal: from Waterloo on Thursday morning, travel to Montreal by Friday evening at 8 pm. By car, this is a 7 hour drive. No problem, right?

Well, we were going to rent a car and drive on Thursday afternoon, but it starting snowing/raining/sleeting so we decided not to drive after all; instead, let's take the train, which is safer in bad weather. Since the night train leaves you kind of tired, we decided to take the Friday morning (9:30 am) train from Toronto.

Friday morning, the weather still sucked, but that's okay. We called a taxi at 6:20 am to take us to the bus station in Waterloo. At 7 am, it finally arrived - delayed by bad weather, of course. So we missed the 7am bus to Toronto. No problem, there's an 8 am bus that should still make it to Toronto in time. Unfortunately, the 8 am bus showed up at 8:30 (bad weather), departed shortly afterwards, and got to Toronto at 10:00 (extra late - bad weather). No problem, though; we rescheduled our train tickets from the 9:30 to the 11:30 train. We changed the reservation by cell phone from the bus, luckily, because by the time we arrived all the trains for the day were fully booked. Turns out all the airports were closed (bad weather) and the people taking flights had all switched to the train.

As we were picking up the tickets, they made an announcement that the 11:30 train would be leaving at 12:30 instead - bad weather. No problem: the 11:30 is supposed to get to Montreal at 4:45, so an hour later is 5:45, and even with additional weather delays I should *certainly* be in Montreal by 8 pm. So, we have some time, let's go for lunch.

At 12:20, we came back and found out that the train had left at 12:07, having been re-re-scheduled while we were gone. In fact, they had made the new announcement before we left the station, but because of a ridiculously loud random music performance (something about the Juno awards) in the middle of the station at the time, all the public announcements were inaudible.

Feeling guilty, they asked us to wait while they figured out what they'd do to get us to Montreal. The result: at 1pm or so, we found out that they could squeeze us on the 3:30 train (arrives around 9:30; useless) or a special 2:30 shuttle bus (could arrive at 8:30 in *good* weather; useless). So Via Rail wasn't going to be able to help.

Last chance: rent a car after all (there's a rental place at the train station) and drive it to Montreal. That takes at least 6 hours in good weather. By 1:30 we had almost finished filling out the rental forms, meaning that we *could* be in Montreal by 7:30 on a good day. Sadly, it wasn't a good day. (Interestingly, if we had known at 11:30 that we would miss the train, the rental would have saved us.)

I mentioned above that the airports were closed too (bad weather).

The Moral of the Story

Despite a metric tonne of backup plans (an extra day; an extra bus; an extra train; backup train should still arrive early; could rent a car if the train was cancelled) and slippage, we *still* didn't get to Montreal on time.

In management, we call this "slippage." In clustering, we call this "cascading failures."

The lesson to learn here is that if you're going to add redundancy (like the extra buses, trains, time, etc) you'd best make sure that the same root cause can't screw up *all* of your backup plans at the same time. That means don't put a five-station Oracle database cluster on the same power circuit, don't write software that shuts down and expects the cluster to take over if it gets confused (because what if *all* the nodes get confused by the same thing?), and don't plug all your backup servers into the same Internet connection. For that matter, don't store them all in the same nuclear bunker in the Swiss Alps. If exactly the wrong thing happens, you'll be in trouble.

31 Mar 2003 (updated 13 Feb 2008 at 17:31 UTC) »
Current Status

Offically Not Dead.

Branch Constraint Theory

Oh yeah, baby, I just invented a series of simple mathematical formulas that tells you why your project is taking so long to release, has so many bugs, and seems to show no signs of getting better. It might explain the weird many-way-branching of the Linux kernel and the annoying ever-slowing Debian release schedule, as well as (more importantly) some of the phenomena we've seen at work.

It's 28 pages of pure fun. I'll see about publicly releasing it if I can trim out the NITI-internal bits.

The most brilliant insight, which seems obvious once you read it but is quite non-obvious until then, is:

If you have two branches, a stable and an unstable branch, and you spend more time in code freeze than in feature addition, then each release will take longer than the last. That's because you start stabilizing the current "unstable" branch no sooner than you release the previous "stable" branch, so every feature addition phase is longer than the previous bug fixing phase. And if, within a release, your bug fixing phase is longer than your feature addition phase, well... it flies out of control, and you're doomed.

Doomed, that is, until you create MORE SIMULTANEOUS BRANCHES! And then you're differently doomed! Woo hoo!

Corrollary: simply adding more testing time at the end of a release cycle will not save you. And I'm now certain of it.

Oh, another interesting insight is that if you add features more slowly you can make more frequent releases. It's a fundamental law of software engineering and I proved it more-or-less mathematically. Go figure.

Update 2008/02/13: The link the Advogato article no longer works. You can find the Branch Constraint Theory PDF here instead.

22 Mar 2003 (updated 22 Mar 2003 at 20:27 UTC) »

Okay, so seeing as it's All My Fault, I suppose I should say something interesting.


Right, so Doc Searls was visiting our office (NITI-Toronto) last week and he said some interesting stuff. A lot of this is straight out of the ClueTrain, but since it was personalized for us, it seemed to hit home that much harder. As time goes on we'll be fiddling with stuff and downplaying the brochureware on our main web site, since (as he correctly points out), nobody cares about or even believes brochureware. Hear hear.

But what nobody mentioned is that our OpenSource site is also brochureware, albeit on less expensive paper. My plan is threefold:

a) Wiki-ize the OpenSource site so people internally will actually keep it up to date.

b) Finally get public anonymous CVS working, so people can find out about and download and maybe contribute to all the new free stuff: WvIPsec (AKA "Thank God it's not FreeS/WAN"), WvStreams, UniConf, ExchangeIT, SSoya, WvTFTP, and a bunch of things I probably already forgot. Note to self: consider making advogato projects for all these so I can add myself to them. Hmm.

c) Start blogging. Note: I hate the word "blog" and all its derivatives, but unfortunately that's all we've got. I'll call this my advogato diary nonetheless, and if you're lucky this is the last instance of the word "blog" you will hear from my virtual lips. Also, while some people claim that the interesting thing about a bl- err, diary, is that you can read all about people's personal lives, I basically have no interest in yours and expect you to have no interest in mine. So I'll stick to the technical stuff wherever convenient.

By the way, although it's All My Fault, choosing advogato as the target was All pphaneuf's fault.


Down with it!

(You will hear more from me on this topic.)

Trust Certifications

Since a bunch of NITI people have joined advogato, I felt as if I should certify them.

I think the whole web-of-trust thing (pgp, advogato - it's all the same) is amazingly cool, although nobody has ever taken proper advantage of it. Come on, let me define my own trust roots and then apply the trust calculation to arbitrary things - now we're getting somewhere.

Now, for advogato in particular, I feel rather silly certifying someone like slajoie, who is, franky, a genius with real-life experience, at Apprentice level just because most of his work has been on non-free software. Someone with years of coding experience who's not a big name in free software is just plain not an apprentice. We need a different word. But I'll follow the rules and certify him that way anyway, because luckily it doesn't matter.

Also upgraded my own self-certification to Master. "Modesty will get you nowhere," that's my new motto. Hence the bl- err, diary.


No comment.

3 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!