Real-life Buffer Underruns
Okay, this is about a week old but I'm so impressed by it that I'll post it
anyway. While I was visiting my family last week, I finally learned how
water pressure tanks work.
Background info: my family lives outside of town, and gets their water from
a well. Wells are in the ground, and typically the water would rather be in
the ground than come up to your house and shoot out of your taps. In a
city, they make the water come out by pressurizing the water at the Central
Water Pressurizing Authority (or whatever), but out in the country, you're
left to your own devices.
What you *can* do, and some people do this but it's bad, is to just turn on
your pump whenever you turn on the tap. The pump works like an airplane
propeller, sucking the water from the ground (or a holding tank, or whatever)
and pushing it through your pipes into your tap. The problem: this means
your pump has to turn on and off an awful lot, and response to water
requests is rather sluggish.
The alternative is a pressure tank, in which you pump water into the tank,
increasing the pressure so that the water wants to get out. The more water
you add, the more it wants to get out. Now here's the catch: why does
adding water make it want to get out more? What is "water pressure"
exactly? Since liquids can't be compressed, how can adding more liquid
increase the pressure?
My dad finally told me the answer to this last week. (It turns out he's
always known this, but I never asked. Go figure.) The answer is: the
pressure tank is actually a sealed tank full of air, and the water goes into
an expandable baggie *inside* that. An empty tank has some amount of air
pressure (with one valve for adding/removing air) and an empty baggie
attached to the in/out water pipes.
Adding water to the tank decreases the volume of air, but not the amount of
air. Thus, because of everyone's favourite chemistry formula, PV=nRT, the
air pressure increases. The air wants to expand and push the water
out of the tank, and the more water you add, the more the air pressure
increases - and thus, the more the water pressure increases. The "water
pressure" that you can measure on the water outflow pipe is exactly equal to
the air pressure in the tank.
Why do we care? Well, this means that in fact most water pressure tanks are
more than 50% empty even when they're "full." They're mostly air. And if
you tune them wrong, it's even worse. So don't expect that your 65 gallon
water pressure tank will ever have 65 gallons of water in it, or you'll
seriously confuse yourself.
And now we get into my area of interest, namely networking. What's the
point of the tank again? To make it so the pump doesn't have to instantly
respond to changes in water requests. Also to make it so that, even if the
well runs dry temporarily, there'll still be water available to service our
requests. But standard tank tuning algorithms (which my dad also knew,
because he read the instructions for his tank) involve setting up the tank
to activate the pump when the tank is almost empty and deactivate the
pump when the tank is almost full (where "full" is using our new
definition, meaning still more than half empty). In programming, we call
these two points the "low water mark" (LWM) and the "high water mark" (HWM),
Problem is, if the well runs dry while you're nearing the low water mark,
you have almost nothing left in your buffer, er, tank, to service the
requests. So, for maximum recoverability, you want the low water mark to be
as high as possible. But this decreases the "block size" of demands on the
service provider, er, pump. Where with the tank almost empty you could say
"send me 30 gallons of water", now you have to say "send me 2 gallons of
water" much more often. In the typical heavily optimized case, your tank
doesn't buy you anything because the LWM=HWM, just like with no tank at all.
You'll only notice the improvement if the service disappears altogether for
Of course, you're also receiving water that's 30 gallons old - your tank has
increased the staleness of the data, er, water, that you receive.
Okay, the other problem, and this is the point of the story: the reason my
dad bought the tank in the first place. The well was
running dry pretty often this winter, since it was a pretty dry year. To
improve the situation, my dad thought it would make sense to buy a larger tank
(buffer), so that if there was a temporary high demand for water followed by
a long period of low demand, the tank could fill up and not stress the well
too much. Now, this didn't really work, mostly because of the HWM/LWM
tuning problem above, and partly because no buffer will save you if the
long-term average bandwidth of the network is less than average demand.
(Water caching is gross.)
In fact, the larger tank made things worse once spring came and the
water was abundant again: because the transaction size (HWM minus LWM) was
much larger than with the old tank, it would run the pump for longer in one
shot. Unfortunately, HWM-LWM for the tank is more water than would fit
in the entire well, so you're guaranteed to underrun your buffer every
time you go to the network for more service, thus causing TCP to start
limiting your bandwidth and... oh wait, I'm mixing my metaphors again.
Anyway, the tank made it appear that there still wasn't enough bandwidth,
even when the congestion went away.
The Moral of the Story
Adding buffers doesn't fix anything unless you tune them properly. And they
sometimes make your system unusable.