Advogato blog for logic

Django'd

Mon, 10 Mar 2008 02:06:44 GMT

Well, it took a bit longer than expected, but I've migrated this blog from Pyblosxom to Django. More technical details after the jump.

Adventures in blogging

Mon, 10 Mar 2008 02:06:44 GMT

I've been using PyBlosxom for the last few years for my online presence here, and while it certainly does the job, I have a habit of getting bored with stuff that works after a while. So, I'm in the process of whipping up my own blog using Django, because hey, web frameworks are the new red. Or something. :-)

Actually, real reason is that PyBlosxom is category-centric; there's no real inherent understanding of a post having multiple categories (perhaps tagging/multi-category functionality is there in a newer version, but upgrading has always been difficult because of a few changes I've made). I found most posts I made ended up touching on multiple categories; racing and geek stuff often overlap, for example. So, lacking native tagging of articles, and not wanting to mess with some of the tag add-ons I've seen for PyBlosxom, I figured I'd just write my own.

What's interesting about this move is that the most recent framework I have experience with is Rails, so it's been interesting to see a Pythonista spin on the idea. What's funny is that, from a functionality perspective, they're really quite similar. Scaffolding, database interaction, view and administration generation are all disquietingly similar. Except, you know, that Ruby vs. Python thing. ;-)

As expected, the basic blog application was a snap. I've been spending most of my time on "fit and finish"; putting together template tags for displaying the sidebar calendar, working out a quickie scheme for importing all the old data, and adapting my current "static" content to the flatpages contrib framework. I'm also trying to make sure I don't break the old URLs too badly.

Once I have most of what PyBlosxom does for me today implemented, I'll probably just cut over at that point. Afterward, I'll concentrate more on some of the specialty stuff I'd like to pull in: timeslips, logs from the cars, dive logs, etc. Fun stuff. :-) And at some point, I'll throw the whole project into SVN so others can poke at it. (I might have to post up my calendar template tag on Django Snippets at the very least, since the only other one up there uses datetime instead of the new calendar module.)

Adventures in blogging

Fri, 29 Feb 2008 18:05:27 GMT

Once I have most of what PyBlosxom does for me today implemented, I'll probably just cut over at that point. Afterward, I'll concentrate more on some of the specialty stuff I'd like to pull in: timeslips, logs from the cars, dive logs, etc. Fun stuff. And at some point, I'll throw the whole project into SVN so others can poke at it. (I might have to post up my calendar template tag on Django Snippets at the very least, since the only other one up there uses datetime instead of the new calendar module.)

Back to the source

Fri, 20 Jul 2007 13:04:13 GMT

Following on the heels of that great old-school DEC advertisement, I get another blast from my non-academic past at University: an interview with Richard Bartle, one of the creators of the original MUD that ran at Essex University back in the late 1970s and early 1980s. MUDs are the majority of the reason I'm doing what I do today (after a fashion), and were my first introduction to social computing and network programming. What's really interesting to me is that he's saying the same things that any of us who were involved in building these text-based online delusions in the 80s and 90s were constantly harping on about, and it all still applies even though the graphics have gotten better. (In fact, he makes the excellent point that with the improvement in visual stimulus, the ability to interact with the environment has actually gotten worse, because of the nature of ad-hoc natural language.)

It was a good read; I haven't played with these things in long time (aside from a brief flirtation with creating a Python-based runtime-mutable object environment similar to the facilities that LP-style MUDs had), so it was a nice walk down memory lane, and a reminder that there really isn't much new under the sun, just a prettier interface on it.

Quick initrd note

Wed, 18 Jul 2007 18:05:59 GMT

This is mainly a note for myself, but someone else might find it handy. Working with modern initrds requires monkeying with cpio, which I can never remember the correct command line parameters for off the top of my head. So, here's a quick workflow (borrowed from Chip Shabazian) for manipulating an initrd image.

mkdir initrd
cd initrd
gzip -dc path/to/unzipped/initrd.img | cpio -id
# Make the changes you need
find . | cpio -c -o | gzip -9 > /path/to/new/initrd.img

Chip Shabazian's presentation at LinuxWorld

Are you on...the INTERNET?

Fri, 13 Jul 2007 03:04:20 GMT

This is too cool for words:

God, I feel old.

Observations on Geek Behavior

Tue, 27 Feb 2007 17:04:25 GMT

I came across a del.icio.us link to this article on Coding Horror this morning, and it interested me on several levels.

First, there's the obvious: the article talks about the difficulty in locating programmer candidates that can perform basic programming assignments in an interview setting. Several commenters hit on some of the things that can contribute to this problem, including the psychology of an interview environment, the ability of the candidate to manage their stress, and the obvious problem that there's a lot of CS graduates who haven't spent enough time behind a keyboard to be useful in a production programming shop. That's what his article is about, and I'll leave that discussion there.

I'm much more interested in the response to the article. Responses range from senior developers to folks still in college or high school, and the resounding pattern is: they want to solve the problem. That's not what the author was shooting for, obviously, but readers were falling all over themselves (myself included) to post up the first response in their favorite language.

My speculation is that programmers who love what they do - the kind of folks who hang out on programming-related websites, who read the latest O'Reilly books when they come out, who participate in open source projects and "scratch itches" regularly - can't ignore an opportunity for practice, much like what Dave Thomas talked about when he introduced his idea of "Code Kata". Throw out the accusation that most programmers can't solve a simple problem, and you've practically demanded that they take a swipe at it.

There's more to it, though; lower the barrier enough, and more and more people get attracted to the idea. You see this kind of behavior on topical mailing lists: ask a complex question, and you may get an answer, but not a lot of people will jump up and offer their help. Ask an easier question, perhaps something that a lot of people know the answer to (or can easily derive the answer to), and suddenly, you get an overwhelming response. There's an aspect of ego at play here that's interesting; it's an opportunity to present yourself as a subject matter expert, or at least as someone knowledgable of issues surrounding the question.

Interesting stuff, to be sure. I suspect there's a good group dynamics case study or paper to be had here for someone with the time.

Ruby ActiveRecord and lazy instantiation

Tue, 20 Feb 2007 19:05:09 GMT

Let's say I have a data set in a SQLite database. And, let's say I need to iterate over every row in that database, performing some action based on it. There's a few ways I can do this. First, the old, direct-to-driver version:

require 'sqlite3'

db = SQLite3::Database.new("test.db")
db.execute("select * from test_entries") do |row|
  ...
end

SQLite example

That works, but of course, it's bound forever to SQLite, and we'd like to abstract that a bit. The "new hotness" appears to be ActiveRecord. So, let's reproduce that using AR:

require 'active_record'

class TestEntry < ActiveRecord::Base; end

ActiveRecord::Base.establish_connection({
  :adapter => 'sqlite3', :dbfile => 'test.db'
})

TestEntry.find(:all) do |entry|
  ...
end

ActiveRecord example

Here's a tip: don't do that. That find call actually instantiates an ActiveRecord::Base for every single row in the database, then returns the result set, which you're now iterating over. So, for anything but a trivially-small result set, you're going to use a ton of memory, and create an enormous workload for very little good reason.

So, I suppose this blog entry is really just a whine: why isn't there a lazy-instantiation iterator for result sets in ActiveRecord? In this (common, I'd think) case where you want to do some sort of trivial filter or action across the entire table on a per-row basis, but don't care about the row after you've worked with it, that would be a huge win. The interesting thing is that this is exactly how the auto-generated association methods work, and you actually have to go out of your way to do eager loading of associations (since just automatically doing joins under the hood is an obviously bad idea).

My specific use case is an application log stored in a table that I'd like to iterate over for some basic statistical analysis, and a single day of logs is a 600M SQLite database (approximately 3.6M rows). This won't fly for doing a single day of processing, and I'd like to be able to do something for monthly and quarterly reporting.

I'll probably resort to DBI to accomplish this in a database-agnostic manner, but that feels like an incredible step backwards. I just can't understand why the find* methods weren't written as iterators that do lazy instantiation of the result set; the interface would be effectively the same, but the potential performance gains could be tremendous in certain cases.

Linux on a BladeRunner system

Tue, 20 Feb 2007 19:05:09 GMT

So, you bought one of those shiny new Penguin Computing BladeRunner systems, and were thinking to yourself: "I wish there were a good guide to getting Linux to do what it should on these things". Well, I'm going to try and cover two of the basics here: serial console configuration, and interface bonding. Everything else is pretty much stock stuff, but I had a heck of a time figuring out configurations that worked here. The discussion below assumes you're running RHEL or Fedora, but the idea should be fairly clear.

First step: learn about some of the remote management features of the BladeRunner. You have a couple of configuration commands that are useful here, related to console management and power. Log into the chassis, and type conf; for some reason, they put this under "configuration" rather than "management" or some other similar tree of the command set. Now, you can control power to individual blades with server-blade power <num> [cycle|forced-off|off|on] (where <num> is the number of the blade you're managing). on means exactly what it sounds like; power on the blade if it's currently off, just like it would if you hit the power button on the front. off sends an ACPI power vevent to the blade, advising it to shut down; it doesn't actually force power to be pulled; that's what forced-off does. cycle removes power to the blade, and adds it again; off the top of my head, I don't recall if it does so gracefully or not. Check the documentation, it might be there (but probably not).

The next command that's interesting here is server-blade console-redirect <num>: that grabs the serial console for that particular blade, and displays it to your current session. The serial console configuration appears to be set in stone, from what I can see: 57600 bps, 8 data bits, no parity, 1 stop bit, with hardware (RTS/CTS) flow control. If you power up a blade fresh from the factory, you'll notice that the BIOS is already set up to redirect output to both the serial port and the built-in KVM in the chassis.

That's about all I'll say about the chassis management side of things, since they actually do cover this stuff fairly well in the documentation. The problem I found with the documentation was that it didn't cover host-side configuration of certain things, specifically interface bonding and serial console configuration. Maybe they thought it was out-of-scope for their documentation; I hate to be the bearer of bad news, but the chassis is pretty useless without systems to run on it. (Here endeth the snide remarks, hopefully.)

Next up is host-side serial console configuration. This is fairly standard stuff, once you know what the settings need to be, but I'll spell it out here too. First is the bootloader; GRUB, in my case. You'll want to add a couple of lines to the GRUB configuration file (in the main stanza):

  serial --unit=0 --speed=57600 --word=8 --parity=no --stop=1
  terminal --timeout=10 serial console

Optionally, you can also add hiddenmenu to the options; that reduces the amount of text being displayed at boot time to a minumum, unless you need it. Set the timeout on the terminal line to whatever is appropriate for you; 10 seconds before you're kicked to the primary display was good enough for my needs, since I didn't want the boot process to take much longer than it already did.

Next up, also in the grub configuration file (on the command line of each kernel you want to boot), are the kernel parameters that get your boot output sent to the right place. Adding console=tty0 console=ttyS0,57600n8r to the kernel command line seemed to do the job for me (display on /dev/ttyS0 at 57000 bps, no parity, 8 data bits, hardware flow control). If this is a stock Red Hat/Fedora installation, I suggest getting rid of the rhgb and quiet entries on that line as well; for server use, you really want to see that console output, so that when you inevitably get a kernel that has a bad day, you don't have to monkey with grub configuration via a 57600bps serial line to capture the panic.

The final step is making it possible to log in on the console. Adding the following line to /etc/inittab worked great for my needs:

  co:2345:respawn:/sbin/agetty -h ttyS0 57600 vt100

The co is an name for the entry, but it's become a defacto standard for the inittab name for the serial console entry. 2345 should be obvious (runlevels to operate in), respawn means run it again after you log out, not just once. The agetty command line just specifies hardware flow control (-h), linux device (ttyS0), speed (57600), and terminal type (vt100); change terminal type to anything appropriate for your environment, but vt100 is pretty safe for most folks.

That covers serial console configuration. For more details on that piece of the puzzle, I strongly recommend reading over the Remote Serial Console HOWTO; it discusses some of the ins-and-outs of other boot loaders, some other configurations you might be interested in trying, and some general advice for this kind of setup.

Next up is interface bonding; every blade has two Broadcom NetXtreme BCM5780S Gigabit Ethernet NICs built in, each connected to one of the two management blades through an internal backplane. I've been using the tg3 drivers for these quite successfully; the bcm5820 module is also available as a third-party add-on, but appears to be deprecated at this point. So, your /etc/modules.conf or /etc/modprobe.conf (depending on version) will likely have a couple of lines in it like alias eth0 tg3 for each interface. Next up is bonding configuration: add the following lines:

  alias bond0 bonding
  options bonding mode=2 arp_interval=500 arp_ip_target=10.0.0.1

Change 10.0.0.1 to whatever your default gateway is. You might be asking yourself why you shouldn't use MII instead of ARP on these systems, and it's a good question. The failure mode of the chassis internal network precludes it; you can end up with one management switch down, but the MII status of the ethernet interface still shows it being up. You can test this out by setting your bonding configuration to use MII instead, then yanking out one of the chassis management blades.

The next piece is very specific to RHEL or Fedora; please refer to your Linux vendor's documentation for details on how to do it elsewhere. In /etc/sysconfig/network-scripts, create a file called ifcfg-bond0 that looks exactly like a normal ifcfg-* file (in fact, you might want to just move your existing ifcfg-eth0 or similar file there); specify how you want this combined interface to behave. Then, for each of ifcfg-eth0 and ifcfg-eth1, add a file that looks like:

  DEVICE=eth0
  TYPE=Ethernet
  BOOTPROTO=none
  ONBOOT=yes
  MASTER=bond0
  SLAVE=yes

(Change DEVICE= to the appropriate line for each device.) More information on interface bonding under linux is available from the OSDL Linux networking site.

That's the main stuff. Obviously, test all of these changes with a complete boot to make sure console redirection is happening the way you want it, and to be sure that your bonded interfaces are showing up as a single bond0 interface (and that you have connectivity to and from them). Good luck.

Sysadmin Day

Tue, 20 Feb 2007 19:05:09 GMT

Have you hugged your systems administrator lately?