20 Feb 2007 logic   » (Journeyer)

Production vs. Development

I've been doing UNIX administration for quite a few years now, and I know a few other people who would call this their profession. I've been noticing, though, a rather drastic difference in attitude toward systems and users in some of them, and I think I've finally nailed down where that difference might be coming from. Fair warning: as usual, this is written basically as a stream-of-consciousness kind of thing, so I may or may not actually get to the point.

You have a few different priorities as a systems administrator, and your job is to juggle them using your best judgement (weighing business needs, available resources, personal time, etc). One of these priorities is production support; keeping the systems that run your business online and performing whatever task it is that they are supposed to be doing (in the case of an Internet-facing business, for instance, this could mean keeping the web, app, mail, and other servers running; for a development shop, it could mean keeping the build systems working smoothly). Inevitably, though, there is another audience that systems administrators are asked to deal with: users. In some cases, that means developers; in others, it means the sales team. In either case, as a technical lead, your job is to help them get their job done. Some administrators get lucky: they only have to deal with one environment or the other. In most shops, however, the administrative staff shares responsibilities. This sounds fine, until you consider the psychology of the two situations.

Production support generally involves saying no a lot. Change is bad; slow, methodical improvements are fine over time, but radical departures from what is known to work are frowned upon. Taken to the logical extreme, you end up with a heavy-weight change control process where even small system changes are scrutinized heavily. And to those who hate such things (and I'm right there with you), this is a good thing. Production systems are differentiated from others by the fact that the business lives and dies by them functioning correctly. By extension, things that effect those systems ought to be tightly controlled and monitored.

But development (or, in some environments, user) systems are a completely different thing. Joel Spolsky has talked about this kind of thing at length: the systems are a tool to help the users do useful work. If a development group needs a machine or two for testing, then find a couple somewhere. If a user needs access to the Internet to research a sales pitch they're working on, give it to them. This doesn't mean giving them everything they ask for; this means finding out what they need to do their jobs, and working toward that in a collaborative manner. The systems administrator provides the role of technology expert and facilitator in this arena.

Take someone who has done production support for years, and give them a development group to support, and Bad Things Happen. The admin can't understand why on earth they want to do things like arbitrarily shuffle 50GB files around the network, use various outdated tools that only support older insecure protocols, or talk on IRC or an instant messenger with people outside the company. They lack perspective: they don't see how that style of systems management affects real people trying to get their job done.

I won't even get into the mess that can happen if you try the reverse (a development-only systems administrator in a production role).

I'm finding that there are very few administrators who can successfully balance both mindsets. As geeks, we seek out uniformity because it regularizes our workloads, but you simply can't reconcile the two: to be successful at both, you have to treat both as the unique environments that they are. I've talked before about administrative geeks needing to step beyond their job title, and this is part of it: working with a mindset of "value to company" rather than the micro-view of what you happen to be doing today.

Anyone can say "no" all the time.

Syndicated 2006-05-23 09:10:00 from esm

Latest blog entries     Older blog entries

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!