31 Aug 2002 goingware   » (Master)

Food for Thought

Zaitcev's and kroah's comments on my efforts to get into embedded systems programming certainly made me wonder if I'm doing the right thing. (Thanks kroah for letting me know who you were!)

I think I'm still doing the right thing, but hopefully I could go into embedded work somewhat less naively than I might otherwise have.

I'm well aware that the nature of the work makes it harder to make one's code free. But I don't think it's hopeless.

For one thing, embedded Linux is booming. There are even some books about it.

There have been some setbacks, though. The Embedded Linux Journal used to be a print publication, but its existence as a separate dead-tree magazine ceased not too long ago, I think a victim of bad economic times. But it's still on the web and I think it now exists as a section within the Linux Journal.

Of course, since Linux was originally developed for desktop and server PC's, it has some limitations that prevent it from being used in some embedded applications. Lots of embedded chips don't have enough memory to run a Linux kernel, let alone any userspace applications. For this one can use eCos. eCos can run in much less memory that Linux, and is architected to make it easy to remove features you don't need from the build so they don't take up space in the ROM image.

There also appear to be some open source Forth implementations.

There are a couple problems with free software being used for embedded systems. One is that one often cannot look at the binary running on an embedded chip, so it is hard to tell that it is running software which should be Free. Likely there are a lot of GPL violations out in the market, running on embedded chips with no one but the manufacturer knowing about it. The best we can hope for is that whistleblowers will leak the secrets, and I expect that will happen from time to time.

The other is that many embedded products don't provide a way for the user to reload the software. They might not be using flash rom, and there may be no I/O ports on the device to load new software onto it. So even if the manufacturer publishes the source, you may not be able to use it.

(I asked RMS about this specifically a while back and he said that the GPL did not require one to be able to load new software on a device that runs a GPL'ed binary. I think this is a real problem. While one could manufacture new compatible hardware, there may not be enough of the hardware spec made public to be able to reproduce it.)

One more problem is that the manufacturer might publish the source but not the hardware specs, so you might be able to write software that does the same thing, but not be able to figure out how to alter it.

Another problem is that a lot of the commercial offerrings in the embedded linux market include proprietary parts. So some firmware may only contain free software, but the tools required to build it may be proprietary. There are also some packages to give Linux real-time support, that don't actually link into the Linux kernel, and some of these packages are proprietary. Both of these are a real problem.

But I don't think it's hopeless. After all, the server software market used to be a completely closed-source proprietary world. Even BSD used to be closed-source because it contained a lot of AT&T Unix code. At some point, U.C. Berkeley started publishing all the parts of BSD that weren't AT&T derived, and then the AT&T parts were rewritten from scratch, so we did eventually end up with an Open Source BSD. (I understand AT&T threatened UCB with a lawsuit over this, which could have been a huge disaster for the University if AT&T won. But AT&T backed down when Berkeley asked AT&T about all the BSD code it was using in Unix.)

In my early career, when I was a sysadmin for the BSD-based SunOS, I was unable to get the source code to SunOS. If I wanted to rebuild the kernel, I had to link together a bunch of .o files that came from Sun - I didn't even have the kernel source. Now, although the license is restrictive, one can at least obtain the Solaris source code. Even the source to the Mac OS X kernel is available (but under a license that the Open Source Initiative has certified but the Free Software Foundation says is incompatible with the GPL.

I think most embedded work is still mostly closed source. I just signed an NDA to get the manufacturer's source code to the chip I'm about to program, and my modifications to it will also be a secret.

But Free Software is not yet at the point where every programmer can make their living writing or even using it.

I think most paid involvement with Free Software still consists of either integrating Free applications or writing extensions to it that are never released either because the license doesn't require it (like BSD) or because the GPL'ed code is only used on a server without actually distributing the binary. (Some have proposed adding a "performance" clause to the GPL which would address this. I brought this up too with RMS, and he said he wanted to address it in the next revision to the GPL, but said "it can't be soon".)

I have heard that RMS feels that if one cannot write Free Software, then one should not write software at all - that is, one should not even take employment as a programmer unless the source one creates is Free. (I don't know for sure that he feels that way, so I won't claim that he really feels this way.)

I don't agree with that. I think that most people with the talent to program well do not have the ability to do as well for themselves and their families in some other occupation. If I could have changed careers these last couple years into some saner job that would still have kept all my bills paid, I certainly would have done it, but even after the dot-com crash, programming is the only work I could find that would pay enough that I can keep me and Bonita housed and fed and our creditors at bay.

I think RMS would likely say one should take a job flipping burgers rather than write closed source, but I don't feel it's right to ask people to be so self-sacrificing. Many advocates of Free Software do not feel as strongly about their principles as RMS does, and even if they do, I think the movement would be better off if we were all in a position of economic advantage.

Finally, I don't think it helps our cause if people aren't allowed to write closed-source software. While the software we write as employees and consultants may not be Free, the skills we sharpen while we write it can be, and usually are, brought to bear on Free Software problems.

I'm certified as a Master here but the fact is that I haven't written that much Free Software at all. I guess I could claim that my skill qualifies me for that certification but I got that skill writing proprietary software - and reading and debugging the proprietary source other employees had written.

Quite often new technologies, techniques and algorithms are developed in a proprietary fashion but openly published so they can be reproduced by Free Software authors. And even patents eventually expire. The RSA patent used to be so onerous but it is expired now, so it benefits all.

I'm not saying that I feel very good about making a living writing proprietary software, but it is not all bad. What gets me down more is when I have difficult clients or clients who don't pay me on time or don't pay me at all. I've yet to have either an employer or a client tell me I can't write Free Software on my own time, and a couple clients have been enthusiastic about me contributing work I've done for them - with pay - back to the community.

kroah comments on the unpleasant nature of embedded programming, like whacky undocumented hardware and size restrictions, and also says:

Typically you are working at a hardware based company, and have to live with design decisions made by people focused on the bottom line of product cost (remember, in production the firmware is free, and no one will recall the 6 months that it took you to get a beeper not to warble, even though they could have solved it with a 5 cent part in the design up front.)

Personally, I would be really thrilled if I could get paid for six months in order to get a beeper not to warble. I've been in that position before, and I loved it. I miss it terribly.

I invented a graphics compression algorithm once that needed to have a fast decompressor because it would be used interactively to browse CDROMs on the old 68020 Macintoshes. I used to lie awake nights thinking of how I could cram a few more bits out of certain file structures, because those structures would grow if I tried to shrink others - it was a balancing act. I would also be unable to sleep because I was thinking about how I could write the decompressor in assembly code. In the end I got the two innermost loops to fit into the 64 BYTE code cache of the 68020, with the result that my overall decompressor was twice as fast as the C version. I'd do anything to go back to those days.

But instead what I have is constant pressure to get feature after feature implemented. Almost always the client doesn't particularly care if the code is error free, easy to understand, maintainable or fast - they want it NOW, and they don't want to pay very much money to have it developed.

Throughout my career I've devoted a great deal of energy to learning new things. Whether I could learn a new skill has almost always been a top consideration when considering a new job. In the last four years in particular I have devoted a tremendous amount of work to improving the quality of my code. By quality I mean the reliability, comprehensibility, maintainability and performance all together. I never agreed with the position that fast code cannot be maintainable and I feel I've proven that in the products I've written.

And the end result is that I have found I've gotten extremely good at delivering something that the clients would rather I not give them. I've never cared to work fast and cheap - it takes time to write the kind of code I do.

For a long time I've struggled to convince clients that they really need and should want quality code. The best I've been able to do is convince individual programmers at my client companies that this is the case - look at my homepage where I quote the Director of Research at the client I've had for a year saying "Your code is by far the best in our codebase".

The problem is that I can't convince the people who write my paychecks of the need for quality.

I think the only answer is to program for a different market. A market where there can be no arguing that poor quality leads to business failure. I have the theory that in such a market the clients will be happy to pay me for the kind of work I like to do.

My ultimate goal is to be working on the kind of software where poor quality would lead to injury or loss of human life. I personally don't agree that software should be employed at all in life-critical applications, but people use it no matter what I say, and sometimes there really is no choice, like in spacecraft. So I think it would be best for that software to be written by someone with an attitude like mine. (I don't pretend that I have the expertise yet to write mission-critical software, but I think if I learned how I would be good at it.)

A number of people have suggested to me that I would do well to go into writing console video games. I don't know that reliability is prized so highly in gaming but performance certainly is. Console games often have pretty severe constraints so it is important to have both compact and fast code.

I think I would do well at writing video games if I learned how to. I'm particularly interested in learning more about 3-D graphics. (I just bought Edward S. Angel's OpenGL: A Primer.) So in the short term I'm going to at least study games programming, and maybe I'll eventually do some work in it. It's a hard field to break into also.

But in the long run I don't think I want to do games. It's important to me that the code I write be socially useful, at least in some small way.

My very first commercial product was Last Resort for the Mac from Working Software. It's a keystroke logger, but it's not the spyware people (like the FBI) try to plant on your computers these days - it's meant to help people recover the text they've typed in the event of a crash or a power failure.

I've had several authors tell me that Last Resort saved the books or papers they were writing. One of the editors of Mac User magazine told me he caught his girlfriend cheating on him because of Last Resort, and he later featured it prominently in a computer novel he wrote. I have mixed feelings about that but I think it's probably best that he found out about it - he seemed happy that he found out. (He changed the names of all the software he mentioned in his book, but it was pretty obvious what the real products were meant to be. Last Resort more or less saved the world because it recovered some really important text.)

I've gotten comments like that for a lot of the code that I've written - from users. I imagine I could write games that lots of people would enjoy playing, but I don't think I could get that kind of gratification from their feedback because I don't think I would feel that I had actually made their lives any better.

At least not until there is some kind of significant change in what video games do. I don't want to write violent video games. Maybe if I could write Sim games. I was a big fan of the original SimCity.

Of course the real answer is for me to write nothing but high-quality Free Software. That's what I really want to do. My strategy to get there is to write enough products that I will either license for royalties or publish myself that I can stop doing regular work. If I didn't have to write code for a living I would write code for Free.

It's that kind of thing that led me to start LinuxQuality. Read what I have to say on its homepage:

I feel that software must not just be Free, it must be Good.

Well, I have more I'd like to say but this is getting way too long for a diary entry, and it's late. I would enjoy staying up all night but my wife hates to sleep alone.

I value any comments anyone may have.

Good night.

Latest blog entries     Older blog entries

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!