Standards Implementation Teams

Posted 14 Feb 2002 at 20:31 UTC by whatever Share This

With free access to source code, it is possible for a team of programmers to go through an entire GNU system to add a new feature to all relevant programs.

The context

If there's any adage that everyone in Free Software agrees with, it's "Show me the Source". Any number of documents can be written, but it's not going to happen until there's code.

One of the unique advantages of the Free Software Movement is that everyone has access to the source code without having to work for a company, obtain prior permission or know the right people. This is a strength that closed source businesses cannot replicate.

The problem

The nice thing about standards is that there's so many to choose from. When a standard becomes commonly used, it becomes the de-facto standard for that set of features.

The biggest hurdle facing a new standard is getting enough people interested in the first place. It can be incorporated into an existing product, a demo program can be created or it can be hyped into happening.

The analysis

Free software opens up another possibility. With free access to source code, it is possible for a team of programmers to go through an entire GNU system to add a new feature to all relevant programs.

For example, I would like to be able to query every text editor, from edlin to KDevelop, to find out what file and line number the cursor is sitting on.

Another example. I would like to enter my email address in one place, and every program on the system will know where to get it.

These changes needn't be full implementations to be useful. In most cases, all that's needed is to be able to handle an extension in an intelligent manner instead of crashing. Instead of printing rubbish, a program can display information about the missing feature.

Even if you disagree with these two examples, the point is that these are very easy changes that are only useful if done on a system wide basis. Some changes may involve tens to hundreds of programs.

Think about the programs you code on. How hard would it be to put in a "--bug-report" switch which would print out useful linking and access information? Why don't you bother? Because it's not standard.

What if this switch was on every program on the system? How useful would that be? It wouldn't take very long for a team of 50 people to do this, especially since they'd likely be modifying the programs they already work on anyway.

The proposal

We can form a group of interested programmers and observers to discuss and agree on some of the more useful ideas. The simpler, the better. The more universal, the better.

When enough people agree on a system wide change to make a successful run at the problem, then go ahead - do it!

To make things easier, the new system could be based on Debian to guarantee ready access to all source code, and to ensure there will be no licensing issues. The same is true for BSD or any other open system.

The start

There needs to be enough people interested in these ideas to form teams large enough to modify entire systems.

Fortunately, the required skill level is low as quantity, not quality, counts here. It's easy work so this could be a good place for newbies to get involved. Expert input would be appreciated though!

If you think this is a cool idea and you could be interested, please drop me a line on and I'll keep a tally and post up the results.

Hopeless, posted 14 Feb 2002 at 21:20 UTC by mslicker » (Journeyer)

There is over 30 million lines of code in a distribution such as Redhat. It is both impossible and undesirable to try to impose any kind of structure or consistency onto it.

I think GNU is actually like this by design. In recent public letter Richard Stallman said this:

GNOME, and GNU as a whole, have always had a policy of supporting as many languages as possible. "Whatever languages users wish to use, GNU should support," is the way I generally put it. This is why I eagerly encouraged the development of GNU Fortran, despite having vowed in 1970 never to use that language again myself. GNOME should certainly support .NET programs and C# programs, using DotGNU by preference since that is a GNU package; supporting Mono as an alternative C# implementation would be ok too. I don't think there is any real dispute about this. A free replacement for Visual Basic which works with GNOME would be a major step forward; any capable team that wants to launch this project should please contact

So the general policy of GNU has never been consitency, or any kind of design, but what the "users wish to use". Some people actully like like this, as gives them all kinds of choices. Like they can choose from 50 different languages, 100 different text editors, 5 web browsers, 20 e-mail clients, 6 GUI toolkits, ect.

I personally only need one web browser, one e-mail client, one programming language, one gui. And I would expect all these things to operate as one consistent whole. This is my current direction, and is why I no longer participate with the GNU project. Although I still have a lot of respect for the GNU project, and the emphasis on freedom.

BSD's get this right, posted 14 Feb 2002 at 22:47 UTC by djm » (Master)

The BSD's tend to keep their entire system (kernel+userland) in CVS. Linux distributions tend to pull canonical packages from many locations around the net and patch them.

While the "Linux way" may offer scaling benefits, the keep-everything-in-CVS sure helps from a consistency and security point of view. If you find a problem you can easily grep for it and fix it _everywhere_ it occurs. With packages+patches you have to go through a tedious unpack+apply-your-patches+grep+diff cycle.

Too many standards, posted 15 Feb 2002 at 01:46 UTC by tk » (Observer)

Besides the problem mslicker mentioned, there is the problem that there are just too many standards. This is not a nice thing, contrary to what the article says. Oftentimes there's just no de-facto standard, and the poor hacker is forced to either try to comply with all commonly-used standards -- a tall order, and results in software bloat -- or comply with only one or two standards and risk incompatibility with a sizeable fraction of the world.

Take for example the documentation of the GNU Project itself. Classical Unix uses troff for its documentation. Then came GNU, which decided in its infinite wisdom that troff is `obsolete', and we should all be using info. The result is that we have a huge amount of documentation both ways: while most of the GNU Project uses info, much of what's outside the GNU Project -- including the documentation of Linux system calls -- uses troff. (To complicate matters further, there's also documentation in HTML, text, PostScript, PDF, and what-have-you.)

Going on a quest for the One True Standard in such a situation will not only be time- and energy-consuming, it'll also be quite pointless, in my opinion.

Clarification, posted 15 Feb 2002 at 05:27 UTC by whatever » (Journeyer)

As djm points out, BSD already do something similar for security so that proves this concept is possible.

The phrase "The nice thing about standards is that there's so many to choose from" is sarcastic. tk points out a problem with too many documentation formats. OK, so let's fix it! Form a team, pick a format, and start submitting those patches.

There's only 4000 packages in Debian. Adding a standard feature may be 20 or 30 pre-written lines (in the various languages) or just linking against a library. Gentoo and Rock Linux have automated builds so the BSD advantage is already available.

These are simple consistency cleanups and simple features that are so painfully obvious that EVERYONE can agree on them, but never gets done because there's no point in only doing one or two programs.

Too many documentation standards, posted 15 Feb 2002 at 05:56 UTC by duff » (Journeyer)

Take for example the documentation of the gnu Project itself. Classical Unix uses troff for its documentation. Then came gnu, which decided in its infinite wisdom that troff is `obsolete', and we should all be using info. The result is that we have a huge amount of documentation both ways: while most of the gnu Project uses info, much of what's outside the gnu Project -- including the documentation of Linux system calls -- uses troff. (To complicate matters further, there's also documentation in html, text, PostScript, pdf, and what-have-you.)

I like this example. When reading man pages I find it quite annoying to see "for the real documentation, see info". However, this is where another standard could actually help. The problem cited is because the storage implementation and the presentation medium are tightly coupled. If all of the documentation were stored in some commom format (XML for instance), then "man" would be a tool to present information in a man-like manner and info would be a tool to present the same information in an info-like manner. All someone has to do is come up with the standard format ;-)

Anyway, the original article has an interesting idea and I don't think things like "over 30 million lines of code" would be an impediment. The main problem would be convincing people to incorporate your "standard" into their source base. Sure you can come up with things like "--bug-report", but if Guido doesn't want that in his python, then you've just made a fork for very little gain or you're distributing a system that has patched versions of everything. This seems counter productive to me. But maybe that's just me.

Documentation standard, posted 15 Feb 2002 at 10:13 UTC by jonas » (Master)

The problem isn't that we need another standard for documentation. The problem is that you need to convince everyone to use it. Even if you form a team to translate all existing documentation to one single format, that has increadibly little use if the maintainers of the software doesn't agree to it. The same thing for feature patches; you can't just add a feature and expect that the maintainer will accept it as is. Sometimes, the feature is orthogonal with some other plans, and sometimes the maintainer simply wants it implemented some other way. You need to spend a lot of time with those issues and get involved in the development before you start making such changes.

But then again, it's not completely useless; if 5% of the maintainers accept a change in documentation format, that provides an incentive for more to do the same.

Debian already does this, posted 15 Feb 2002 at 13:24 UTC by dan » (Master)

Look at the Debian Policy Manual. For example

If your package needs to know what hostname to use on (for example) outgoing news and mail messages which are generated locally, you should use the file /etc/mailname. It will contain the portion after the username and @ (at) sign for email addresses of users on the machine (followed by a newline).

It may take more work to get a new requirement into Debian Policy than it does to think of the requirement, but that's a good thing: it stops silly requirements (do you honestly believe that a --bug-report switch on every single binary is either useful or well-specified?) from creeping in without peer review.

GNU standards, posted 15 Feb 2002 at 17:28 UTC by abraham » (Master)

The RMS quote is misleading, GNU has rigid standards for both code and user interface. For example, there already is a GNU standard for getting the bug reporting address from a GNU command line tool.

However, RMS knows that most developers on GNU systems are going to write applications for their own use, not for the GNU systems. Therefore, the GNU system should support whatver languages and tools they need.

It is not so simple...., posted 15 Feb 2002 at 18:59 UTC by Malx » (Journeyer)

[b]--bug-report[/b] is useless for GUI programm. It whould still try to open X display(and possibly crash). Also GUI user uses menu to run them.
I whould suggest to create special external script/software to send reports "bug /bin/sh" , "bug `chich mozilla`" and text files in package with list of protocols (mail or bugzilla etc) and adresses.

/etc/mailname - It could be cool feature, but it is only until you are runing hosting server with 1000 virtual hosts (with their own mail adresses ad domains). If some of them incorporating mail lists or just mailing something from scripts /etc/mailname whould not be good default solution.
Again if you scale this example to corporation with hundreds of computers and same (or not?) mail domain you whould possibly get other problems. Actually $MAILNAME shell variable could be better solution (btw it is a standart to put such settings in ENV vars :)))).

Docs.... I thinks implementing convertors whould be good solution. man2html is well-known cgi script. Why not to make other convertors and every tool (man/info/...) whould search for own files and for files it could convert to it's format. (may be *ANY*2xml, xml2*ANY* whould be optimal implementation)

Personally me whould not call this standarts, but features. You need to collect all lists of features from existing software and make standart , periodically updated list of most popular features of software.
Then programmer could search (and be refered to ;) specific feature and sample implementation (for example - "ability to use HTTP check-for-update", "ability to use HTTP-proxy with Basic user/pass auth" , "command line configuration", "gui config options", "user ~/.config option", "pipes standart data exchange support", "Xdnd", etc.etc.)

Missing the point, posted 15 Feb 2002 at 21:46 UTC by whatever » (Journeyer)

The point isn't whether having email information in one place is a new idea or whether "--bug-report" is a good idea or a bad idea.

The purpose of the article was to point out that free access to source code creates the advantage of being able to perform system wide modifications.

The challenge is to think of useful things that can take advantage of implementing features that run across multiple programs. The article proposes setting up a group to discuss these exact issues.

How can we use this? What low hanging fruit can we knock off? There's not going to be any crappy standards foisted onto anyone, since there's no way we can force people to accept patches. As duff pointed out, this would create forks of every package out there.

However, as jonas pointed out, it's not completely useless since if even only 5% of packages get changed, this creates an incentive for others to start accepting the offered patches.

The discussion group could come up with some ideas for what could be done with across the board changes. We email the ideas through to all the affected maintainers asking "would you accept a patch that does this?", and if enough of them say "yes" then that's a "go ahead".

If they say "no", then nothing's been lost. No sweat.

standards, posted 15 Feb 2002 at 21:55 UTC by mslicker » (Journeyer)

abraham I don't think my quote of RMS was misleading. This does reflect GNU's general policy, and doesn't imlpy GNU doesn't have standards.

However, I think I misunderstood the article. I seems whatever would like a minimal consistency between programs. This seems reasonable, although still a lot of work. Debian looks like the proper outlet for this type of work, as it is a public distribution which already has a policy like this.

tk I agree, that there is too many standards. It makes the barier high for new systems, and hence we are stuck with bloated and stagnate systems. There is approach that I think will work. If applications are driving force of the system (which they should be), then you need only implement standards as the applications need them. Furthermore, you need only implement the parts of standard which the applications use. Chosing the right subset can help alleviate the bloat. And of course with free software, you have choices of what standards to focus on which to ignore, and you can always add more features as you need them.

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!

Share this page