etbe is currently certified at Master level.

Name: Russell Coker
Member since: 2001-02-19 14:53:50
Last Login: 2009-02-24 04:55:31

FOAF RDF Share This



I do general Linux programming and sys-admin work. I am mostly known for my work on NSA Security Enhanced Linux.


Recent blog entries by etbe

Syndication: RSS 2.0

Systemd Notes

A few months ago I gave a lecture about systemd for the Linux Users of Victoria. Here are some of my notes reformatted as a blog post:

Scripts in /etc/init.d can still be used, they work the same way as they do under sysvinit for the user. You type the same commands to start and stop daemons.

To get a result similar to changing runlevel use the “systemctl isolate” command. Runlevels were never really supported in Debian (unlike Red Hat where they were used for starting and stopping the X server) so for Debian users there’s no change here.

The command systemctl with no params shows a list of loaded services and highlights failed units.

The command “journalctl -u UNIT-PATTERN” shows journal entries for the unit(s) in question. The pattern uses wildcards not regexs.

The systemd journal includes the stdout and stderr of all daemons. This solves the problem of daemons that don’t log all errors to syslog and leave the sysadmin wondering why they don’t work.

The command “systemctl status UNIT” gives the status and last log entries for the unit in question.

A program can use ioctl(fd, TIOCSTI, …) to push characters into a tty buffer. If the sysadmin runs an untrusted program with the same controlling tty then it can cause the sysadmin shell to run hostile commands. The system call setsid() to create a new terminal session is one solution but managing which daemons can be started with it is difficult. The way that systemd manages start/stop of all daemons solves this. I am glad to be rid of the run_init program we used to use on SE Linux systems to deal with this.

Systemd has a mechanism to ask for passwords for SSL keys and encrypted filesystems etc. There have been problems with that in the past but I think they are all fixed now. While there is some difficulty during development the end result of having one consistent way of managing this will be better than having multiple daemons doing it in different ways.

The commands “systemctl enable” and “systemctl disable” enable/disable daemon start at boot which is easier than the SysVinit alternative of update-rc.d in Debian.

Systemd has built in seat management, which is not more complex than consolekit which it replaces. Consolekit was installed automatically without controversy so I don’t think there should be controversy about systemd replacing consolekit.

Systemd improves performance by parallel start and autofs style fsck.

The command systemd-cgtop shows resource use for cgroups it creates.

The command “systemd-analyze blame” shows what delayed the boot process and
systemd-analyze critical-chain” shows the critical path in boot delays.

Sysremd also has security features such as service private /tmp and restricting service access to directory trees.


For basic use things just work, you don’t need to learn anything new to use systemd.

It provides significant benefits for boot speed and potentially security.

It doesn’t seem more complex than other alternative solutions to the same problems.

Related posts:

  1. systemd – a Replacement for init etc The systemd projecct is an interesting concept for replacing init...
  2. Some Notes on DRBD DRBD is a system for replicating a block device across...
  3. licence for lecture notes While attending LCA it occurred to me that the lecture...

Syndicated 2015-01-12 18:07:00 from etbe - Russell Coker

Conference Suggestions

LCA 2015 is next week so it seems like a good time to offer some suggestions for other delegates based on observations of past LCAs. There’s nothing LCA specific about the advice, but everything is based on events that happened at past LCAs.

Don’t Oppose a Lecture

Question time at the end of a lecture isn’t the time to demonstrate that you oppose everything about the lecture. Discussion time between talks at a mini-conf isn’t a time to demonstrate that you oppose the entire mini-conf. If you think a lecture or mini-conf is entirely wrong then you shouldn’t attend.

The conference organisers decide which lectures and mini-confs are worthy of inclusion and the large number of people who attend the conference are signalling their support for the judgement of the conference organisers. The people who attend the lectures and mini-confs in question want to learn about the topics in question and people who object should be silent. If someone gives a lecture about technology which appears to have a flaw then it might be OK to ask one single question about how that issue is resolved, apart from that the lecture hall is for the lecturer to describe their vision.

The worst example of this was between talks at the Haecksen mini-conf last year when an elderly man tried at great length to convince me that everything about feminism is wrong. I’m not sure to what degree the Haecksen mini-conf is supposed to be a feminist event, but I think it’s quite obviously connected to feminism – which is of course was why he wanted to pull that stunt. After he discovered that I was not going to be convinced and that I wasn’t at all interested in the discussion he went to the front of the room to make a sexist joke and left.

Consider Your Share of Conference Resources

I’ve previously written about the length of conference questions [1]. Question time after a lecture is a resource that is shared among all delegates. Consider whether you are asking more questions than the other delegates and whether the questions are adding benefit to other people. If not then send email to the speaker or talk to them after their lecture.

Note that good questions can add significant value to the experience of most delegates. For example when a lecturer appears to be having difficulty in describing their ideas to the audience then good questions can make a real difference, but it takes significant skill to ask such questions.

Dorm Walls Are Thin

LCA is one of many conferences that is typically held at a university with dorm rooms offered for delegates. Dorm rooms tend to have thinner walls than hotel rooms so it’s good to avoid needless noise at night. If one of your devices is going to make sounds at night please check the volume settings before you start it. At one LCA I was startled at about 2AM but the sound of a very loud porn video from a nearby dorm room, the volume was reduced within a few seconds, but it’s difficult to get to sleep quickly after that sort of surprise.

If you set an alarm then try to avoid waking other people. If you set an early alarm and then just get up then other people will get back to sleep, but pressing “snooze” repeatedly for several hours (as has been done in the past) is anti-social. Generally I think that an alarm should be at a low volume unless it is set for less than an hour before the first lecture – in which case waking people in other dorm rooms might be doing them a favor.

Phones in Lectures

Do I need to write about this? Apparently I do because people keep doing it!

Phones can be easily turned to vibrate mode, most people who I’ve observed taking calls in LCA lectures have managed this but it’s worth noting for those who don’t.

There are very few good reasons for actually taking a call when in a lecture. If the hospital calls to tell you that they have found a matching organ donor then it’s a good reason to take the call, but I can’t think of any other good example.

Many LCA delegates do system administration work and get calls at all times of the day and night when servers have problems. But that isn’t an excuse for having a conversation in the middle of the lecture hall while the lecture is in progress (as has been done). If you press the green button on a phone you can then walk out of the lecture hall before talking, it’s expected that mobile phone calls sometimes have signal problems at the start of the call so no-one is going to be particularly surprised if it takes 10 seconds before you say hello.

As an aside, I think that the requirement for not disturbing other people depends on the number of people who are there to be disturbed. In tutorials there are fewer people and the requirements for avoiding phone calls are less strict. In BoFs the requirements are less strict again. But the above is based on behaviour I’ve witnessed in mini-confs and main lectures.


It is the responsibility of people who consume substances to ensure that their actions don’t affect others. For smokers that means smoking far enough away from lecture halls that it’s possible for other delegates to attend the lecture without breathing in smoke. Don’t smoke in the lecture halls or near the doorways.

Also using an e-cigarette is still smoking, don’t do it in a lecture hall.


Unwanted photography can be harassment. I don’t think there’s a need to ask for permission to photograp people who harass others or break the law. But photographing people who break the social agreement as to what should be done in a lecture probably isn’t. At a previous LCA a man wanted to ask so many questions at a keynote lecture that he had a page of written notes (seriously), that was obviously outside the expected range of behaviour – but probably didn’t justify the many people who photographed him.

A Final Note

I don’t think that LCA is in any way different from other conferences in this regard. Also I don’t think that there’s much that conference organisers can or should do about such things.

Related posts:

  1. A Linux Conference as a Ritual Sociological Images has an interesting post by Jay Livingston PhD...
  2. Suggestions and Thanks One problem with the blog space is that there is...
  3. Length of Conference Questions After LCA last year I wrote about “speaking stacks” and...

Syndicated 2015-01-08 12:02:52 from etbe - Russell Coker


reason=”verification failed; insecure key”

I’ve recently noticed OpenDKIM on systems I run giving the above message when trying to verify a DKIM message from my own domain. According to Google searches this is due to DNSSEC not being enabled. I’m not certain that I really need DNSSEC for this reason (I can probably make DKIM work without it), but the lack of it does decrease the utility of DKIM and DNSSEC is generally a good thing to have.

Client (Recursive) Configuration

The Debian Wiki page about DNSSEC is really good for setting up recursive resolvers [1]. Basically if you install the bind9 package on Debian/Wheezy (current stable) it will work by default. If you have upgraded from an older release then it might not work (IE if you modified the BIND configuration and didn’t allow the upgrade to overwrite your changes). The Debian Wiki page is also quite useful if you aren’t using Debian, most of it is more Linux specific than Debian specific.

dig +short TXT | tail -1

After you have enabled DNSSEC on a recursive resolver the above command should return “Yes, you are using DNSSEC“.

dig +noall +comments

The above command queries a zone that’s deliberately misconfigured, it will fail if DNSSEC is working correctly.

Signing a Zone

Digital Ocean has a reasonable tutorial on signing a zone [2].

dnssec-keygen -a NSEC3RSASHA1 -b 2048 -n ZONE

The above command creates a Zone Signing Key.

dnssec-keygen -f KSK -a NSEC3RSASHA1 -b 4096 -n ZONE

The above command creates a Key Signing Key. This will take a very long time if you don’t have a good entropy source, on my systems it took a couple of days. Run this from screen or tmux.


When you have created the ZSK and KSK you need to add something like the above to your zone file to include the DNSKEY records.


%.signed: %
        dnssec-signzone -A -3 $(shell head -c 100 /dev/random | sha1sum | cut -b 1-16) -k $(shell echo ksk/K$<*.key) -N INCREMENT -o $< -t $< $(shell echo zsk/K$<*.key)
        rndc reload

Every time you change your signed zone you need to create a new signed zone file. Above is the Makefile I’m currently using to generate the signed file. This relies on storing the KSK files in a directory named ksk/ and the ZSK files in a directory named zsk/. Then BIND needs to be configured to use instead of

The Registrar

Every time you sign the zone a file with a name like will be created, it will have the same contents every time which are the DS entries you send to the registrar to have your zone publicly known as being signed.

Many registrars don’t support DNSSEC, if you use such a registrar (as I do) then you need to transfer your zone before you can productively use DNSSEC. Without the DS entries being signed by a registrar and included in the TLD no-one will recognise your signatures on zone data.

ICANN has a list of registrars that support DNSSEC [3]. My next task is to move some of my domains to such registrars, unfortunately they cost more so I probably won’t transfer all my zones. Some of my zones don’t do anything that’s important enough to need DNSSEC.

Related posts:

  1. Time Zones and Remote Servers It’s widely regarded that the best practice is to set...
  2. Dynamic DNS The Problem My SE Linux Play Machine has been down...

Syndicated 2015-01-02 14:08:37 from etbe - Russell Coker


Yesterday I installed the wp-spamshield plugin for WordPress [1]. It blocks automated comment spam systems by using JavaScript and cookies, apparently most spammers can’t handle that. Before I installed it I was getting hundreds of spam comments per day even with the block spam by math plugin enabled. Now I’ve had it running for 24 hours without any spam. The real advantage of this is that now when a legitimate comment gets flagged as spam I’ll notice it, previously I was deleting hundreds or thousands of comments at a time without reading them.

deb wheezy wordpress

The above repository has the wordpress-wp-spamshield package for Debian/Wheezy. I have no immediate plans for uploading it to Debian because the security support for WordPress plugins doesn’t fit in with the Debian model. I am prepared to negotiate about this if someone has good reasons for including it or any of the other WordPress plugins I’ve packages.

My packaging work is under the GPL (of course) so any DD who disagrees with me could just rebuild the package and upload it. Within Debian there is no rule taking another DD’s GPL’d code that they decided not to upload and then uploading it. There is a consensus that such things are not appropriate without permission, but anyone who wishes can take this blog post as permission.

Related posts:

  1. WordPress Plugins I’ve just added the WordPress Minify [1] plugin to my...
  2. Creating WordPress Packages deb wheezy wordpress I maintain Debian packages of a...
  3. WordPress Maintainability For a while I’ve been maintaining my own WordPress packages....

Syndicated 2014-12-23 02:20:40 from etbe - Russell Coker

BTRFS Status Dec 2014

My last problem with BTRFS was in August [1]. BTRFS has been running mostly uneventfully for me for the last 4 months, that’s a good improvement but the fact that 4 months of no problems is noteworthy for something as important as a filesystem is a cause for ongoing concern.

A RAID-1 Array

A week ago I had a minor problem with my home file server, one of the 3TB disks in the BTRFS RAID-1 started giving read errors. That’s not a big deal, I bought a new disk and did a “btrfs replace” operation which was quick and easy. The first annoyance was that the output of “btrfs device stats” reported an error count for the new device, it seems that “btrfs device replace” copies everything from the old disk including the error count. The solution is to use “btrfs device stats -z” to reset the count after replacing a device.

I replaced the 3TB disk with a 4TB disk (with current prices it doesn’t make sense to buy a new 3TB disk). As I was running low on disk space I added a 1TB disk to give it 4TB of RAID-1 capacity, one of the nice features of BTRFS is that a RAID-1 filesystem can support any combination of disks and use them to store 2 copies of every block of data. I started running a btrfs balance to get BTRFS to try and use all the space before learning from the mailing list that I should have done “btrfs filesystem resize” to make it use all the space. So my balance operation had configured the filesystem to configure itself for 2*3TB+1*1TB disks which wasn’t the right configuration when the 4TB disk was fully used. To make it even more annoying the “btrfs filesystem resize” command takes a “devid” not a device name.

I think that when BTRFS is more stable it would be good to have the btrfs utility warn the user about such potential mistakes. When a replacement device is larger than the old one it will be very common to want to use that space. The btrfs utility could easily suggest the most likely “btrfs filesystem resize” to make things easier for the user.

In a disturbing coincidence a few days after replacing the first 3TB disk the other 3TB disk started giving read errors. So I replaced the second 3TB disk with a 4TB disk and removed the 1TB disk to give a 4TB RAID-1 array. This is when it would be handy to have the metadata duplication feature and copies= option of ZFS.

Ctree Corruption

2 weeks ago a basic workstation with a 120G SSD owned by a relative stopped booting, the most significant errors it gave were “BTRFS: log replay required on RO media” and “BTRFS: open_ctree failed”. The solution to this is to run the command “btrfs-zero-log”, but that initially didn’t work. I restored the system from a backup (which was 2 months old) and took the SSD home to work on it. A day later “btrfs-zero-log” worked correctly and I recovered all the data. Note that I didn’t even try mounting the filesystem in question read-write, I mounted it read-only to copy all the data off. While in theory the filesystem should have been OK I didn’t have a need to keep using it at that time (having already wiped the original device and restored from backup) and I don’t have confidence in BTRFS working correctly in that situation.

While it was nice to get all the data back it’s a concern when commands don’t operate consistently.

Debian and BTRFS

I was concerned when the Debian kernel team chose 3.16 as the kernel for Jessie (the next Debian release). Judging by the way development has been going I wasn’t confident that 3.16 would turn out to be stable enough for BTRFS. But 3.16 is working reasonably well on a number of systems so it seems that it’s likely to work well in practice.

But I’m still deploying more ZFS servers.

The Value of Anecdotal Evidence

When evaluating software based on reports from reliable sources (IE most readers will trust me to run systems well and only report genuine bugs) bad reports have a much higher weight than good reports. The fact that I’ve seen kernel 3.16 to work reasonably well on ~6 systems is nice but that doesn’t mean it will work well on thousands of other systems – although it does indicate that it will work well on more systems than some earlier Linux kernels which had common BTRFS failures.

But the annoyances I had with the 3TB array are repeatable and will annoy many other people. The ctree coruption problem MIGHT have been initially caused by a memory error (it’s a desktop machine without ECC RAM) but the recovery process was problematic and other users might expect problems in such situations.

Related posts:

  1. BTRFS Status March 2014 I’m currently using BTRFS on most systems that I can...
  2. BTRFS Status April 2014 Since my blog post about BTRFS in March [1] not...
  3. BTRFS Status July 2014 My last BTRFS status report was in April [1], it...

Syndicated 2014-12-05 08:09:27 from etbe - Russell Coker

1152 older entries...


etbe certified others as follows:

  • etbe certified etbe as Master
  • etbe certified wichert as Master
  • etbe certified cananian as Master
  • etbe certified hpa as Master
  • etbe certified nate as Journeyer
  • etbe certified bcollins as Master
  • etbe certified alan as Master
  • etbe certified JHM as Master
  • etbe certified taj as Master
  • etbe certified espy as Master
  • etbe certified clameter as Master
  • etbe certified davem as Master
  • etbe certified BrucePerens as Master
  • etbe certified esr as Master
  • etbe certified Skud as Journeyer
  • etbe certified branden as Master
  • etbe certified Joy as Master
  • etbe certified cas as Journeyer
  • etbe certified srivasta as Master
  • etbe certified rms as Master
  • etbe certified ajt as Master
  • etbe certified riel as Master
  • etbe certified paul as Journeyer
  • etbe certified mdz as Journeyer
  • etbe certified lupus as Master
  • etbe certified marcel as Journeyer
  • etbe certified Sam as Journeyer
  • etbe certified jaq as Journeyer
  • etbe certified dopey as Journeyer
  • etbe certified joey as Master
  • etbe certified rkrishnan as Journeyer
  • etbe certified Netsnipe as Master
  • etbe certified claviola as Master
  • etbe certified cjwatson as Master
  • etbe certified cmiller as Master
  • etbe certified evo as Master
  • etbe certified baux as Journeyer
  • etbe certified aaronl as Master
  • etbe certified cdub as Master
  • etbe certified kroah as Master
  • etbe certified neurogato as Apprentice
  • etbe certified omnic as Journeyer
  • etbe certified knghtbrd as Master
  • etbe certified hands as Master
  • etbe certified jooon as Journeyer
  • etbe certified zx64 as Journeyer
  • etbe certified slef as Master
  • etbe certified mirwin as Apprentice
  • etbe certified mentifex as Apprentice

Others have certified etbe as follows:

  • etbe certified etbe as Master
  • ajv certified etbe as Master
  • taj certified etbe as Master
  • faye certified etbe as Master
  • paul certified etbe as Master
  • Guillaume certified etbe as Journeyer
  • acme certified etbe as Journeyer
  • Joy certified etbe as Journeyer
  • claviola certified etbe as Master
  • walters certified etbe as Journeyer
  • mdz certified etbe as Journeyer
  • kmcmartin certified etbe as Master
  • srivasta certified etbe as Master
  • neil certified etbe as Master
  • kroah certified etbe as Master
  • cdub certified etbe as Master
  • rkrishnan certified etbe as Master
  • Sam certified etbe as Master
  • jaq certified etbe as Journeyer
  • jooon certified etbe as Master
  • dopey certified etbe as Journeyer
  • omnic certified etbe as Journeyer
  • zx64 certified etbe as Master
  • seeS certified etbe as Master
  • Netsnipe certified etbe as Master
  • cjwatson certified etbe as Master
  • LaForge certified etbe as Master
  • futaris certified etbe as Master
  • timriker certified etbe as Master
  • domi certified etbe as Master
  • byte certified etbe as Master
  • tc certified etbe as Master
  • pasky certified etbe as Journeyer
  • trs80 certified etbe as Master
  • fxn certified etbe as Master
  • joey certified etbe as Journeyer
  • ncm certified etbe as Journeyer
  • dwmw2 certified etbe as Master
  • lkcl certified etbe as Master
  • Pizza certified etbe as Master
  • mattl certified etbe as Journeyer
  • Mmarquee certified etbe as Master
  • slef certified etbe as Master
  • Funklord certified etbe as Master

[ Certification disabled because you're not logged in. ]

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!

Share this page