Advogato blog for richdawe

File::ExtAttr 1.08, (Open)Solaris and 2 xattr schemes

Sat, 19 Jul 2008 12:07:48 GMT

I released File::ExtAttr 1.08, which has some changes to make it report errors more consistently ($! should always contain the value of errno now). It also has some build changes, which will hopefully avoid all the CPAN Testers FAIL reports on platforms that don't have the development packages installed for using xattrs (libattr-devel rpm on Linux).

I development this release on various platforms, but the most exciting one was OpenSolaris 2008.05 (AKA Project Indiana). This is so much easier to install and use than Solaris 10. It's much more like using a Linux distro. The OpenSolaris LiveCD is very easy to install, and things just seem to work. It took about 6 steps to get a fully functional development environment for File::ExtAttr.

While developing 1.08 on OpenSolaris, I discovered that it has two separate ways of storing extended metadata: extended file attributes, and extensible system attributes. The extensible system attributes were added as part of supporting CIFS on ZFS. It's still unclear to me why the existing metadata scheme wasn't good enough -- maybe the semantics of the original xattr interface weren't compatible with the use cases?

(The original xattr scheme is that you open the file as though it were a directory, and then the xattrs can be accessed through directory entries. This is clever, but different to the way Linux, *BSD and Mac OS X implement xattrs.)

I'm now not sure which xattr API I should be exposing through File::ExtAttr. I guess it comes down to which one will be more portable across platforms. I think that will be determined by which xattrs are preserved by tar, etc.

Anyow, here's the change log for File::ExtAttr 1.08:

1.08 2008-08-19

     - (richdawe) Add a typemap for usage of "const char *" in the XS.
                  This may help fix the build with Perl 5.6.x or earlier.

     - (richdawe) Remove NetBSD 3.x from list of supported OSes,
                  since File::ExtAttr's test suite will never pass on it.

     - (richdawe) Update Makefile.PL to fail more gracefully when the build
                  pre-requisites are not present. On Linux use
                  Devel::CheckLib to check for libattr. Also exit
                  more gracefully if libattr's headers are not present.

     - (richdawe) OpenBSD isn't supported, so bail gracefully
                  in Makefile.PL on that platform.

     - (richdawe) Make sure that the errno value from any failed
                  system calls is propagated into $! (#32679, #32680).

     - (richdawe) File::ExtAttr no longer generate noisy warnings
                  when an xattr system call fails. All error reporting
                  is now via the function return values and $!.

     - (richdawe) Operations with non-default or non-"user" namespaces
                  will now fail with EOPNOTSUPP instead of ENOATTR
                  on Mac OS X, *BSD and Solaris. This behaviour
                  matches the behaviour on Linux.

     - (richdawe) Added a note to the documentation about Solaris
                  extensible system attributes, which are different
                  to extended file attributes.

rpm: Filtering dependencies differently for different subpackages

Wed, 19 Mar 2008 23:06:49 GMT

Recently I was trying to work out how to filter rpm Requires/Provides dependencies differently for different subpackages. I was trying to produce a subpackage that was the same as another subpackage, but stripping out some library dependencies. Call the one subpackage foo and the other foo-nodeps. (Don't ask why I was trying to do this.)

rpm has a way of hooking the dependency generation, as described in FilteringAutomaticDependencies at the Fedora wiki. This is pretty magical. You disable rpm's internal dependency generation. You can then override the default external dependency generation scripts (if you want). Normally rpm uses find-requires and find-provides in /usr/lib/rpm, or /usr/lib/rpm/redhat on some Red Hat or Red Hat-derived systems. If you do override the scripts, it's likely you'll want to call them and filter their output.

When you define your own dependency generation scripts, they are applied to all subpackages. There is no information passed to the script to indicate which package/subpackage it is being call for. You can pass arbitrary parameters to your custom find-requires/find-provides scripts. But there are no macros that you can use to pass that in as a parameter (%name is always the main package's name -- there's no %subpackage macro AFACIS).

A solution was to pass that information in via the filesystem. In the %install script I'd create a file per package. Something like this:

mkdir -p %{buildroot}/NOTINSTALLED
touch %{buildroot}/NOTINSTALLED/foo.ghost
touch %{buildroot}/NOTINSTALLED/foo-nodeps.ghost

Then in each package's file list I'd put the appropriate file:

%files
...
%ghost /NOTINSTALLED/foo.ghost

%files nodeps
...
%ghost /NOTINSTALLED/foo-nodeps.ghost

The %ghost ensures that the file isn't installed, but is still passed to the find-requires/find-provides scripts. A custom find-requires script can then find out which subpackage it's being called for. Something like this:

cat > .files
if (grep -q -E '^/NOTINSTALLED/foo-nodeps.ghost$' 2>/dev/null); then
  # Filter out dependencies on libfoo
  /usr/lib/rpm/find-requires | grep -v -E '^libfoo.so'
else
  /usr/lib/rpm/find-requires
fi

FileVault

Sat, 12 Jan 2008 09:04:43 GMT

I enabled FileVault home directory encryption on my work Macbook Pro the other day, which is running Mac OS X Tiger (10.4). I was a little bit hesitant about doing this, because of various horror stories about it not working, or performance being terrible. I made sure I had a proper backup, before starting the process.

The 160 GB hard disk was about 40% full before I started. It took 1.5 hours to encrypt my 41 GB home directory, followed by 3.5 hours to securely erase the old unencrypted version.

The performance afterwards seems to be generally the same before. The only exception I've hit so far is with Parallels resizing an expanding disk in a virtual machine, and there the performance is terrible. This makes some kind of sense: There are two disk images being expanded, the Parallels VM disk inside the encrypted home directory, and then the encrypted disk image containing the home directory. Pre-allocating the disk image for the VM helps, because it means no resizing is required at run-time.

Some of my preferences seem to have been lost. I had to make Firefox my default browser again. And I had a hot corner to turn on the sceensaver, which mysteriously stopped working -- I fixed that by setting a different corner to do "show desktop", after which the screensaver hot corner worked again (strange).

Recycling Compact Flourescent Lightbulbs (CFLs)

Sun, 11 Nov 2007 22:12:02 GMT

I read in Scientific American that Compact Flourescent Lightbulbs (CFLs) contain mercury, and some US stores/states have recylcing programmes at stores or kiosks. I didn't realise they contained mercury, and probably would have just thrown them in the bin with my other rubbish. Links:

Compact Fluorescent Light Bulbs at energystar.gov
Compact fluorescent lamp at Wikipedia
Toxic Mercury In CFL Bulbs
lamprecycle.org (not much use here in the UK)

I had a look to see what's on offer in the UK. The "Recycle Now" website doesn't seem to contain any information about how to recycle CFLs. Greenpeace has an article about CFLs, which suggests I should be able to take them back to the retailer. Do I have to prove that I bought the bulb from the retailer in the first place?

File::ExtAttr 1.06, mab2ldif

Sun, 4 Nov 2007 11:15:20 GMT

I released File::ExtAttr 1.06 to fix building on Mac OS X. File::ExtAttr provides an interface to extended file attributes (meta-data) that's consistent across Linux, Mac OS X, *BSD, Solaris.

I also released mab2ldif, which takes a Mork-format address book (e.g.: as used by Thunderbird) and converts it into an LDIF file. You can import the LDIF file into Thunderbird. I wrote this to recover my old Thunderbird address book from an old computer that died. You can actually export the address book from Thunderbird into LDIF, but if Thunderbird won't run, you'll need this tool to get your data back.

richdawe @ 2007-11-04T10:23:00

Sun, 4 Nov 2007 11:15:20 GMT

I saw on Adam Leventhal's blog that Apple ship a DTrace provider for Perl with Leopard:

"Not only did Apple port DTrace, but they've also included a bunch of USDT providers. Perl, Python, Ruby -- they all ship in Leopard with built-in DTrace probes that allow developers to observe function calls, object allocation, and other points of interest from the perspective of that dynamic language."

It's also mentioned on Unix Technology page for Leopard.

I had a quick look to see if Apple had released any patches. I didn't find any -- at some point I should dig around their open source section, to see if it's included in that.

I did find OpenSolaris Bug ID 6355891 asking for Perl support to be added to DTrace.

Women in Tech/Biz

Sun, 28 Oct 2007 19:11:58 GMT

I read a few interesting articles about women and tech/biz recently:

"What I learned about women & business today" at Y Combinator News
The entire "Women in Technology" series at O'Reilly, but specifically:
- "Women Who Risk: Making Women in Technology Visible";
- Comments in response to "To Sir, with Love: How To Get More Women Involved in Open Source" (I found the discussions involving the author much more interesting than the article);
- "Interview with danah boyd" (interesting bit about effects of a 3D environment on men vs. women).

On an unrelated note, I found Judith Donath's talk at Google "Signals, Truth and Design (with an emphasis on information and fashion)" interesting.

The War of Art

Sat, 8 Sep 2007 13:08:07 GMT

I recently read The War of Art by Steven Pressfield. It's a book about overcoming creative blocks and other factors that prevent you from being creative, called Resistance.

I found it to be an entertaining, quick read (~2 hours). I didn't actually find it as useful as I was expecting. I originally bought it to try to get past some blocks, but I overcame them naturally. In fact, at one point I was procrastinating so much that I thought about reading this book, rather than doing anything else -- and that spurred me to stop procrastinating and just do something. One of the key ideas in the book is that by just starting something, you will overcome the blocks.

At a few points the author talked about difficult parts of his life, which seemed to be key to him forming his idea of Resistance (in all its manifestations). I felt there wasn't quite enough detail at these points (and there isn't much more on his website). It would have been useful to understand his journey. That said, it didn't really detract from the presentation of the ideas.

A lot of the ideas in this book are quite simple. But it's easy to lose track of them, when you are being distracted. And the book contains some good quotes. So I think I would turn to this book, if I were in the middle of a creative funk.

postfix config-o-rama

Sat, 1 Sep 2007 21:06:20 GMT

I spent a lot of today finally setting up e-mail for my domain, phekda.org. My goals were:

Set up an SMTP SUBMIT server (running on port 587), so that I can send mail from @phekda.org addresses from anywhere.
Require mail to be submitted over TLS.
Authenticate the client by requiring that the client presents a certificate issued by my private certificate authority (CA). Since I'm only going to issue certificates to people/machines I trust, possession of a certificate is implicit authentication.

TinyCA

I used TinyCA2 to set up my own personal CA. It's really easy to use. I created a CA for phekda.org. I also created a "bad" CA for testing that my postfix box would only accept certificates issued for phekda.org.

Here's what I generated in total:

CA cert for phekda.org
Server cert for mail.phekda.org, signed by CA phekda.org
Client cert for my desktop machine, signed by CA phekda.org
CA cert for bad.ca
Client cert for my desktop machine, signed by CA bad.ca (for testing)

Tip: You can generate password-free keys with TinyCA2. To do this you create the key as normal, specifying the password. When you export the key into a PEM file, you can choose to export without the password.

Server-side postfix configuration

My server is running postfix 2.3.8 on Debian 4.0. The server-side config was split into two halves: general TLS configuration in main.cf, and the config to turn on an SMTP daemon on port 587 with TLS enabled.

Here's the config I added to main.cf:

smtpd_tls_req_ccert = yes
smtpd_tls_session_cache_database = sdbm:/etc/postfix/smtpd_scache

smtpd_tls_CAfile = /etc/postfix/CAcert.pem
smtpd_tls_cert_file = /etc/postfix/server-cert.pem
smtpd_tls_key_file = /etc/postfix/server-key.pem

# Log TLS info, in logs and headers.
smtpd_tls_loglevel = 2
smtpd_tls_receivedheader = yes

Note that these entries in main.cf don't actually enable TLS. smtpd_tls_req_ccert requires SMTP clients to use STARTTLS, when TLS is enabled. The smtpd_tls_*file entries set up everything that's needed on the server-side for TLS encryption. I turned on the last couple of options for debugging purposes.

Here's the line I added to master.cf, split over multiple lines for clarity. You won't need the backslashes, when you recombine them into one line.

587       inet  n       -       n       -       -       smtpd \
-o smtpd_enforce_tls=yes \
-o smtpd_tls_req_ccert=yes \
-o smtpd_recipient_restrictions= \
  permit_mynetworks, \
  permit_tls_all_clientcerts, \
  reject_unauth_destination

smtpd_recipient_restrictions allows clients with authenticated certificates to relay, in addition to local users. Although I'm not sure why a local user would relay through port 587.

Server-side testing

I tested this using OpenSSL's s_client, to set up a client SMTP session using the client certificates I generated with TinyCA2. You fire up openssl s_client with appropriate options, then enter SMTP commands as normal, e.g.:

ehlo fred
mail from:<me@my.domain.example>
rcpt to:<someone@somewhere.else.example>
data
Subject: just a test

.

You need to go all the way, to check that the message can actually be delivered.

Connection should be accepted, because the client is using a certificate issued by the CA for phekda.org:

openssl s_client -connect mail.phekda.org:587 -starttls smtp \
  -CAfile phekda.org-cacert.pem \
  -key katrina.phekda.gotadsl.co.uk-key.pem \
  -cert katrina.phekda.gotadsl.co.uk-cert.pem

Connection should not be accepted, because the client is using a certificate not issued by the CA for phekda.org:

openssl s_client -connect mail.phekda.org:587 -starttls smtp \
  -CAfile phekda.org-cacert.pem \
  -key mail.bad.ca-key.pem \
  -cert mail.bad.ca-cert.pem

And a slight variation:

openssl s_client -connect mail.phekda.org:587 -starttls smtp \
  -CAfile bad.ca-cacert.pem \
  -key mail.bad.ca-key.pem \
  -cert mail.bad.ca-cert.pem

Tip: One thing to beware of is that OpenSSL will do a TLS renegotiation if you use "RCPT TO", so use "rcpt to" instead.

Client-side postfix configuration

I have several e-mail accounts. I want to keep sending from my old domain @phekda.gotadsl.co.uk, but I also want to be able to send from @phekda.org. These messages would be sent via the same postfix server running on my desktop machine.

Before making the changes, all my mail was smart-hosted through my ISP's mail server -- i.e.: all my mail went through my ISP's mail server. Afterwards, my @phekda.org was routed over TLS to mail.phekda.org on port 587, and the rest of the mail was smart-hosted.

To achieve what I wanted, I set up sender-based routing (SBR). Normally mail is routed by recipient address -- SBR overrides the recipient-based routing. Configuring sender-based routing was the hardest part to achieve, because postfix's documentation of SBR and its sender_dependent_relayhost_maps configuration format is a little, uh, brief. Fortunately the postfix source code is readable, and I figured it out from that.

My desktop box is running postfix 2.4.3 on Fedora 7. The client-side postfix config is split into three parts: routing and TLS configuration in main.cf; sender-based routing (SBR) map file, sender_dependent_relayhost; TLS policy map file, smtp_tls_policy.

Firstly, here's the configuration in main.cf:

# Smart-host via Nildram...
relayhost = [smtp.gotadsl.co.uk]

# ...except for certain senders, who we relay through other boxes.
sender_dependent_relayhost_maps = hash:/etc/postfix/sender_dependent_relayhost

# TLS configuration for sending mail to phekda.org
smtp_tls_CAfile = /etc/postfix/CAcert.pem
smtp_tls_cert_file = /etc/postfix/client-cert.pem
smtp_tls_key_file = /etc/postfix/client-key.pem

smtp_tls_loglevel = 1

smtp_tls_policy_maps = hash:/etc/postfix/smtp_tls_policy

Here is /etc/postfix/sender_dependent_relayhost_maps:

#
# Regenerate using:
#   postmap hash:sender_dependent_relayhost < sender_dependent_relayhost
#

# phekda.org sender should be submitted to mail.phekda.org.
@phekda.org	[mail.phekda.org]:587

It wasn't clear how I could configure all phekda.org subdomains to be routed in the same way. It looks like I would have to specify them all manually. Any domains not configured in this file are routed using the normal mechanisms, which in this case ends up being the smarthost specified by relayhost.

Here is /etc/postfix/smtp_tls_policy:

#
# Regenerate using:
#   postmap hash:smtp_tls_policy < smtp_tls_policy
#

phekda.org	secure
[mail.phekda.org]:587	secure

These configuration files need building into .db files before postfix can use them -- this is done using postmap. I wrote a simple Makefile to automate that.

Client-side testing

I tested sending to my gmail account using @phekda.gotadsl.co.uk and @phekda.org addresses. I did this using plain ol' telnet. From the postfix log in /var/log/maillog, I could see where the messages were being routed to. E.g.:

Sep  1 20:30:38 katrina postfix/smtp[27281]: 3225BD: to=<richdawe@gmail.com>,
relay=smtp.gotadsl.co.uk[195.112.4.54]:25, delay=8.6, delays=8.2/0.19/0.11/0.09,
dsn=2.0.0, status=sent (250 Ok: queued as 3501A2BAE63)

Sep  1 20:51:31 katrina postfix/smtp[27374]: 9E258D: to=<richdawe@gmail.com>,
relay=mail.phekda.org[80.68.89.241]:587, delay=25, delays=24/0.13/1.1/0.15,
dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 0E501803B)

It was easy to see when the config was broken.

Perl and DTrace

Fri, 20 Jul 2007 12:06:59 GMT

I finally got DTrace working on Perl, as described in Alan Burlison's blog post on how to DTrace Perl. I have a patch to add DTrace support to Perl, which includes some instructions and example scripts.

I'm giving a talk on DTrace and Perl next Wednesday evening at Birmingham Perl Mongers.