Older blog entries for dreier (starting at number 39)

Signs you may not be dealing with a straight shooter

A play in two scenes.

DRAMATIS PERSONAE

  • X - a senior manager
  • X’s admin - keeper of X’s schedule
  • Y - a hard-working developer

SCENE I

setting: X’s admin’s desk

X’s admin: Hi, Y.

Y: Hi.  I’d like to get some time with X when there’s an opening.

X’s admin: How about next Tuesday at 1?

Y: Perfect.  Please put me on the schedule then.

SCENE II

setting: X’s office, next Tuesday at 1

X: Hi, Y.

Y: Hi, good to see you.

X: Thanks for coming by.  We haven’t talked for a while and I just wanted to touch base.

Y: …

Syndicated 2008-12-01 06:09:42 from Roland's Blog

On over-engineering

I’ve been trying to get a udev rule added to Ubuntu so that /dev/infiniband/rdma_cm is owned by group “rdma” instead of by root, so that unprivileged user applications can be given permission to use it by adding the user to the group rdma.  This matches the practice in the Debian udev rules and is a simple way to allow unprivileged use of RDMA while still giving the administrator some control over who exactly uses it.

I created a patch to the Ubuntu librdmacm package containing the appropriate rule and opened a Launchpad bug report requesting that it be applied.  After two months of waiting, I got a response that basically said, “no, we don’t want to do that.”  After another month of asking, I finally found out what solution Ubuntu would rather have:

Access to system devices is provided through the HAL or DeviceKit interface. Permission to access is managed through the PolicyKit layer, where the D-Bus system bus service providing the device access negotiates privilege with the application requesting it.

Because of course, rather than having an application simply open a special device node, mediated by standard Unix permissions, we’d rather have to run a daemon (bonus points for using DBus activation, I guess) and have applications ask that daemon to open the node for them.  More work to implement, harder to administer, less reliable for users — everyone wins!

Sigh….

Syndicated 2008-11-19 17:33:51 from Roland's Blog

Free Software Syndrome

While thinking about the discussion that my recent series of posts about the career benefits of contributing to the open source community, I started wondering about whether this could actually have some effect on how software gets designed.  We’ve all heard of “Not Invented Here Syndrome,” where developers decide to reimplement something, even when the technically better way to go would be to reuse some existing code.

I think there is also a small but growing tendency towards a “Free Software Syndrome,” where developers push management to release something as open source, not necessarily to get the benefits of community testing and review, more developers, or anything like that, but simply so that the developers can be open source developers.

Syndicated 2008-11-05 19:28:38 from Roland's Blog

The enemy within

I saw several interesting responses to my last post.  I liked Val’s succint way of putting the career impact of free software work:

If you are even moderately successful in open source, you will probably end up “employed for life.”

I’m not sure I would go that far, since someone early in their career now might be working for 30, 40 or 50 more years, and it’s hard to say how much the artilects will care about open source after they’ve converted the entire mass of the solar system into computronium.  But certainly as long as the software industry looks the way it does now, developers who are able, both technically and socially, to work in the open source community are going to be in high demand.  (By the way, Val: two ‘e’s in “Dreier” :razz: — the native English-speaking brain seems hard-wired to misspell my last name)

There were also a few comments to my blog post from people who would like to do more in the open source community, but who are prevented by their current employer.  I understand that no job is ideal, and that people often can’t or won’t change jobs for many reasons — having a great commute, a gazillion dollars in stock still to vest, gambling debts to pay or any of a million other things might be reasons to stay in a particular position.

However, I think it’s important for people in those less than ideal positions to view this for what it is — these employers are taking money out of your pockets and limiting your future earnings just as surely as if they forbid you from other professional development such as learning a new programming language.  I’ve also found that many (although surely not all) software developers are too passive about accepting issues like this.

You need to keep in mind that you are responsible for getting the best for yourself — your interests are not your employers interests, and your employer is surely not going to put your interests first, so you better put your own interests first!  Of course this means that when negotiating things (and you are negotiating from your first job interview to the end of your exit interview), you need to explain why what you want is really in your employer’s interest.

There are many ways you can explain why contribution to open source is in an employer’s interest too, from the direct benefits: “This code has no real relevance to our business, but if we release it and get it upstream, other people will help us maintain it, which saves us money and time in the long term,” to slightly less concrete: “Contributing to free software is good marketing, and it’s going to help us sell as well as recruit and retain good developers,” to the personal: “I know we were told bonuses will be down this year, but something that you can do for me that doesn’t cost anything in your budget is let me contribute this code, which we already agreed the company has no interest in.”

Syndicated 2008-11-05 19:20:27 from Roland's Blog

Quit Today

I’m a slow blogger.  I’ve been meaning to post some thoughts about Greg’s [in]famous LPC keynote for a while now, but it’s taken me nearly two months to get to it.  I’ll start off by saying the same thing that I told Greg in person: I don’t think it was an appropriate setting for for Greg to single out Canonical for criticism.  It doesn’t matter who started it, it doesn’t matter what the merits of a particular argument are, and it doesn’t make sense for Greg to say he was not speaking as a Novell employee, since he is a Novell employee.  But I don’t really want to get drawn into that debate.

The slide that really stuck with me from Greg’s talk is the one from the conclusion that says, “Developers who are not allowed to contribute to Linux should change jobs.”  In the text for the talk, Greg writes, “The solution, quit and go work for one of the companies that allow you to do this!”  And I have to agree with this advice, because I think contributing to free software is in the rational self-interest of nearly any software developer.

When I started in this game about a decade ago, I was a typical Silicon Valley “senior software engineer” type: bright guy, knows how to code, decent resume.  I had a circle of people who I had worked with who knew me, but if I went for a job interview, the people interviewing me were usually meeting me for the first time.  But I was fortunate enough to end up in a job where I was working on open source InfiniBand drivers for Linux, and ended up becoming the Linux kernel InfiniBand/RDMA maintainer.  I should mention that it wasn’t a question of being “allowed” to contribute to Linux — I knew that InfiniBand needed open-source Linux support, and I didn’t listen to anyone who said “no.”

It has been great fun and very rewarding to build the Linux InfiniBand stack, but I just want to focus on the venal career side of this.  And the point is that tons of people know me now.  They know what I can do in terms of kernel coding, and maybe more importantly, they can see how I do it, how I respond to bug reports and how I handle the techno-diplomacy of collaborating on mailing ists.  And this has had a definite effect on my career.  I’m not just YASSE (yet another senior software engineer); I get calls from people I’ve never met offering me really interesting and very senior jobs.  And when hiring, I am certainly much more comfortable with candidates who have a visible history of contribution to open source, simply because I can see both their technical and social approach to development.

I do think that the contributions that have this value to individuals can be beyond Greg’s kernel/gcc/binutils/glibc/X view of the “Linux ecosystem.”  If people have done substantial work on, say, bzr or the Ubuntu installer, that’s still something I can go look at when I’m thinking about hiring them.  Of course, making contributions to a highly visible project carries more weight than contributing to a less visible project, but on the other hand, maybe there’s more room to shine in a project with a smaller community of developers.

Finally, Greg’s advice to “quit” made me think of a book from a few years ago, Die Broke that has as one of its main pieces of advice to “quit today.”  However, this advice is just a provocative way of saying that workers should be conscious of the fact that their employer probably has little to no real loyalty to employees, and so individual workers should focus on their own best interests, rather than what might be best for their employer.  And I think that metaphorical view applies just as well to Greg’s advice to quit: if you feel that you can’t contribute to open source in your present job, what’s stopping you?  Do you really need to change employers to start contributing, or can you just tweak your current job?  What will happen if you tell your manager, “Open source is good for our business for reasons X, Y and Z, and also it’s important to me for my career development, so can we come up with a way for me to start contributing?”

Syndicated 2008-10-29 17:54:14 from Roland's Blog

It takes all kinds

Seen at the library: a dude with a stack of Cathy books on the table in front of him, barely controlling his laughter as he reads My Granddaughter Has Fleas!!

Syndicated 2008-05-20 22:30:02 from Roland's Blog

Hardcore

I just saw a dude order a 4-shot red eye.

Syndicated 2008-04-15 21:21:48 from Roland's Blog

Lies, d… oh, forget it

I noticed the recent blog post “Cisco Set to Dominate Linux Market?” (which lwn.net also linked), but the part that caught my eye was:

According to a recent Linux Foundation study, Cisco is already contributing to Linux and currently represents 0.5 percent of changes (which is a good number). I would expect that with the AXP in the market, Cisco’s contribution rate will go up.

Now, I don’t work on AXP or anything related to ISRs, so I have no idea what those groups plan to do with respect to Linux, but it was somewhat amusing to see the Linux Foundation report cited to show how much work CIsco does on the kernel.

This isn’t the first time I’ve seen this study cited to show what a kernel development powerhouse Cisco is.  In the report, Cisco is credited with 442 commits to the kernel; however, more than 400 of of those commits are mine, and about 30 are Don Fry maintaining the pcnet32 net driver.  So if you take away my work on InfiniBand/RDMA, Cisco’s contributions to the Linux kernel are pretty minimal.

I’m not sure if I have much of a point except that I wish we really did have more than one or two isolated developers at Cisco really engaged with the upstream kernel.

Syndicated 2008-04-12 03:45:39 from Roland's Blog

He who divides and shares is left with the best share

I’ve been talking to a lot of people about the “iWARP port sharing problem” lately, so I thought it might be a good idea to write a quick summary to point at and bring new people up to speed without constantly repeating myself.

To start with, iWARP is an RDMA (remote direct memory access) protocol that runs over TCP (or conceivably SCTP or any other stream protocol). It was defined by the IETF rddp working group, and the standard is in RFC 5040 and later RFCs. So what’s so great about RDMA?

The rationale for RDMA is laid out in great detail in RFC 4297, but the basic idea is that allowing network messages to carry information about where they should be received and allowing the NIC to place the data directly in that buffer allows fundamentally better performance.

To take a concrete example, think of iSCSI: an initiator sends a bunch of SCSI commands to a target (probably queuing up multiple commands), and the target processes the commands, possibly out of order, and returns the responses to the initiator. Without RDMA (or at least, without “direct data placement,” which is pretty equivalent to RDMA), for each read that the initiator does, it has to receive the data from the target, look at which command the data corresponds to, and copy it into the buffer that the SCSI midlayer wants it in. With RDMA and the “iSCSI Extensions for RDMA” (iSER, which is RFC 5046), the target can send the data in response to a read command and have it placed directly in the receive buffer on the initiator, which saves the copy and uses 3x less memory bandwidth (which is huge if the data is running at 10Gb/sec). In the SCSI world, this is nothing particularly exciting: pretty much every Fibre Channel HBA in the world already does the equivalent thing. What’s cool about iWARP is that it allows similar optimizations for NFS (the IETF nfsv4 working group is defining a standard for NFS/RDMA, and kernel 2.6.24-rc1 already has the client side of this draft protocol merged) as well as other applications that we haven’t thought of yet.

The way that iWARP is implemented is that RDMA NICs handle the full iWARP protocol including TCP in hardware — yes, the dreaded “TCP offload engine.” This is crucial to the performance: if the network data isn’t processed to the point of knowing where to put it on the NIC’s side of the PCI bus, then the memory bandwidth savings of copy avoidance is lost. So while one can imagine an iWARP implementation with stateless NIC hardware using some super-fancy header splitting and chipset DMA engine tricks, it’s not clear that it will perform as well as current iWARP NICs do.

Now, in addition to handling TCP connections, iWARP NICs also have to act like normal NICs so that they can handle normal network traffic such as ARPs, pings or ssh logins. What this means is that some packets are received normally and passed up the standard network stack, while other packets that belong to iWARP connections are consumed by the NIC.

This is what leads to the “port sharing problem.” One application might do a normal bind() to accept TCP connections on port X. It might even let the kernel choose a port number for it. Then another application (possibly even the same application) does an iWARP bind and tells the iWARP NIC to accept TCP connections on the same port X. This might happen because two different applications do the bind and have no way of coordinating with each other, or it might happen because one application just passes 0 in the sin_port field of its bind requests, and the kernel chooses the same port for both the normal and iWARP bind(). Whatever the reason, the end result is not good: the NIC and the network stack are left fighting for the same packets, and someone has to lose.

The reason this is an issue is because the kernel’s network stack and iWARP stack have completely separate port allocators, so there is no way for applications to prevent port collisions from happening. The obvious solution is to have normal TCP and iWARP port numbers allocated from the same space.

Unfortunately, the Linux networking developers are not too interested in cooperating on this. It seems that some people have just decided that anyone who wants to use iWARP is wrong to want that (no matter how much better than the alternatives it is for that user’s app) and will just reflexively reject anything iWARP-related without trying to engage in constructive discussion. (Given that attitude, it’s rather ironic when the same people preach about open-mindedness and “thinking outside the box,” but let’s not get sidetracked…)

Given the current deadlock, the advice I’ve been giving to the various iWARP NIC companies is just to sell a lot of iWARP NICs and make the problem so big that we’re forced to find a solution. I don’t see any other way to force people to work together.

Syndicated 2007-11-06 19:15:21 from Roland's Blog

30 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!