Older blog entries for Pizza (starting at number 115)

27 Oct 2012 (updated 7 Nov 2012 at 00:08 UTC) »

Photo printers and libusb, oh my!

A while back, I picked up a Canon SELPHY ES1 dyesub photo printer, only to discover it didn't work properly under Linux, thanks to it needing an intelligent spooler to prevent lockups, and being just different enough from earlier models to require tweaks to gutenprint.

So, in the spirit of Free Software, I patched gutenprint, and wrote the spooler. Over time, the spooler got smarter, supported more printers, achieved sentience, and brought about world peace.

...Until Apple revved OSX's printer subsystem and Canon neglected to release updated drivers for, well, most everything. Since my tool was written to use linux's native usblp support, OSX users were SOL.

Last night, I finished rewiring it to use libusb, and accept input on stdin. This enables it to be used in a CUPS filter chain, and theoretically supports OSX. But that remains to be tested..

...And I finally moved it to version control:

        git://git.shaftnet.org/selphy_print.git
        http://git.shaftnet.org/git/gitweb.cgi?p=selphy_print.git

It still needs USB IDs for most of the SELPHY line, but otherwise seems to work fine with my ES1, ES2, and ES30 printers, at least on Linux.

Syndicated 2012-10-27 15:48:49 (Updated 2012-11-07 00:08:27) from Solomon Peachy

Oooh, I can haz missle launcher?

[pizza@stuffed ~]$ lsusb
Bus 001 Device 003: ID 04f9:0039 Brother Industries, Ltd 
Bus 002 Device 002: ID 1941:8021 Dream Link WH1080 Weather Station / USB Missile Launcher
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub

Unfortunately, what I have hooked up to my system is a rather pedestrian weather station, not the more interesting missile launcher. And yes, I know it just fires nerf darts or something like that, but I can still dream about an Anti-Air missile installation controlled via USB, can't I?

On the other hand, I doubt .mil publishes their interface specifications, and reverse engineering a missile launcher is likely rather expensive. Not to mention mildly dangerous..

Syndicated 2012-09-18 17:19:14 from Solomon Peachy

I can haz kernel hacker?

My employer is looking for another software engineer. Not to toot my own horn, but they basically need another me -- a low-level system hacker type. We design and manufacture our own hardware, so there will be a ton of device driver work, board ports, and general kernel hacking work for both Linux and various RTOSes. There will also be a healthy amount of network protocol and general "big picture" system design sort of stuff.

An ideal candidate will be able to code C in their sleep, and have a pretty healthy understanding of the intracacies (and harsh realities) of the IEEE802.11 alphabet soup.

We also are in need of a kickass QA person.

Oh, we're located in Melbourne, Florida, but we're open to long-distance relationships. If you're interested, drop me a line.

Syndicated 2012-06-13 00:22:44 from Solomon Peachy

Good lord, can we get new account creations turned off? The spambots are creating accounts almost as fast as I can flag them as spam...

9 Jul 2011 (updated 19 Feb 2012 at 04:12 UTC) »

The (continuing) joy of photo printers (and free software)

Way back in 2007 I picked up a Canon SELPHY ES1 compact photo printer. After some gnashing of teeth, I hacked support into gutenprint, and all was good. Canon meanwhile announded three successive generations of their ES-series of printers, and I just assumed they worked the same way, since the CP-series were all compatible with each other.

It turned out the ES-series weren't. In fact, every generation was subtly incompatible. (Indeed, the latest generation of CP-series is also incompatible with everything else!). But thanks to a motivatd soul, I was supplied with printer dumps for nearly every dyesub printer Canon created. It piqued my interest, and I started generating dumps for the rest of the ES-series, filling in the support matrix for all paper types. The code was committed to guetenprint about a week ago, but remained untested.

I decided to trolling eBay for the second and third-generation printers (ES2/20 and ES3/30, respectively), and much to my surprise, my lowball bids on two of them won. My "new" ES2 turned up a couple of days ago, and the ES30 should be here tomorrow. I spent late last night and part of tonight rewriting the smart print spooler the ES1 needed, and about 30 minutes ago I finally succeeded with successful prints with the ES1 and ES2.

The ES2 is slightly larger than the ES1, has a better UI and a more efficient paper path. It also adds Memory Stick support. Otherwise, it's identical under the hood, with the same print speed and output quality as the ES1. We'll see what the ES3/30 brings to the table when I eventually end up with one.

But in keeping with tradition, here's the first picture I (successfully) printed with the ES2:

This was shot outside my office a couple of years ago.

I'd like to find a colorimeter so I can get a calibrated printer profile, but as usual, there's an endless line of more important stuff I'd like to get first -- like some grids for my studio flashes. Damnit, photography's an expensive thing to dabble in..

Syndicated 2011-07-09 02:28:13 (Updated 2012-02-19 04:12:21) from Solomon Peachy

dmraid sucks. mdraid sucks less. 3Ware FTW.

About two hours ago I was happily puttering around in a shell on the server that hosts most of my digital existence. Until, after a routine software update, I was suddenly presented with some disturbing sights:

[pizza@stuffed] ~]$ yum  
Bus error
[pizza@stuffed] ~]$ dmesg
-bash: /bin/dmesg: input/output error

It seems that something had gone very, very wrong. I trumped home to find the console full of disk error messages. But that shouldn't have brought the system down; The OS was installed on a pair of drives set up in a RAID1 mirror. There's no excuse for a read failure -- If one drive failed, the other should have picked up the slack and kept the system working.

Well, *should*. I made the mistake of setting up the RAID1 mirror using the "dmraid" tools, which piggyback on the motherboard's "fakeraid" metadata. Unfortunately, it seems that this mode of operation doesn't handle failures worth a damn. And that there's no easy way to migrate from the "dmraid" stuff to the very mature and robust native Linux "dmraid" tools.

The dmraid tools have one major disadvantage though -- they are much more dificult to boot from, and from the motherbord's perspective, if the primary drive fails, the array becomes unbootable. So by using dmraid intead of mdraid, I ended up trading one (minor) failure case for a much more serious one.

How does one work around this quandry? By going for a real hardware RAID controller. The ones with onboard pocessors that do all the heavy lifting. The ones that hide the messy details from the OS. The ones that are designed to JustWork(tm) and NotFail(tm).

The ironic thing is that this server already has a hardware RAID controller in it, a 3Ware 9550SXU-8LP, but it's maxed out with two 4-drive RAID5 arrays totalling about 7TB. This server's predecessors both had 3Ware RAID controllers -- and in my decade-long experience with 3Ware controllers, I've not had a single controller-related failure, ever. They JustWork(tm), handling too-numerous-to-count drive failures cleanly and transparently.

So, I just ordered another 3Ware 9550SXU-4LP controller to plug my OS array into. It's used so I got it for cheapcheapcheap, and like the one I already have it's two generations older than their current top-line models, but it'll be more than good enough for a simple RAID1 array. Migrating the dmraid array over to the new controller will be an interesting proposition even without the flaky drive complicating things, but since downtime is unavoidable thanks to dmraid's failings, I might as well make sure things are done right.

If I hadn't picked up another hardware controller, I'd have migrated to mdraid, and booted off of an USB stick. That solution has worked great at work, and those USB sticks are trivial to back up and replicate in case of failures.

In other news, the tally from this weekend is 5581 RAW photos taking up some 62Gigs of space. Been to busy to post anything, but as my backlog empties out, you'll be seeing more here again.

Syndicated 2011-02-07 18:31:29 from Solomon Peachy

On 802.11

On 802.11:

apenwarr waxed poetic about various aspects of the 802.11 standard, and as someone who knows far more about the specs than any supposedly-sane person ever should, I think I can clarify a few things.

First, if you're trying to get up to speed with it, I suggest you start with the original 802.11-1999 spec. Everything else is built on top of this, and the current 1200 pages of so of the (incomplete!) rollup spec is due to the numerous amendments that generally just add complexity and confusion when you don't know what the core protocol is about.

Second, you have to remember that 802.11 is a *radio* protocol, not a wired protocol, and subsequently the fundamental Physical (PHY) layer is completely different than a wired layer because of this. That said, your two basic questions are actually due to the 802.2/3 heritage that 802.11 builds on. But first your bit about antennae.

Radio waves aren't constrained to nice neat cables. They radiate outwards in all directions. They bounce off of things, and these bounced signals may make it to the receiver too. Since these bounced signals have to travel further, they arrive later than the original signal, causing ghosts some of you may remember from the analog TV days. This phenomena is called multipath interference, and there's a ton of complexity in the receivers to detect and deal with this.

802.11n gets its speedups over 802.11g from three things: Better modulation (65Mbps vs 54Mbps); Wider bandwidth (40MHz channels vs 20MHz, doubling throughput to 130Mbps), and finally supporting multiple simultaneous streams (which takes us up to 600Mbps with four streams, but nothing I've seen supports more than 300Mbps with two streams). The problem is that you can't transmit multiple streams from the same antenna; they'll interfere with each other. That same multipath mess that causes problems before is instead deliberately harnessed -- but to do that, you need need multiple antennae, one for each spatial stream. Similarly, you'll need multiple antennae on the receiver, one more than the number of streams.

Meanwhile. Wired ethernet is not considered "reliable" but compared to wireless, it is bulletproof. Wired ethernet can detect collisions as they happen due to every transmitter sharing the same wire (CSMA/CD), but Wireless transmitters have no way of knowing if the receiver was being locally interfered with or not. This is the reason for adding a positive acknowledgement and retransmissions at such a low level.

Similarly, stations may (and often are) highly mobile, and may drop off of a network at any time. If the station connects to a different access point, how is the rest of the network to know that it's moved? By making association an explicit action, the AP knows to send a notification to the rest of the network to update their MAC address tables. Which brings us to "why can't we join networks simultaneously?" Fundamentally, it's because each radio only has one MAC address. If the station supported using multiple MAC addresses, then it could join multiple networks. There are other factors in play (mainly synchronization/timing; a STA is slaved to the AP's clocks), but that's one of the big ones. Oh, and the 802.11 spec can't just assume everyone's using IP, and there are 802.11 chipsets that support multiple MAC addresses.

Disassociate messages are there to explicitly tell the AP to free up the resources that the STA is using. It's not strictly necessary, but instead a highly useful optimization when you consider the bigger picture of multiple APs servicing the same logical network and that a single AP can only handle so many STAs before they interfere themselves into oblivion or simply run out of resources. (Anyone who's been to tradeshows with public wifi has seen this for themselves). Also keep in mind that the AP can also send out disassociations to force the STA to hand off to a different AP or if the AP has to go away for some reason (such as switching channels due to radar interference). Explicit notifications are always preferable to implicit ones, especially on a highly unreliable medium.

QoS stuff is (unfortunately) here to stay, and provides tangible throughput improvements by adding additional mechanisms to reduce collisions and minimize round trips (and their latencies, the real throughput killer), something that "over-provisioning" simply can't deal with -- remember, you can always add more wires bonded together ad nauseum, but you can't just add more RF spectrum to achieve the same thing. Again, radio, being a shared medium, is completely different. The nodes all have to be smart to not step on each other's toes or the whole house of cards collapses.

802.11 as it stands now is actually pretty well designed; it's complicated because it is trying to solve some very complicated problems. Trying to grok the whole thing at once is migraine-inducing, but if you start from the original 802.11-1999 spec, work your way through the amendments chronologically, and keep in mind its ethernet heritage (and the fundamental differences RF brings over a hardline connection) it'll make more sense more quickly.

Anyway, I'll shut back up now..

Syndicated 2010-11-09 16:44:57 from Solomon Peachy

17 Apr 2010 (updated 9 Nov 2010 at 18:09 UTC) »

Asus G50Vt quirks with Linux

Over the months I've owned this laptop, I've run into quite a few little quirks, and have been slowly knocking them out one by one. Tonight I managed to get the last of 'em quashed, and I thoght I'd write everything up in case anyone else finds this stuff useful.

Among the quirks: (and note these are current as of 2.6.32)

Buggy IOMMU. If you enable virtualization in the BIOS, the kernel spews out boatloads of iommu warnings/errors due to some kind of glitch. It's presumably a kernel bug, but until it's fixed, you gotta add 'intel_iommu=off' to the kernel command line.

Keyboard lagging This was particularly annoying, but adding 'i8042.noloop i8042.nopnp' to the kernel command line made quite a difference. It ain't perfect, but it's at least no longer infuriating. It's worth noting that the laggy keyboard also afflicts operation under Vista too -- only I can't do anything about that!

ExpressCard slot not working post-suspend. Due to a kernel bug, the PCIExpress hotplug driver wasn't working unless the ' pciehp.pciehp_force=1' option is added to the kernel command line. Now I can hotplug ExpressCards left and right, yay.

Post-suspend, the ethernet interface not coming up with GigE speeds. Apparently the network chipset somehow decides to stop advertising 1000Mbps support to the switch, so it's not autonegotiated properly. The workaround is pretty simple: As root, run:

'ethtool -s eth0 speed 1000M'
and the hardware re-enables 1000M operation and negotiates with the switch properly.

Headphone jack not working. When something's plugged into the headphone jack, the speakers are muted -- and so is the headphone jack. if you sorta plug it in loosely, you hear both the speakers and the headphone simultaneously. Something's screwy with the HDA Codec routings! This was a particularly annoying fix to solve, and has been broken since the 2.6.28 kernel. There's been a patch rotting in ALSA's bugtracker for more than a year now.

Tonight I got sick of the headphone jack screwiness, and started poking around and discovered the 'hda analyzer' tool; and when poking around with the various routing lines referenced by the patch, I suddenly start hearing output from the headphones. Plugging and unplugging them JustWorks(tm). Apparently the default routing for the headphone jack is incorrect. A bit more digging led me to the hda-verb tool, which can then be invoked with:

hda-verb /dev/snd/hwC0D0 0x21 SET_CONNECT_SEL 0x0d
And voila, everything JustWorks(tm). Yay!

I added a rule to the pm-utils scripts that kicks both ethtool and hda-verb to fix things up after a suspend, and I'm as happy as a clam now.

When combined with the 'asusg50oled' app that does something useful with the little OLED display above the keyboard, this laptop is actually more functional in Linux than it is under Vista -- It won't come out of a suspend properly there -- and I can even directly control the LEDs around the touchpad. oooo.

Other joy I encountered included a slightly buggy ExpressCard CompactFlash adapter. Modern CF cards, basically being full PATA implementations, can operate at UDMA speeds -- current high-end cards can sustain 90MB/s throughput, well in excess of the ~25-ish that the best USB-based readers can accomplish. I picked up a 'SIIG ExpressCard/54 R/W' adapter, which promises to let me run all-out. So, once I got the hotplug problem fixed up, I slapped in my UDMA-capable CF card.. and.. barely managed 20MB/s, actually worsethan my USB reader.

Further investigation showed that the kernel PATA later was saying that the '80-wire detection' was failing, forcing the card to revert to at most UDMA-33 speeds. Apparently the CF adapter wasn't reporting the proper cable status -- CF cards by definition are 40-pin, but the cables are nonexistent so they can operate at full UDMA/133 speeds if they support it. Unfortutaely, SIIG used the same PCI subvendor/model ID as the reference design for the PATA chipset... so there was no way to hack in a kernel quirk to work around this buggy hardware. But not all was lost!

There was a way to force this to be overridden, but only on a per-adapter basis -- and since the ExpressCard is removable, each time a hotplug event happens it's assigned a new adapterid. I was able to work around this by forcing the kernel to default to a "short 40-pin cable" mode, and explicitly setting the hard drive and DVD drive to full sata operation -- adding 'libata.force=short40c,1:sata,2:sata' to my kerel command line. And my "300X" CF cards sustain 40MB/s. Whee!

So what's my cmdline after all of this?

"ro root=/dev/sda7 rhgb pciehp.pciehp_force=1 intel_iommu=off SYSFONT=ter-u16b LANG=en_US.UTF-8 KEYTABLE=us libata.force=short40c,1:sata,2:sata i8042.noloop i8042.nopnp"
Yeah, it's a mouthful..

Syndicated 2010-04-17 04:33:17 (Updated 2010-11-09 18:09:47) from Solomon Peachy

To PostGIS Or Not To PostGIS, that is the question...

Development of Photo Organizer has slowed down lately, in part thanks to RealLife(tm) getting in the way, but mostly due to the remaining feature requests becoming increasingly more invasive. This isn't to say that these features aren't a good idea, but rather that due to PO's craptastic code structure, a seemingly "minor feature" would require a major internal overhaul.

Features like replacing the internal permission model with a finer-grained group-based model. Moving to a real templating engine. Better "social" features. Adding an external RPC API. Adding some sort of caching of search results or other complex queries that involve permission tests. And so on.

One deceptively simple feature request is to integrate PostGIS support. While PO currently extracts GPS data out of images and stores it in the database, it doesn't really do anything useful with that data. Integrating PostGIS support would instantly give PO access to a very powerful geospatial backend that can tie in to all sorts of other spatially-aware systems. There is a near-endless list of upsides, even if PO never uses anything more advanced than spatially-aware searching.

The downsides, however, are doozeys -- From an administration perspective rather than from a code perspecitve. First, due to the level of effort it would take to make PostGIS support optional, we'd have to require it across the board. PostGIS is not part of the standard PostgreSQL distribution, and would consequently make setting up a PO installation more difficult. It would greatly complicate upgrading an existing PO installation to a newer version of PostgreSQL and/or PostGIS, and upgrading to newer PO releases could also get more complex.

So all of that said, PostGIS support would be interesting and cool, but is it necessarily the right direction to take? I know PO is already used by at least one municipality to hold photos relating to their tax rolls, but without a better idea of real-world workflows, I don't know what PO can do to better tie in to the rest of their (or anyone else's) systems.

Meanwhile, regardless of PO's support for PostGIS, more user-visible features like "pull up a google map with locations of this set of photos marked" can be implemented, and now that I have a GPS widget for my camera, I'm actually interested in such things. :)

I get nearly no feedback from PO users; indeed aside from the freshmeat subscriber stats I really have no idea how many folks actually use PO. My best efforts with Google show a few dozen public PO installations, including at least two which the admins have independently translated into Russian. Come on folks, send me patches so all users can benefit from this work!

So, peanut gallery, any thoughts?

Syndicated 2008-10-13 16:46:41 from Solomon Peachy

13 Oct 2008 (updated 18 Nov 2008 at 04:11 UTC) »

To PostGIS Or Not To PostGIS, that is the question...

Development of Photo Organizer has slowed down lately, in part thanks to RealLife(tm) getting in the way, but mostly due to the remaining feature requests becoming increasingly more invasive. This isn't to say that these features aren't a good idea, but rather that due to PO's craptastic code structure, a seemingly "minor feature" would require a major internal overhaul.

Features like replacing the internal permission model with a finer-grained group-based model. Moving to a real templating engine. Better "social" features. Adding an external RPC API. Adding some sort of caching of search results or other complex queries that involve permission tests. And so on.

One deceptively simple feature request is to integrate PostGIS support. While PO currently extracts GPS data out of images and stores it in the database, it doesn't really do anything useful with that data. Integrating PostGIS support would instantly give PO access to a very powerful geospatial backend that can tie in to all sorts of other spatially-aware systems. There is a near-endless list of upsides, even if PO never uses anything more advanced than spatially-aware searching.

The downsides, however, are doozeys -- From an administration perspective rather than from a code perspecitve. First, due to the level of effort it would take to make PostGIS support optional, we'd have to require it across the board. PostGIS is not part of the standard PostgreSQL distribution, and would consequently make setting up a PO installation more difficult. It would greatly complicate upgrading an existing PO installation to a newer version of PostgreSQL and/or PostGIS, and upgrading to newer PO releases could also get more complex.

So all of that said, PostGIS support would be interesting and cool, but is it necessarily the right direction to take? I know PO is already used by at least one municipality to hold photos relating to their tax rolls, but without a better idea of real-world workflows, I don't know what PO can do to better tie in to the rest of their (or anyone else's) systems.

Meanwhile, regardless of PO's support for PostGIS, more user-visible features like "pull up a google map with locations of this set of photos marked" can be implemented, and now that I have a GPS widget for my camera, I'm actually interested in such things. :)

I get nearly no feedback from PO users; indeed aside from the freshmeat subscriber stats I really have no idea how many folks actually use PO. My best efforts with Google show a few dozen public PO installations, including at least two which the admins have independently translated into Russian. Come on folks, send me patches so all users can benefit from this work!

So, peanut gallery, any thoughts?

Syndicated 2008-10-13 15:46:41 (Updated 2008-11-18 04:11:58) from Solomon Peachy

106 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!