Older blog entries for Pizza (starting at number 118)

Pondering Photo Organizer's Future

Last week I formally released an update to Photo Organizer. It was the first release in two years, and had no new features whatsoever -- just a largeish pile of bugfixes.

I've been thinking about PO's future lately; where I'd like it to go, and to a lesser extent, how to get there. I still don't have a good answer. Pretty much every feature on the wishlist is pretty invasive, and I just don't have the drive to make it happen, because PO has largely done what I've needed.

Most features I've added over the years have been due to my workflow requirements, and I've largely hit a dead end on that front. I've started to experiment with alternative workflow models, but until something gels into a new ideal, PO won't change much.

Since I started on a photo-a-day commitment, my workflow has had to cope with more regular postings, and it's exposed a few cracks in how I used to do things. I'd like to automate some things better, but I'm running into bigger-picture limitations in how I combine PO with my blogging software. For example, none of my "witty" captions have made it back into PO. Fixing this properly means writing a RESTful API to manipulate arbitrary data, which is a huge undertaking.

That said, the biggest driver of change is the massive event-type photography binging like what I got up to last weekend. (Last night I batch imported 44 gigabytes of NEF files; 3,588 images) -- I need a better way of grouping, tagging, and ranking images to manage this sort of thing better. Proper hotkey support would make this a lot easier.

Meanwhile, if the other seeds I've been sowing come to fruition, I'll be working with models on more formal shoots. This will require yet another type of workflow -- less organization, but considerably more post-processing. This means the current versioning system must be beefed up to handle multiple RAW renderings and sidecars. Distributing the final images also exposes weaknesses of PO's permission system.

At the of the day, PO's just a tool that's supporting my needs as a photographer. I'd rather be behind a camera than in front of a computer, especially given that I already spend eight or so hours a day writing software for my day job.

Anyway. As I type this, my PO instance houses some 143,434 images, of which 120,070 are mine. Not counting filesystem overhead, they take up some 1,509,037,063,959 bytes of disk space. Yowza.

In other words, PO's going to be maintained pretty much forever.

Syndicated 2012-12-11 00:29:50 from Solomon Peachy

66/354: IT WORKS!

This deceptively simple printer test page has been literally five years in the making. It is the first time anyone's been able to successfully JustPrint(tm) with the "newer" Canon Dyesub printers under Linux, at least without jumping through some annoying hoops.

But it's not quite done yet.

I still need USB PIDs for the Canon ES3, CP-200(*), CP500, CP520, CP530, CP710(*), CP770(*), and CP790.

The ES20, ES3, CP-10, CP-100(*), CP-200(*), CP-220, CP-300(*), CP500, CP710(*), CP770(*), CP800, CP810, and CP900 models are theoretically supported but, need testing/confirmation.

Finally, the ES40 and CP790 are unsupported until someone comes along and either donates a printer (or cash) or is willing to be a guinea pig.

So, if you have one of these printers and want to use it under Linux (or modern OSX), drop me a line.

* == Already in the mail, so to speak.

Syndicated 2012-11-13 23:57:05 from Solomon Peachy

63/365: They're Breeding...

I'm now up to eight Canon dye-sublimimation photo printers, with two more bundles of joy on their way, all in the name of Free Software. It's interesting; a proprietary printer driver was the spark that spawned the Free Software movement, and I'm continuing the tradition with these.

The Gutenprint stack is far more capable than the original Canon dyseub printer drivers.

On a slightly related note, Anyone out there have a coloriometer I can borrow? I'd like to fully calibrate these things.

Syndicated 2012-11-10 22:56:24 from Solomon Peachy

27 Oct 2012 (updated 7 Nov 2012 at 00:08 UTC) »

Photo printers and libusb, oh my!

A while back, I picked up a Canon SELPHY ES1 dyesub photo printer, only to discover it didn't work properly under Linux, thanks to it needing an intelligent spooler to prevent lockups, and being just different enough from earlier models to require tweaks to gutenprint.

So, in the spirit of Free Software, I patched gutenprint, and wrote the spooler. Over time, the spooler got smarter, supported more printers, achieved sentience, and brought about world peace.

...Until Apple revved OSX's printer subsystem and Canon neglected to release updated drivers for, well, most everything. Since my tool was written to use linux's native usblp support, OSX users were SOL.

Last night, I finished rewiring it to use libusb, and accept input on stdin. This enables it to be used in a CUPS filter chain, and theoretically supports OSX. But that remains to be tested..

...And I finally moved it to version control:


It still needs USB IDs for most of the SELPHY line, but otherwise seems to work fine with my ES1, ES2, and ES30 printers, at least on Linux.

Syndicated 2012-10-27 15:48:49 (Updated 2012-11-07 00:08:27) from Solomon Peachy

Oooh, I can haz missle launcher?

[pizza@stuffed ~]$ lsusb
Bus 001 Device 003: ID 04f9:0039 Brother Industries, Ltd 
Bus 002 Device 002: ID 1941:8021 Dream Link WH1080 Weather Station / USB Missile Launcher
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub

Unfortunately, what I have hooked up to my system is a rather pedestrian weather station, not the more interesting missile launcher. And yes, I know it just fires nerf darts or something like that, but I can still dream about an Anti-Air missile installation controlled via USB, can't I?

On the other hand, I doubt .mil publishes their interface specifications, and reverse engineering a missile launcher is likely rather expensive. Not to mention mildly dangerous..

Syndicated 2012-09-18 17:19:14 from Solomon Peachy

I can haz kernel hacker?

My employer is looking for another software engineer. Not to toot my own horn, but they basically need another me -- a low-level system hacker type. We design and manufacture our own hardware, so there will be a ton of device driver work, board ports, and general kernel hacking work for both Linux and various RTOSes. There will also be a healthy amount of network protocol and general "big picture" system design sort of stuff.

An ideal candidate will be able to code C in their sleep, and have a pretty healthy understanding of the intracacies (and harsh realities) of the IEEE802.11 alphabet soup.

We also are in need of a kickass QA person.

Oh, we're located in Melbourne, Florida, but we're open to long-distance relationships. If you're interested, drop me a line.

Syndicated 2012-06-13 00:22:44 from Solomon Peachy

Good lord, can we get new account creations turned off? The spambots are creating accounts almost as fast as I can flag them as spam...

9 Jul 2011 (updated 19 Feb 2012 at 04:12 UTC) »

The (continuing) joy of photo printers (and free software)

Way back in 2007 I picked up a Canon SELPHY ES1 compact photo printer. After some gnashing of teeth, I hacked support into gutenprint, and all was good. Canon meanwhile announded three successive generations of their ES-series of printers, and I just assumed they worked the same way, since the CP-series were all compatible with each other.

It turned out the ES-series weren't. In fact, every generation was subtly incompatible. (Indeed, the latest generation of CP-series is also incompatible with everything else!). But thanks to a motivatd soul, I was supplied with printer dumps for nearly every dyesub printer Canon created. It piqued my interest, and I started generating dumps for the rest of the ES-series, filling in the support matrix for all paper types. The code was committed to guetenprint about a week ago, but remained untested.

I decided to trolling eBay for the second and third-generation printers (ES2/20 and ES3/30, respectively), and much to my surprise, my lowball bids on two of them won. My "new" ES2 turned up a couple of days ago, and the ES30 should be here tomorrow. I spent late last night and part of tonight rewriting the smart print spooler the ES1 needed, and about 30 minutes ago I finally succeeded with successful prints with the ES1 and ES2.

The ES2 is slightly larger than the ES1, has a better UI and a more efficient paper path. It also adds Memory Stick support. Otherwise, it's identical under the hood, with the same print speed and output quality as the ES1. We'll see what the ES3/30 brings to the table when I eventually end up with one.

But in keeping with tradition, here's the first picture I (successfully) printed with the ES2:

This was shot outside my office a couple of years ago.

I'd like to find a colorimeter so I can get a calibrated printer profile, but as usual, there's an endless line of more important stuff I'd like to get first -- like some grids for my studio flashes. Damnit, photography's an expensive thing to dabble in..

Syndicated 2011-07-09 02:28:13 (Updated 2012-02-19 04:12:21) from Solomon Peachy

dmraid sucks. mdraid sucks less. 3Ware FTW.

About two hours ago I was happily puttering around in a shell on the server that hosts most of my digital existence. Until, after a routine software update, I was suddenly presented with some disturbing sights:

[pizza@stuffed] ~]$ yum  
Bus error
[pizza@stuffed] ~]$ dmesg
-bash: /bin/dmesg: input/output error

It seems that something had gone very, very wrong. I trumped home to find the console full of disk error messages. But that shouldn't have brought the system down; The OS was installed on a pair of drives set up in a RAID1 mirror. There's no excuse for a read failure -- If one drive failed, the other should have picked up the slack and kept the system working.

Well, *should*. I made the mistake of setting up the RAID1 mirror using the "dmraid" tools, which piggyback on the motherboard's "fakeraid" metadata. Unfortunately, it seems that this mode of operation doesn't handle failures worth a damn. And that there's no easy way to migrate from the "dmraid" stuff to the very mature and robust native Linux "dmraid" tools.

The dmraid tools have one major disadvantage though -- they are much more dificult to boot from, and from the motherbord's perspective, if the primary drive fails, the array becomes unbootable. So by using dmraid intead of mdraid, I ended up trading one (minor) failure case for a much more serious one.

How does one work around this quandry? By going for a real hardware RAID controller. The ones with onboard pocessors that do all the heavy lifting. The ones that hide the messy details from the OS. The ones that are designed to JustWork(tm) and NotFail(tm).

The ironic thing is that this server already has a hardware RAID controller in it, a 3Ware 9550SXU-8LP, but it's maxed out with two 4-drive RAID5 arrays totalling about 7TB. This server's predecessors both had 3Ware RAID controllers -- and in my decade-long experience with 3Ware controllers, I've not had a single controller-related failure, ever. They JustWork(tm), handling too-numerous-to-count drive failures cleanly and transparently.

So, I just ordered another 3Ware 9550SXU-4LP controller to plug my OS array into. It's used so I got it for cheapcheapcheap, and like the one I already have it's two generations older than their current top-line models, but it'll be more than good enough for a simple RAID1 array. Migrating the dmraid array over to the new controller will be an interesting proposition even without the flaky drive complicating things, but since downtime is unavoidable thanks to dmraid's failings, I might as well make sure things are done right.

If I hadn't picked up another hardware controller, I'd have migrated to mdraid, and booted off of an USB stick. That solution has worked great at work, and those USB sticks are trivial to back up and replicate in case of failures.

In other news, the tally from this weekend is 5581 RAW photos taking up some 62Gigs of space. Been to busy to post anything, but as my backlog empties out, you'll be seeing more here again.

Syndicated 2011-02-07 18:31:29 from Solomon Peachy

On 802.11

On 802.11:

apenwarr waxed poetic about various aspects of the 802.11 standard, and as someone who knows far more about the specs than any supposedly-sane person ever should, I think I can clarify a few things.

First, if you're trying to get up to speed with it, I suggest you start with the original 802.11-1999 spec. Everything else is built on top of this, and the current 1200 pages of so of the (incomplete!) rollup spec is due to the numerous amendments that generally just add complexity and confusion when you don't know what the core protocol is about.

Second, you have to remember that 802.11 is a *radio* protocol, not a wired protocol, and subsequently the fundamental Physical (PHY) layer is completely different than a wired layer because of this. That said, your two basic questions are actually due to the 802.2/3 heritage that 802.11 builds on. But first your bit about antennae.

Radio waves aren't constrained to nice neat cables. They radiate outwards in all directions. They bounce off of things, and these bounced signals may make it to the receiver too. Since these bounced signals have to travel further, they arrive later than the original signal, causing ghosts some of you may remember from the analog TV days. This phenomena is called multipath interference, and there's a ton of complexity in the receivers to detect and deal with this.

802.11n gets its speedups over 802.11g from three things: Better modulation (65Mbps vs 54Mbps); Wider bandwidth (40MHz channels vs 20MHz, doubling throughput to 130Mbps), and finally supporting multiple simultaneous streams (which takes us up to 600Mbps with four streams, but nothing I've seen supports more than 300Mbps with two streams). The problem is that you can't transmit multiple streams from the same antenna; they'll interfere with each other. That same multipath mess that causes problems before is instead deliberately harnessed -- but to do that, you need need multiple antennae, one for each spatial stream. Similarly, you'll need multiple antennae on the receiver, one more than the number of streams.

Meanwhile. Wired ethernet is not considered "reliable" but compared to wireless, it is bulletproof. Wired ethernet can detect collisions as they happen due to every transmitter sharing the same wire (CSMA/CD), but Wireless transmitters have no way of knowing if the receiver was being locally interfered with or not. This is the reason for adding a positive acknowledgement and retransmissions at such a low level.

Similarly, stations may (and often are) highly mobile, and may drop off of a network at any time. If the station connects to a different access point, how is the rest of the network to know that it's moved? By making association an explicit action, the AP knows to send a notification to the rest of the network to update their MAC address tables. Which brings us to "why can't we join networks simultaneously?" Fundamentally, it's because each radio only has one MAC address. If the station supported using multiple MAC addresses, then it could join multiple networks. There are other factors in play (mainly synchronization/timing; a STA is slaved to the AP's clocks), but that's one of the big ones. Oh, and the 802.11 spec can't just assume everyone's using IP, and there are 802.11 chipsets that support multiple MAC addresses.

Disassociate messages are there to explicitly tell the AP to free up the resources that the STA is using. It's not strictly necessary, but instead a highly useful optimization when you consider the bigger picture of multiple APs servicing the same logical network and that a single AP can only handle so many STAs before they interfere themselves into oblivion or simply run out of resources. (Anyone who's been to tradeshows with public wifi has seen this for themselves). Also keep in mind that the AP can also send out disassociations to force the STA to hand off to a different AP or if the AP has to go away for some reason (such as switching channels due to radar interference). Explicit notifications are always preferable to implicit ones, especially on a highly unreliable medium.

QoS stuff is (unfortunately) here to stay, and provides tangible throughput improvements by adding additional mechanisms to reduce collisions and minimize round trips (and their latencies, the real throughput killer), something that "over-provisioning" simply can't deal with -- remember, you can always add more wires bonded together ad nauseum, but you can't just add more RF spectrum to achieve the same thing. Again, radio, being a shared medium, is completely different. The nodes all have to be smart to not step on each other's toes or the whole house of cards collapses.

802.11 as it stands now is actually pretty well designed; it's complicated because it is trying to solve some very complicated problems. Trying to grok the whole thing at once is migraine-inducing, but if you start from the original 802.11-1999 spec, work your way through the amendments chronologically, and keep in mind its ethernet heritage (and the fundamental differences RF brings over a hardline connection) it'll make more sense more quickly.

Anyway, I'll shut back up now..

Syndicated 2010-11-09 16:44:57 from Solomon Peachy

109 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!