Older blog entries for Pizza (starting at number 107)

To PostGIS Or Not To PostGIS, that is the question...

Development of Photo Organizer has slowed down lately, in part thanks to RealLife(tm) getting in the way, but mostly due to the remaining feature requests becoming increasingly more invasive. This isn't to say that these features aren't a good idea, but rather that due to PO's craptastic code structure, a seemingly "minor feature" would require a major internal overhaul.

Features like replacing the internal permission model with a finer-grained group-based model. Moving to a real templating engine. Better "social" features. Adding an external RPC API. Adding some sort of caching of search results or other complex queries that involve permission tests. And so on.

One deceptively simple feature request is to integrate PostGIS support. While PO currently extracts GPS data out of images and stores it in the database, it doesn't really do anything useful with that data. Integrating PostGIS support would instantly give PO access to a very powerful geospatial backend that can tie in to all sorts of other spatially-aware systems. There is a near-endless list of upsides, even if PO never uses anything more advanced than spatially-aware searching.

The downsides, however, are doozeys -- From an administration perspective rather than from a code perspecitve. First, due to the level of effort it would take to make PostGIS support optional, we'd have to require it across the board. PostGIS is not part of the standard PostgreSQL distribution, and would consequently make setting up a PO installation more difficult. It would greatly complicate upgrading an existing PO installation to a newer version of PostgreSQL and/or PostGIS, and upgrading to newer PO releases could also get more complex.

So all of that said, PostGIS support would be interesting and cool, but is it necessarily the right direction to take? I know PO is already used by at least one municipality to hold photos relating to their tax rolls, but without a better idea of real-world workflows, I don't know what PO can do to better tie in to the rest of their (or anyone else's) systems.

Meanwhile, regardless of PO's support for PostGIS, more user-visible features like "pull up a google map with locations of this set of photos marked" can be implemented, and now that I have a GPS widget for my camera, I'm actually interested in such things. :)

I get nearly no feedback from PO users; indeed aside from the freshmeat subscriber stats I really have no idea how many folks actually use PO. My best efforts with Google show a few dozen public PO installations, including at least two which the admins have independently translated into Russian. Come on folks, send me patches so all users can benefit from this work!

So, peanut gallery, any thoughts?

Syndicated 2008-10-13 16:46:41 from Solomon Peachy

13 Oct 2008 (updated 18 Nov 2008 at 04:11 UTC) »

To PostGIS Or Not To PostGIS, that is the question...

Development of Photo Organizer has slowed down lately, in part thanks to RealLife(tm) getting in the way, but mostly due to the remaining feature requests becoming increasingly more invasive. This isn't to say that these features aren't a good idea, but rather that due to PO's craptastic code structure, a seemingly "minor feature" would require a major internal overhaul.

Features like replacing the internal permission model with a finer-grained group-based model. Moving to a real templating engine. Better "social" features. Adding an external RPC API. Adding some sort of caching of search results or other complex queries that involve permission tests. And so on.

One deceptively simple feature request is to integrate PostGIS support. While PO currently extracts GPS data out of images and stores it in the database, it doesn't really do anything useful with that data. Integrating PostGIS support would instantly give PO access to a very powerful geospatial backend that can tie in to all sorts of other spatially-aware systems. There is a near-endless list of upsides, even if PO never uses anything more advanced than spatially-aware searching.

The downsides, however, are doozeys -- From an administration perspective rather than from a code perspecitve. First, due to the level of effort it would take to make PostGIS support optional, we'd have to require it across the board. PostGIS is not part of the standard PostgreSQL distribution, and would consequently make setting up a PO installation more difficult. It would greatly complicate upgrading an existing PO installation to a newer version of PostgreSQL and/or PostGIS, and upgrading to newer PO releases could also get more complex.

So all of that said, PostGIS support would be interesting and cool, but is it necessarily the right direction to take? I know PO is already used by at least one municipality to hold photos relating to their tax rolls, but without a better idea of real-world workflows, I don't know what PO can do to better tie in to the rest of their (or anyone else's) systems.

Meanwhile, regardless of PO's support for PostGIS, more user-visible features like "pull up a google map with locations of this set of photos marked" can be implemented, and now that I have a GPS widget for my camera, I'm actually interested in such things. :)

I get nearly no feedback from PO users; indeed aside from the freshmeat subscriber stats I really have no idea how many folks actually use PO. My best efforts with Google show a few dozen public PO installations, including at least two which the admins have independently translated into Russian. Come on folks, send me patches so all users can benefit from this work!

So, peanut gallery, any thoughts?

Syndicated 2008-10-13 15:46:41 (Updated 2008-11-18 04:11:58) from Solomon Peachy

Photo Organizer 2.36 is (finally) out

It's been stuck in -rc status for four months. Much less feedback this time around, which can be attributed to less interest, or perhaps the code's been more robust this time around. We'll see.

There are many more user-visible changes than usual this time around, ihcluding a nice dark theme, pretty URLs, and per-folder/album thumbnails. Oh, and a 40x speed improvement on a hot-path sql query. Yikes.

Each release has made PO's internals less obnoxious and easier to change, but I've hit another brick wall and the next set of internal improvements will be pretty invasive, with no real user-visible benefit.

Unfortunately, development has slowed down considerably lately, in part due to RealLife(tm).. but as always, it's nice to get feedback.

I also just switched PO over to using git. Due to differences in the usage model (from svn), there was no easy way to migrate the old history in the same repo and still continue using git's best pracices. C'est la vie.

Syndicated 2008-08-18 00:41:35 from Solomon Peachy

Photo Organizer 2.35

Yeah, Photo Organizer 2.35 came out two weeks ago, but I'd figure I should toot my own horn a little bit.

A lot of work went into making client/event management more, well, manageable. Multi-day events and the ability to directly tie clients to events tie into date-based searching to make it easy to find out just what you took for any given point in time.

Also new is pluggable authentication, two-step registration, sortable folder/album listings, much (much) faster exporting, plus a large pile of under-the-hood changes to facilitate future features. Oh, and an Italian translation.

v2.35a will probably be released this week with a small pile of bugfixes. Most of these bugs were found while testing out changes made to the development trunk.

On that note, there are a lot of cool things in the pipeline for v2.36; the most visible of which is a new theme! Rickard Olsson got the ball rolling and contributed a dark theme, which I then mangled a bit and committed. When combined with pretty URLs and per-folder thumbnails, things look pretty slick. It's funny how sometimes just how effective superficial changes can be.

Syndicated 2008-02-20 03:00:46 from Solomon Peachy

More ES1 gutenprint goodness

Gutenprint has accepted my second patch, so it now has a working Selphy ES1 raster driver. Unfortunately, it still requires a custom print spooler, but I'm now one step closer.

es_print_assist.c is now updated to properly poll the printer status, so it can now take the raw dump from gutenprint and shove it out to the printer with minimal delay.

The third step will be to rework it so that it can deal with an arbitrary file on stdin, properly parsing the dumpfile to determine length and paper type.. and for step four, adapting it into a proper CUPS backend. Yay.

Syndicated 2007-11-24 00:15:28 from Solomon Peachy

One patch accepted, one more to go..

The fine folks behind Gutenprint accepted my patch to support the Canon Selphy ES series, but thanks to a boneheaded mistake on my part, what got committed didn't actually work. So there's a fixup patch pending.

The real fun, however, is the need to write a custom CUPS backend to properly spool data to the printer. I have a little helper app (es_print_assist.c) that batches the writes properly, but it dumbly waits instead of properly polling the printer for its status. CUPS is a lot more complicated to figure out than gutenprint, so further progress will be much slower.

Meanwhile, Photo Organizer 2.35 is coming along nicely; I'm at the point where I have to decide whether to go into -rc stabilization now, and save the next round of invasive changes for 2.36, or go ahead and make one or more of those changes now.

In particular, I want to be able to have PO auto-generate full-resolution JPEGs from the source RAW images. On the surface this is straightforward, but I want to implement this properly, by genericizing the "generate a down-scaled image and apply this set of transforms to it" code. This way additional sizes would be trivially easy to add, as would some of the changes I have in mind to make watermarking much more useful. Progress has been slow, but I'm almost done getting the low-level bits in place.

Anyway. Tons of stuff to do, never enough time..

Syndicated 2007-11-15 18:25:43 from Solomon Peachy

11 Nov 2007 (updated 11 Nov 2007 at 17:53 UTC) »

The joy of photo printers (and free software)

For some time now, I've wanted to pick up a compact photo printer to take with me on assignment, with the blessings of those I am taking photographs for. A little under two weeks ago, I finally did, purchasing a Canon SELPHY ES1.

It's a sweet little printer, using the old technique of dye-sublimation to create true continuious tone prints, rather than glorified halftoning that even the best inkjet printers use. Not only do the prints come out looking indistinguishable from what a photo lab would produce -- they're water- and smudge-proof.

I did my homework; apparently the majority of Canon's dyesub printers were supprted under Linux via the gutenprint drivers, but not the ES1 specifically. No big deal, it should just work. Even in the absence of direct Linux printing, I could print from the camera directly or shove a memory card into the printer. All in all, things should Just Work.

They didn't.

My first test involved taking a few converted-from-RAW JPEG images out of my archives, copying them to a CF card, and trying to print that. I got a rather crass Incompatible JPEG Format error message out of the printer. Interestingly my camera also errored out on those images, complaining that The image could not be displayed.. After some heavy digging it turns out the printer makes heavy use of the EXIF data, and if it's not present (or in many cases, simply modified!) the printer gives up. WTF? Why can't Canon document what it needs in a JPEG rather than just displaying a useless error message?

As I shoot RAW images, not being able to convert, crop, tweak, then print a random image via a CF card seriously sucked. So, I'll try Plan B: Print the images from the camera via the universal PictBridge interface.

No good.

Apparently my Nikon D200 camera can't print RAW images. WTF? Even if the camera could only print JPEGs, the NEFs have a full-res embedded JPEG image in the file that would print just fine. Sigh. Onto Plan C: Print directly from my laptop.

No good.

Apparently the SELPHY ES1 is incompatible with Canon's older dyesubs. To some extent I expected this, as it uses a different ribbon/dye pack, but that's mostly because the printer's physical engine is oriented differently -- and it's also why I bought this model over the others. Thanks to this incompatibility, I can't print from Linux either. Onto Plan D: Print from Windows. Surely that will work, right?

Sort of.

The printer worked just fine from Windows... but the prints were all quite dark. Too dark. After some digging, I found the driver's options panel and knocked the brightness up a few notches.. and while not perfect (yellow-ish color balance, mostly) the images were finally acceptable. But this would mean I'd need to boot into Windows to print, which really sucks as the rest of my RAW workflow is Linux-based.

Fortunately, the printer is USB-based, which means that thanks to a wonderful tool called Snoopy2, it's trivial to get a full dump of the entire communications chain between the printer and its driver. Armed with this dump, I could figure out the protocol and hack support into gutenprint.

After an initial learning curve, I succeeded. I was able to generate a binary dump indistinguishable from what Windows generated (except, of course, for the image data). So, cackling with glee, I proceeded to dump this out to the printer.

No good.

The device write() apparently blocked on the very first chunk of data. After much experimentation, I discovered that the logical chunks of data needed to be broken apart and written separately. The initialization sequence and the Yellow, Magenta, and Cyan image data all needed to have pauses between them (the printer sends a status message when it's ready) or the printer's USB interface locks up altogether. Sigh. So I split apart my dump file into its logical chunks, and dump them separately to the printer.

Success!

Not only did it print, but the brightness and color balance looked great. Yes, the images look much better than what their Windows driver manages to put out.

Ah, I love Free Software. When it doesn't JustWork(tm), you can fix it so it does.

All that remains is getting my patch

integrated into upstream gutenprint, and figuring out a way to intellently spool the printer data in a CUPS-compatible manner.

Oh, this was the first image I printed:

I took it last weekend at Paradise Beach. I have no idea who this guy is, but he was out kite-surfing on a windy but otherwise beautiful day.

Oh, as a footnote -- about a month before I ordered my ES1, Canon announced its successor models, the ES2 and ES20. Same basic specs, but when untethered the printers had fancier (and faster) feature sets. I needed a printer for next weekend (November 16-18) and nobody had a useful ETA on when they'd show up, so I bought the ES1 at a discount. On the 9th, four days after I received my ES1, everyone suddently got them in stock. Sigh.

Syndicated 2007-11-11 14:38:31 from Solomon Peachy

23 Sep 2007 (updated 11 Nov 2007 at 15:10 UTC) »

Photo Organizer 2.34 (finally) released!

To quote the press release:

The long-awaited version 2.34 of Photo Organizer is finally here. Nearly every facet of Photo Organizer has been enhanced in some way. The most visible improvement is the refactored UI that makes extensive use of CSS and supports multiple languages. Working with larger sets of images and especially clients is also considerably simpler, with the ability to tie folders and clients to datebook events. There are also many behind-the-scenes changes to improve reliability, facilitate future scalibility and of course, a massive pile of little features and tweaks.

You can find it at: po.shaftnet.org

Syndicated 2007-09-23 17:26:50 (Updated 2007-11-11 15:10:12) from Solomon Peachy

11 Jul 2007 (updated 11 Jul 2007 at 14:57 UTC) »
hypatia:
"But here's the key: for internal use, not just for showing other people our photos."
"We need something that combines 'we'd like to show people some photos' with 'we have a lot of photos we just store and annotate.'"

These phrases are exactly what I asked myself a few years ago. After some heavy digging around, I settled upon Photo Organizer. I started customizing it to my needs, and ended up contributing so much that I eventually was handed maintainership of the project.

http://po.shaftnet.org is the project's current home.

It's not all that you asked for, but it wants to be. It's database-driven (PostgreSQL), fundamentally multi-user, and is intended to be a photographer's primary image repository. It has decent access controls (beyond "public" and "private", that is) and pretty good tagging abilities. It has good export capabilities, is GPL'ed, and runs on your own server, so there's no danger of lock-in.

As an added bonus you can group together multiple versions of photos. Its filesystem layout makes it fairly easy to back up, but there's no automatic mechanism for doing so. (I use a nightly cron job that does a database dump and an rsync)

My personal installation runs off a server at home, and currently manages just shy of 30K photos taking up some 113 gigs.

Photo Organizer sounds like it is a fairly close match for your stated needs, and is improving continually as new itches come up. :)

linux-wlan-ng still lives!

After nearly three months since the last release, out comes linux-wlan-ng 0.2.7, with changes to support current Linux kernels, plus a few more bugfixes.

The project remains completely obselete, having been nearly completely eclipsed by the drivers in the linux kernel, but it continues to have a few differentiating features, the most significant of which is support for Prism 2/2.5/3 USB widgets.

It's funny; The very split-MAC architechure that linux-wlan-ng was derided for is the future of Linux wireless -- And the same problems are coming up in almost the same order, as are the same mistakes, and with them the inevitable conclusion that some of these problems are hard.

Sometimes I really dislike that I spend most of my time hacking on proprietary code -- linux-wlan and linux-wlan-ng were my employer's experiment with open-source code, and it almost put us out of business. If we can't get paid for support, and we can't get paid to write software, how exactly are we supposed to pay the bills?

Is the entire F/OSS "business model" subsidized by proprietary components?

Syndicated 2007-01-11 20:54:53 from Solomon Peachy

98 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!