Web element grabbing

If you administer anything, you're either web-scraping or your're already in the carpal tunnel ward. How many have a script that visits every application's supposedly all-in-one "dashboard" to grab the one piece of information you need?

I've written my share of yeechy pattern matches. (Bad enough that I actually invented the Reverse Bastard Copyleft: "You can do anything with this script that you want, as long as you claim YOU wrote it.")

In the Media, we have a similar problem: partner sites and multiple content management systems. When I need to get the weekly kernel article from LWN, it's view source, cut, paste. In the never-ending quest to get things organized around here, I ran into this useful article from Sean M. Burke, and now I should be able to do more things like:

element ArticleText http://lwn.net/Articles/287056/ | xsel

Here's the script: element

I think this will be useful when I'm building chunks of stuff to incorporate into a CMS-built page. Instead of having to look at validator output for the whole final page (if your pages validate they throw you out of the Mainstream Media) I'll be able to build my thing in a simple template, validate the page, then extract the right chunk and pass it on.

Syndicated 2008-06-23 16:29:45 from dmarti's blog