Advogato blog for rcaden

Adding ReCaptcha to a Weblog

Thu, 14 Aug 2008 15:06:32 GMT

I've added a ReCaptcha component to the comment form on Workbench to deter spammers. The ReCaptcha system presents two hard-to-read words that must be typed in successfully for a comment to be saved. Here's what the component looks like:

I tried as long as possible to avoid using captchas, but the amount of spam hitting this blog continues to grow, particularly from foreign IP addresses. Workbench has received 16,000 comments and more than 260,000 spam since it began accepting comments in 2002.

The ReCaptcha project serves a useful purpose, digitizing old books and newspaper articles by getting millions of people to identify words that OCR software couldn't recognize. Adding the component took around 10 minutes: I signed up for a ReCaptcha account, stored the PHP library recaptchalib.php on my web server, and added less than 20 lines of PHP to the page that takes comments.

The addition of captchas serves as official notice that my comment flak technique failed to deter spammers. I'm retiring that code.

SiteMeter Crashes Internet Explorer with 'Operation Aborted'

Sat, 2 Aug 2008 15:06:29 GMT

Last night several of my web sites, including the Drudge Retort, began crashing Internet Explorer with the error message "Internet Explorer cannot open the Internet site ... Operation aborted."

I've encountered this error before, and when it occurs out of the blue on a site you haven't changed, the culprit is usually a problem with third-party Javascript code, as CNet's Clientside blog explains:

IE does this when you attempt to modify a DOM element before it is closed. This means that if you try and append a child element to another and that other element (like the document.body) is still loading, you'll get this error.

To find the error, remove JavaScript widgets one at a time from your site until the error disappears. The culprit here was SiteMeter, which made some recent changes to its code. I've pulled the SiteMeter code until they announce a fix.

Displaying Twitter Updates on a Web Page

Tue, 8 Jul 2008 20:10:26 GMT

I recently began using Twitter, a microblogging service for posting short, chat-like blog entries and reading what other users of the service are doing. The site has severe reliability problems, but it's still an entertaining way to get real-time updates from bloggers I read along with others I know who've been sucked into Twitter's maw.

I wrote some code to display my most recent Twitter update on my weblog, Workbench, in a sidebar at upper right. This afternoon, I've released the Twitter-RSS-to-HTML PHP script under an open source license. The script requires MagpieRSS for PHP, an open source PHP library that can parse RSS and Atom feeds.

MagpieRSS caches feed data, so at times when Twitter is glacially slow or can't be accessed, this script won't hurt the performance of your server.

The first release of the script only works with a Twitter user's RSS feed, which can be found in the "RSS" link at the bottom of a user's Twitter page. The only tough part about writing the script was creating regular expressions to turn URLs into hyperlinks and "@" references into links to Twitter user pages:

// turn URLs into hyperlinks
$tweet = preg_replace("/(http://)(.*?)/([w./&=?-,:;#_~%+]*)/", "<a href="\0">Link</a>", $tweet);
// link to users in replies
$tweet = preg_replace("(@([a-zA-Z0-9]+))", "<a href="http://www.twitter.com/\1">\0</a>", $tweet);

If you're reading this and wondering why anyone should bother with Twitter, I recommend reading the updates by Jay Rosen, a journalism chair at New York University who uses the service to share a running dialogue on the media. He punches above his weight in this 140-character-or-less medium.

Following Web Page Redirects with Java

Mon, 26 May 2008 21:17:00 GMT

CNET moved a bunch of its blogs to a different domain this weekend, including Beyond Binary, Coop's Corner, Geek Gestalt, One More Thing, Outside the Lines and The Social. I mention this because the change hosed Meme13, which treated all six as if they were newly discovered sites.

One of my ground rules for developing Meme13 is that I won't hand-edit the site to make it smarter. I need the application to recognize when existing sites in its database have moved.

Meme13 monitors sites using a Java application I wrote that downloads web pages with the Apache HTTPClient 3.0 class library. Web servers indicate that a page has moved by sending an HTTP redirect response of either "301 Moved Permanently," which indicates a permanent move, or "302 Found," which is intended for temporary changes. I wrote a Java method that can find the current location of a web page, even if it has been redirected one or more times:

public String checkFeedUrl(String feedUrl) {
    String response = feedUrl;
    HttpClient client = new HttpClient();
    HttpMethod method = new HeadMethod(feedUrl);
    method.setFollowRedirects(false);
    try {
        // request feed
        int statusCode = client.executeMethod(method);
        if ((statusCode == 301) | (statusCode == 302)) {
            // feed has moved
            Header location = method.getResponseHeader("Location");
            if (!location.getValue().equals("")) {
                // recursively check URL until it's not redirected any more
                response = checkFeedUrl(location.getValue());
            }
        } else {
            response = feedUrl;
        }
    } catch (IOException ioe) {
        response = feedUrl;
    }
    return response;
}

The HeadMethod class requests a web page's headers instead of requesting the entire page, consuming far less bandwidth as it checks for redirects. My Java method looks for both kinds of redirects, because web publishers have a bad habit of using "302 Found" when they've moved a page permanently.

Setting the Link on a ShareThis Widget

Tue, 20 May 2008 02:20:56 GMT

I'm continuing to work on Meme13, a site that packages together the last 13 sites to show up on the Techmeme Leaderboard so they can be sampled as a feed or web site. The site has attracted around 25 RSS subscribers in its first month.

I've added a ShareThis widget on each entry that makes it easy to share content from Meme13 on sites like De.licio.us, Digg and Facebook.

Normally, ShareThis links to the page the widget has been displayed on. That doesn't suit my purposes on Meme13, because I'm trying to promote the originators of the content. If someone reads the article about landing a startup job by Ryan Spoon on Meme13, the ShareThis widget should link to the article on Spoon's blog.

ShareThis has a JavaScript API that can be used to teach the widget new tricks. Here's the JavaScript code to set the widget's target link and display the widget:

<p><script language="javascript" type="text/javascript">
SHARETHIS.addEntry({
title:'<TMPL_VAR title>',
url:'<TMPL_VAR link ESCAPE="HTML">',
}, {button:true} );
</script></p>

The <TMPL_VAR title> and <TMPL_VAR link ESCAPE="HTML"> tags are part of the template language used by Planet Planet, the software that publishes Meme13. Here's how the same thing could be done in PHP:

<p><script language="javascript" type="text/javascript">
SHARETHIS.addEntry({
title:'<? echo $site_title; ?>',
url:'<? echo $site_link; ?>',
}, {button:true} );
</script></p>

How to Crash Your Apache Server with PHP

Fri, 2 May 2008 16:13:49 GMT

I returned from a trip out of town Monday to crashing web servers that ate my lunch all week long. For several days, I used the top command in Linux and watched helplessly as two servers ground to a halt with load averages higher than 100.

Top reports the processes that are taking up the most CPU, memory and time. On the server running Workbench, the culprit was always httpd, the Apache web server. This didn't make sense, because Apache serves web pages, images, and other files with incredible efficiency. You have to hose things pretty badly to make Apache suck.

If you know the process ID of a server hog, Apache can tell you what that process is doing in its server status report, a feature that requires the mod_status module. The report for Apache's web site shows what they look like.

Using this report, I found the culprit: A PHP script I wrote to receive trackback pings was loading the originating site before accepting the ping, which helps ensure it's legit:

// make sure the trackback ping's URL links back to us
$handle = fopen($url, "r");
$tb_page = '';
while (!feof($handle)) {
$tb_page .= fread($handle, 8192);
}
fclose($handle);
$pos = strpos($tb_page, "http://www.cadenhead.org/workbench");
if ($pos === false) {
$error_code = 1;
send_response(1, "No link found to this site.");
exit;
}

Most trackback pings are not legit -- I've received 600 from spammers in just the past three hours. Each ping required Apache to check the spammer's site, download a page if it existed, and look for a link to Workbench. A single process performing this task could occupy more than 50 percent of the CPU and run for a minute or more.

I'm surprised Apache ran at all after I added trackback a couple months ago. I was beginning to think the web server software was idiot-proof, but I've proven otherwise.

Twitter: Where Ruby on Rails Goes Off the Track

Fri, 2 May 2008 00:10:56 GMT

An article posted on eWeek today was written in an alternate universe where Twitter works:

As the maker of one of the largest applications using Ruby on Rails on the Web, Twitter knows a thing or two about scaling applications built with the popular development framework.
Britt Selvitelle, a senior engineer at Twitter, offered a few tips and tricks for scaling Ruby on Rails and expressed particular appreciation for the Rails framework itself and the language is it based on, Ruby.
"For us, for a large part of our system, Ruby has been the tool that fit," Selvitelle said.

The subhead of the article: "Twitter's reliance on Ruby and Ruby on Rails proves the language's resilience."

Twitter's a nice service, but it's one of the most crash-prone sites I've ever visited. The fact it was written in Ruby on Rails makes me wonder whether the Rails framework can scale, at least once you reach the big leagues and have several hundred thousand users hammering on your web application. On the same day as the eWeek article, TechCrunch floated a rumor that Twitter is dumping Ruby on Rails.

-- via Meme13

Meme13: Angering New Bloggers in Tech

Thu, 17 Apr 2008 12:21:22 GMT

Meme13 is getting knocked around a bit by people who think that it's just another scraper republishing RSS feeds, hurting the search-engine rank and traffic of the publishers who created the content. Two of those people are Tony Hung and Darren Rowse, bloggers currently featured on Meme13. Hung writes:

... Meme13 is simply pulling feeds and republishing them all.
Like any good ol' scraper blog. ...
More of the GD same -- and what's really funny (again, not in a ha ha way) is not even that Meme13 acknowledges what side of the debate its on, but that its apparently deaf to one of the bigger memes on the leaderboard that its supposedly tracking, and, one of the "bottom 13" that it wants to highlight who felt pretty vocal about the issue (me)!
I am flabbergasted, exhausted, and ... just flabbergasted.

Hung's lament was the top post on Meme13 for several hours Tuesday night. My day-old robot has already turned against its creator.

I'm mindful of these concerns, but I don't think the new service is leeching. I'm aware that "I'm helping you" is Web 2.0-speak for "I'm helping myself to your work" -- one commenter on Rowse's ProBlogger posts the rather Confucian "verbal statements of warm fuzzy intentions can mask a blatant ripoff" -- but the idea I'm pursuing here is an RSS mashup that brings new subscribers to publishers.

Sites only are featured on Meme13 for a short time -- currently around two weeks -- before they drop off and never appear again. The web site doesn't archive anything, so when Hung's Deep Jive Interests is replaced by a newer blog in around nine days, there will be nothing on Meme13 to hurt his search rank.

When I developed Meme13, I considered the idea of only making it available as a feed, because I think that's where the purpose is most clear.

I created this because I needed it. I want to sample new blogs and subscribe to the interesting ones with as little effort on my part as possible. If you see exactly what a feed contains during its trial period on Meme13, including images, full text and the feed's own ads, it makes it easier to decide if you want to subscribe.

This approach appears to be a novel application of RSS -- a rolling reader that constantly changes its list of subscribed feeds.

I'm going to hammer at this idea for a while before giving up on the concept of mashing together full feed entries. If this experiment angers more of the bloggers it's designed to benefit, you can follow their reactions by subscribing to Meme13.

Meme13: Finding New Bloggers in Tech

Thu, 17 Apr 2008 12:21:22 GMT

Steven Hodson writes:

I get a real kick out of it when people start pontificating on why the tech blogosphere is becoming nothing more than [a] self-fulfilling chamber filled with the dull echos of me-too posting that attach themselves like leeches to the supposed brilliant writings of the blogosphere mucky mucks.

Me too.

Every six months or so, techbloggers reach the joint realization that we're all linking to the same people having the same thoughts about the same subjects. Somebody blames Techmeme, a site that collects the most popular links, and we all link to that guy. The resulting argument shows up on Techmeme. A good time is had by all. Last time around, I said I wanted a Techmeme that pretends the most-linked tech sources don't exist, then looks at what's being linked by everybody else. The top bloggers link to each other constantly. You have to look elsewhere for up-and-coming bloggers who are still working in the sweet spot that lies between obscure and insufferable.

This morning, I'm launching a new site, Meme13, to find those bloggers.

Meme13 mashes together the last 13 sites that made their first appearance on the Techmeme Leaderboard. You can read these sites by visiting Meme13 or subscribing to its feed, which contains the latest entries from all of them.

I've been tracking the leaderboard since Feb. 4. In that time, 175 different sites have made an appearance on the top-100 list. The current Meme13 made their Techmeme debut in the past two weeks:

When a new site appears, the oldest Meme13 site drops off. So far, sites have stuck around for approximately two weeks. Over time Meme13 should get better at finding lesser-known sites as its database grows.

Meme13's an XML hack that downloads OPML data using a XOM-based Java application, stores the elements in a MySQL database and uses Planet Planet to publish the feeds as a web site and Atom feed. It's updated hourly and published automatically.

The site needs a lot of work, particularly on the interface, but I figured it was time to loose this experiment upon the world.

Loading Ad Javascript with PHP

Sun, 2 Mar 2008 00:10:40 GMT

I serve ads on the Drudge Retort using Blogads, a great ad broker that occasionally has trouble serving the ads. When this happens, pages on the Retort load more slowly because they can't fetch a Javascript program and CSS stylesheet required by Blogads.

I decided to fix this problem by writing Cache Remote File, a PHP script that performs three functions:

Save a cached copy of a remote file
Display the cached copy for 10 minutes before requesting the file again
Display the cached copy when the remote server is offline or slow

The script will give up trying to load the remote file after three seconds, which keeps it from hanging when the remote server is having difficulties. It can be customized to load any URL of any content type and requires PHP 4 or higher with cURL support. I've released it under the GPL. Let me know if you have any problems with it or can improve the script.