PersistJS: Cross Browser Client-Side Persistent Storage Without Cookies

I just released PersistJS, a client-side JavaScript persistent storage library. Features include:

• Small (9.3k minified, 3k gzipped)
• Standalone: Does not need any additional browser plugins or JavaScript libraries to work on the vast majority of current browsers.
• Consistent: Provides a consistent, opaque API, regardless of the browser.
• Extensible: Custom backends can be added easily.
• Backwards Compatible: Can fall back to flash or cookies if no client-side storage solution for the given browser is available.
• Forwards Compatible: Supports the upcoming versions of Internet Explorer, Firefox, and Safari (Opera too, if you have Flash).
• Unobtrusive: Capability testing rather than browser detection, so newer standards-compliant browsers will automatically be supported.

If you already know why this is awesome, then you can skip straight to the download. If you're scratching your head, then read on... <h2>Why This is Awesome</h2>

Why use PersistJS? What's the problem with using cookies directly or simply requiring Flash?

Currently the only reliable cross-platform and cross-browser mechanism for storing data on the client side are cookies. Unfortunately, using cookies to store persistent data has several problems:

• Size: Cookies are limited to about 4 kilobytes in size.
• Bandwidth: Cookies are sent along with every HTTP transaction.
• Complexity: Cookies are difficult to manipulate correctly.

Modern web browsers have addressed these issues by adding non-Cookie mechanisms for saving client-side persistent data. Each of these solutions are simpler to use than cookies, can store far more data, and are not transmitted along with HTTP requests. Unfortunately, each browser has addressed the problem in a different and incompatible way. There are currently 4 different client side persistent data solutions:

• globalStorage: Firefox 2.0+, Internet Explorer 8
• localStorage: development WebKit
• openDatabase: Safari 3.1+
• userdata behavior: Internet Explorer 5.5+

Some developers have attempted to address the client side storage issue with the following browser plugins:

• Adobe Flash
• Google Gears

The problem with relying on plugins, of course, is that users without the plugin installed miss out on the feature in question, and your application is dependent on software from a particular vendor. Google Gears, for example, is not widely deployed. Flash is, but it has problems of its own:

• Many users block Flash or require a click in order to enable flash content; this makes Flash unsuitable as a transparent, client-side data store.
• Flash is notoriously unreliable on newer 64-bit machines.
• Some businesses block Flash content as a security measure.

Anyway, if we include Gears and Flash, that means there are no less than 6 incompatible solutions for storing client-side persistent data.

The most notable attempt at addressing this problem is probably Dojo Storage. Unfortunately, Dojo Storage does not support Internet Explorer without Flash, and it does not support Safari or other WebKit-based browsers at all (at least, not without Flash). Also, Dojo Storage is not standalone; it requires a several other Dojo components in order to operate.

PersistJS addresses all of the issues above. It currently supports persistent client-side storage through the following backends:

• flash: Flash 8 persistent storage.
• gears: Google Gears-based persistent storage.
• localstorage: HTML5 draft storage.
• whatwg_db: HTML5 draft database storage.
• globalstorage: HTML5 draft storage (old spec).
• ie: Internet Explorer userdata behaviors.
• cookie: Cookie-based persistent storage.

Each backend exploses the exact same interface, which means you don't have to know or care which backend is being used. <h2>Examples</h2>

Here are some simple examples of PersistJS in use:

// create a new client-side persistent data store
var store = new Persist.Store('My Data Store');

// pretend data
var data = "pretend this is really long data that won't fit in a cookie";

// save data in store
store.set('saved_data', data);

That's all there is to creating a persistent store and adding some data to it. Fetching data back from the store uses a callback function (to support asyncronous backends), but it's still pretty simple to use:

// get data back from store, and prompt user with it
store.get('saved_data', function(ok, val) {
  if (ok)
    alert('saved data = ' + val);
});

Removing data is pretty easy too:

// remove data from store
store.remove('saved_data');

Although it isn't necessary, you can also get some information about the detected backend using the Persist.type and Persist.size attributes:

// build alert message
var info = [
  'Backend: ',
  Persist.type || 'none',
  ', ',
  'Approximate Size Limit: ',
  (Persist.size < 0) ? 'unknown' : Persist.size 
].join('');

// prompt user with information
alert(info);

Finally, if you don't want a particular backend used under any circumstances, you can disable it using the Persist.remove() function:

// never use cookies for persistent storage
Persist.remove('cookie');

<h2>Download</h2>

This is the initial release, so there are bound to be some bugs. PersistJS has been tested with FireFox 2.0, FireFox 3.0rc1, IE7, and Safari 3.1. The Gears and Flash backends work where Gears and Flash 8 are supported.

The most notable omission here is IE6; it should work, but I don't have IE6 handy at the moment (MultipleIEs is temporarily busted).

• Download PersistJS 0.1.0 Tarball (Signature)
• PersistJS Mercurial Repository

Commenting is still busted around here, so any comments should sent via email or posted on the Reddit thread.

Syndicated 2008-05-22 07:19:49 from Pablotron: News

Don't Use ExtJS

<style type='text/css'> blockquote.quote { font-family: arial, verdana, helvetica, sans-serif; border-left: 2px solid #09d; padding: 10px; background-color: #efefef; } </style> A couple of years ago I recommended YUI and Ext (formerly YUI-Ext). I've changed my mind. Don't use Ext at all. The Ext license has changed four times since its inception, and each time the license has become more restrictive. <h2>History</h2>

Ext was originally created as an extension to YUI. It was BSD-licensed, just like YUI. YUI-Ext added several sorely-needed features to YUI. The most notable additions were a layout system, a Tree View widget and a Data Grid widget (YUI has since added each of these, although the YUI widgets are still less flexible than their Ext counterparts). Eventually support was added for jQuery and Prototype as well. The team dropped the "YUI-" prefix, and YUI-Ext became Ext.

Ext 1.0 was relicensed under the LGPL. Although switching from the BSD license to the LGPL is relatively innocuous, it is still significant because the LGPL is more restrictive than the BSD license.

Eventually the Ext team changed the license again. The new license was a custom license that granted conditional LGPL usage rights. Basically the LGPL usage clauses applied, but only if you weren't trying to develop a library or an Ext clone.

Confused? Yeah, me too. Here's the text from the old Ext "Open Source License":

Ext is also licensed under the terms of the Open Source LGPL 3.0 license. You may use our open source license if you:

• Want to use Ext in an open source project that precludes using non-open source software
• Plan to use Ext in a personal, educational or non-profit manner
• Are using Ext in a commercial application that is not a software development library or toolkit, you will meet LGPL requirements and you do not wish to support the project

A lot of open source developers were understandably confused by this hybrid license. The biggest problem was that it wasn't clear whether this license was compatible with other Open Source licenses. It also wasn't clear whether Ext could be legally distributed with Open Source software, since the license only granted LGPL usage rights, and not LGPL distribution rights.

To address these complaints, the Ext team changed the license again: the latest version of Ext is licensed under the GPLv3. This latest change complicates things quite a bit for many users, as we'll see in the next section. <h2>Problems</h2>

The GPL is far more restrictive than the BSD license and LGPL. It is rarely used for libraries, because the viral clause would effectively the library from being used for any non-GPL software. In fact, these problem were addressed over 16 years ago by creating the LGPL:

By 1990, it was becoming apparent that a less restrictive license would be strategically useful for some software libraries; when version 2 of the GPL (GPLv2) was released in June 1991, therefore, a second license - the Library General Public License (LGPL) was introduced at the same time and numbered with version 2 to show that both were complementary. The version numbers diverged in 1999 when version 2.1 of the LGPL was released, which renamed it the GNU Lesser General Public License to reflect its place in the GNU philosophy.

Who is affected by this change? In no particular order:

• Extension Authors: Older Ext user extensions could be licensed as the author saw fit. This is no longer true for the latest version of Ext; new user extensions must be GPL-licensed, because the viral clause prohibits using Ext with non-GPL licenses.
• Commercial Users: The previous licenses, even the questionable custom license, allowed Ext to be used in closed source commercial applications. This is no longer true for the latest version of Ext, because the viral clause prohibits using Ext with non-GPL commercial licenses.
• Non-GPL Open Source Developers: The BSD and LGPL-licensed versions of Ext could be used with other non-GPL software. This is no longer true for the latest version of Ext, because the viral clause prohibits using Ext with non-GPL licenses.

Things get even murkier when you consider linking and distribution. Does generating a dynamic page count as linking to Ext? Does any public web application automatically count as distribution? What about applications which use Ext to and access a common APi, such as a SOAP endpoint or RSS feed?

These questions were posted in a thread about the license change on the ExtJS forums.
Here's how Jack Slocum, the primary Ext developer, responded:

If you are generating any markup or javascript code via the server in a page SPECIFICALLY designed for Ext, then that server code will have to be GPL as well.

For example:

• Suppose you have an index.php that includes Ext JS. According to the FSF, in that case index.php would be also under GPL since it is using ext. Since it must be GPL, it's source must be distributed. Since it is GPL, the "viral" effect of GPL is now in effect and any thing that uses index.php (if anything) on the serverside would also fall under the GPL. (Note: Note this is a pretty gray area)
• Suppose you are using server-side code to generate javascript that interacts with Ext JS. That code must also be GPL.

Like MySql and other GPL software the way to use GPL code without having to license under GPL is to not bundle or distribute the GPL code with your application. If you instead have the end user (developer?) download and install ext js on their own, they are then bound to the license and not you or your software.

For those seeking an FAQ, we have defined and explained some of the reasoning and license implications under these 2 pages:

http://extjs.com/products/license.php

http://extjs.com/company/dual.php

It's worth noting that the examples given at the beginning of this post are just my opinion and it is impossible for us to analyze everyone's usage and say whether or not someone "complies" with the GPL. That really is a task for an attorney or even someone with better knowledge of your application and how Ext JS is used.

In the end, we want Ext JS to be open source friendly and still have a good business model in place to grow. The old Ext License was not open source friendly and pretty much killed all options for use in open source projects. That wasn't our goal so we had to address it.

There are several problems with the statements above. The biggest one is that the original BSD license and subsequent LGPL license had none of this ambiguity.

In other words, the problem the Ext team is trying to fix is one they created themselves. If that wasn't bad enough, the solution actually hurts many Open Source developers far more than it helps.

In an attempt to clarify the situation for non-GPL Open Source developers, I posted several questions in the Ext license thread. I also created a post on Reddit about the license change and summarized my questions there:

As of page 8 of the thread on the license change I have yet to receive a response, simple or otherwise, to any of my comments:

There's this comment:

The new license prevents Open Source software that is using a license other than the GPL from using Ext. Applications which use popular Open Source licenses like the LGPL license, BSD license, MIT license, and the Artistic license would be required to either re-license under the GPL, carefully design their application to meet the requirements in your post, use an older LGPL-licensed version of Ext, or move to another library entirely.

And this one:

What about authors who which to provide their software under a license that is more permissive than the GPL, such as the MIT or BSD licenses?

And this one, which was directly in response to Jack Slocum, the primary author of ExtJS:

Hi Jack,

I can see how switching to a license without exceptions would make things simpler, but what about those of us who release Open Source software under non-GPL licenses such as the BSD, MIT, and Artistic licenses?

I've been an Ext user since its inception as YUI-Ext, but the fact that I cannot seem to get a straight answer to a simple question makes me wary and extremely skeptical.

John Resig, the author of jQuery and Processing.js, responded:

It's important to understand that OSS developers are not their target audience at all. I'm 100% certain that we'll never get a clear response. They're using 'open source' as a buzzword selling point to lure companies in, befuddle them with confusing viral licensing, and obligate them (through the obvious balking that the corporate lawyers will do) to get them to buy a full, corporate, license. It's very sneaky, quite disingenuous, and paints a bad picture for open source development as a whole.

It's been over three weeks since these this exchange on Reddit. None of my questions have been answered on the Ext license pages or in the 68-page license thread on the Ext forums. <h2>Rationale</h2>

According to the Ext license page, Ext licensing is based on the principle of "Quid Pro Quo", or "something for something":

Dual Licensing is based on the principle of Quid Pro Quo - "something for something". In return for the advantages you realize from using an Ext product to create your application, we require that you do one of the following:

• Contribute to the continued development of the product by purchasing commercial licenses from Ext. This option secures you the right to distribute your application under the license terms of your choice
• Contribute to the Open Source community by placing your application under an Open Source license (e.g. GPL v3). This option secures all users the rights to obtain the application's full source code, modify it, and redistribute it.

The justification for using the GPLv3 instead of the LGPL is addressed on the Ext license FAQ page:

We considered once again releasing under straight LGPL but it was not an option as a business. We tried that with version 1.0 and found out quickly that it enabled others (e.g. large commercial entities) to take our work, wrap it up and sell it as their own. With no mention of us at all. We, as a business with a full time team of talented developers, can not exist under those circumstances. We would quickly become diluted and competing with ourselves.

The concern about others taking their work and selling it without attribution is particularly ironic, considering:

• Ext only exists because a large corporation (Yahoo!) decided to share their hard work under a permissive open source license
• Significant portions of Ext, including the Event handling code and reset.css, were copied wholesale from YUI
• The Ext object system comes from YUI, which is based on Dean Edwards' JavaScript inheritance code
• Many of the older Ext icons originally came from the Famfamfam Silk icon set

What do the projects above have in common? That's right, they are all in the public domain or available under extremely permissive Open Source licenses.

The Ext team is certainly entitled to license and sell their software any way they see fit. However, it is hypocritical and dishonest to complain about other people taking your work and selling it as their own when you take other peoples' work and either sell it as your own or relicense it under an extremely restrictive license.

It is tempting to attribute this entire fiasco to a simple misunderstanding on the part of Jack Slocum and the Ext team. Here's what I had to say on Reddit:

It is a bit disconcerting that Ext has such strong roots in existing Open Source software, and yet the project seems at best partially indifferent, and at worst, outright hostile to the Open Source community.

Unfortunately, according to John Resig, this isn't the first time that there have been problems with the Ext team:

We (the jQuery project) worked hard with them to try and fix bugs and add features for an ExtJS integration layer. They turned around and built their own, specialized, library (removing the need for any of our work) and then mutated the licensing into this bizzaro scheme that they have now. We can't, in good consciousness, even recommend their library anymore due to its very nature. On top of this they ended up hiring our lead evangelist to promote their work. I can't speak for everyone on the team but I feel quite frustrated and used.

They're providing a great disservice to the Open Source community in general. They consume with reckless abandon, it's impossible to even hope to borrow code from them, and they turn it all into a money-making machine. No aspect of that sits well with me.

Jack Slocum did respond to this comment on a separate blog. He also wrote a post on his blog. Neither adequately addresses John Ressig's main points or my questions from the Ext forums, so I won't bother quoting his mostly vacuous responses here. <h2>Conclusion</h2>

To recap, the reasons I recommend against using Ext are:

• A trend towards more restrictive licenses
• Team is unwilling or unable to address licensing issues
• Blatant disregard for other Open Source projects

Some suggestions for the Ext team:

• Release Ext under the LGPLv3, BSD license, or GPLv3 with the special provision
• Provide the Ext artwork and CSS under one of the licenses above, or one of the Creative Commons attribution licenses
• Update the Ext license FAQ with detailed information about which Open Source licenses are compatible with Ext, including specific usage scenarios
• Add a list of Open Source software which Ext has borrowed from to the license page and to the license information included in the download
• Put older versions of Ext back on the download page

Finally, here are a list of Ext alternatives. None are as nice as Ext, but they are all available under permissive licenses and they each have an active and enthusiastic user community:

• Dojo/Dijit (BSD license)
• Echo2 (MPL license)
• jQuery/jQuery UI (BSD license)
• YUI (BSD license)

Comments are still broken at the moment. I've posted this article on Reddit, so feel free to comment there.

Syndicated 2008-05-15 07:26:16 from Pablotron: News

EasingJS, a JavaScript Easing Library

I just released EasingJS 0.1.1, a port Robert Penner's ActionScript Easing library to JavaScript. EasingJS allows you to easily generate smooth and stylish animation or color transitions. For some examples, check out the test page.

• Download EasingJS 0.1.0 Tarball (Signature)
• EasingJS Test Page

Syndicated 2008-05-14 22:25:49 from Pablotron: News

Custom Firmware on the PSP

Last night I installed a custom firmware (CFW) on my PSP. The custom firmwares allow you to run unsigned homebrew (e.g. non-Sony sanctioned) applications. <h2>Applications</h2>

The PSP homebrew community has been pretty active; there are several useful applications and emulators for lots of older systems. There are even emulators for obscure systems like the ColecoVision and Neo Geo.

Here's what works for me so far, in no particular order:

• VNC client
• SSH client
• Super Nintendo (SNES) emulator
• Game Boy Color emulator
• MS-DOS emulator
• J2ME/MIDP emulator (e.g Mobile Java, or the crappy games on your cell phone)

I also tried the Genesis, NES, and N64 emulators, but they aren't working yet. Here's a picture of the SNES emulator at work:

Playing SNES on a PSP

The custom firmware also allows you to dump UMDs and run them from a memory stick. Since it's pretty much impossible to fit a PSP and 8 UMDs in the your pockets without looking like a complete tool, I'm going to offload as many UMDs as I can into the 3GB remaining on my memory stick.

The next section explains the firmware installation process. If you don't have a PSP, you may still find my creatively ominous safety warnings entertaining. <h2>Installation</h2>

Installing the custom firmware varies in complexity depending on the model of PSP model and version of the original firmware. If you're fortunate enough to have an older "phat" PSP (e.g. the larger black model) that's running firmware 1.00 or 1.50, then installing the custom firmware is fairly straightforward.

If you know someone with a PSP who already has the custom firmware installed, then the installation process is still easy enough, because they can use their PSP to help you with yours.

If you've got a newer PSP Slim (the smaller white model, like the one in the picture above) and/or are running a newer firmware, then there are no easy options left, so get ready for the comically unpleasant experience below.

In order to install the custom firmware, you'll need a spare battery and a spare memory stick. For the love of Douglas Adams, please do not use this post as a guide! There are several web sites (here and here) that cover the entire installation process in far more detail and with the appropriate safety precautions. If you mess this up you will turn your PSP into a lifeless and possibly explosive plastic brick.

The basic, high-level steps are as follows:

1. Create a Pandora's battery. This is a battery that has been modified to make the PSP into boot from the memory stick.
2. Create a Magic Memory Stick. This is a memory stick that has been specially formatted to boot and perform a firmware upgrade. Note that there are some limits on the capacity and brand of memory stick that can be used; see the guides above for more details.
3. With the PSP powered off and the battery removed, insert the magic memory stick into the PSP.
4. Insert the Pandora's battery. The PSP will power on automatically boot from the magic memory stick.
5. Use the software on the magic memory stick to install the custom firmware. On the PSP Slim the display is blank, so you just have to hit X and wait. The lights on the front of the PSP will blink for several minutes. The PSP will automatically power off when the installation is finished.
6. Remove the Pandora's battery and the magic memory stick. The memory stick can be reformatted and used as usual. The Pandora's battery can not, because some of the battery's safety features are disabled as part of the conversion process. In other words, do not attempt to use the Pandora's battery as a regular battery unless you want your PSP to melt into a smoldering puddle of goo.
7. Power on the PSP using a regular battery or the power cable. Congratulations, you are now running the custom firmware.

The hardest part of this process is creating the Pandora's battery. If you know someone with a PSP that already has the custom firmware installed, then they can run an application on their PSP to temporarily "soft-mod" a regular battery into Pandora's battery.

If you don't know anyone with a PSP that already has the custom firmware installed, then the only way to convert a regular battery into a Pandora's battery is to "hard-mod" it; that means cutting open the battery casing and disconnecting one of the leads on the internal circuitry.

The guides I read (see above) have plenty of pictures, but I was still surprised by how small the pieces actually were. Here's a picture I just took of my Pandora's battery, including a ruler and quarter as size references:

Inside a Pandora's Battery

Creating a Magic Memory Stick is much simpler. Basically you:

• format the memory stick in a special way (using mspformat)
• copy the necessary firmware installation and upgrade files into place
• generates an Initial Program Load (IPL) file,
• copy the generated IPL file to the first sector of the memory stick (using mspinst)

If you're using Windows, the "TotalNewbi Installer" and "Pandora Easy GUI" tools can automate this process. In theory, anyway. When I tried to use them in my Windows XP VMWare instance, they both had problems. The TotalNewbi Installer simply refused to work, and the Pandora Easy GUI blue-screened XP each time I ran it.

Here's what finally worked:

• used Pandora Easy GUI to copy the firmware files into place and generate the installer definition file (mspinst.idl)
• used dd in Linux to copy mspinst.ibl into the first sector of the Memory Stick

The good news is that creating the Pandora's Battery and Magic Memory Stick are the hardest steps in the process. Once you get past them, everything else is relatively straightforward. Even better, the process can be used to install custom firmware on any PSP, regardless of hardware model or firmware version.

That's it for me. If you're interested in the history and technical details of PSP homebrew, check out this extremely detailed PSP homebrew Wikipedia entry.

Syndicated 2008-02-16 15:37:32 from Pablotron: News

...as if things weren't broken enough around here

Hello from the new server!

I managed to glom things together just enough to get the bit-rot that is this site's code running on the new server. All the usual caveats about brokenness apply.

Syndicated 2008-02-05 04:47:12 from Pablotron: News