Older blog entries for lkcl (starting at number 623)

26 Aug 2009 (updated 26 Aug 2009 at 20:49 UTC) »
john resig on -to-javascript abstraction

ha. i didn't find this at the time, otherwise i would have written about it.

Pyjamas, GWT, and Objective-J all hinge around a central concept: Abstracting away the authoring of JavaScript-heavy web applications by allowing the developer to program entirely in their natively language (be it Python, Java, or an Objective-C-like-language accordingly).

yep. so, ultimately, the question becomes: do you trust the compiler to do its job? answer if your name is john resig: no. answer if your name is a pyjamas or GWT user: yes.

there's something that many experienced javascript developers miss out, and, because they are experienced, simply do not understand, and it's this: that in order to become an experienced javascript developer, you have to have... experience!

for many people, the jump into web programming is simply too much. you have to know:

  • HTML
  • CSS (on several incompatible browsers)
  • javascript
  • javascript implementation quirks on several browsers
  • DOM programming - on several browsers
  • asynchronous event-driven programming
  • the quirks of several browsers' implementations of async event-driven programming
  • frameworks and their foibles

despite the massive amounts of programming in the field of web development, the above list is just far, far too much for most people. experienced programmers who have successfully navigated the above, thanks to either curiosity, long-term exposure or inate adaptability, tend to forget that people are different, learn differently, have taken a different path from them...

on the pyjamas mailing list, we get a surprising number of people who have never done even user-interface programming let alone web programming in their lives, and some even who have never done python. for these novices, the simplicity with which they can get visual results from their efforts - in a web browser, and can feel that they could move on from those simple efforts to rival all the other people around them that they see have made massive web sites - that's very gratifying for them.

they don't care about the fact that there's a -to-javascript compiler behind it all - they just want results, and they've looked at pure javascript, and decided for whatever reason that it's not going to be for them. thus, they put their trust in the expertise of those people who *do* know javascript, and they leverage their work - and their expertise, riding off the back of it, to create their own work, without worrying about the details.

to safeguard that trust, we put the compiler through the wringer, with a series of library tests. ultimately what we want to do is to actually put as much of the standard python regression tests through the pyjs compiler! as a result, with the focus that's being spent on javascript JIT execution, there's a good chance that pyjs would actually become a very good python accelerator (like psyco).

bizarre, huh?

anyway - getting back to john's post:

This is a large abstraction - much more so than what is provided by most JavaScript libraries - you are programming in another language which is outputting JavaScript code. You are likely to never see a DOM object or any pieces of the native JavaScript language.

wrong, on both accounts. in most good javascript frameworks, you tend (i hope!) not to see too much of the DOM model, either. particularly in qooxdoo you have very good widget abstraction. the pyjamas API layers things extremely well, so that it's only if you want to write your own widgets do you get to mess with DOM objects.

and you do so through a very "thin" layer, which is there deliberately so that you can have browser-specific overrides , to take care of the browser quirks.

this is something that you simply cannot do with a "standard" javascript library (or you can, but it's incredibly messy, involving a series of case statements or involving prototype overrides). the compiler is where the hard work is done. in the case of pyjamas (i don't know how GWT works) it's done by merging AST - abstract syntax trees (!)

also, again: wrong about "never see javascript". it's there if people need it - and many people just... don't need it.

onward:

A huge benefit of JavaScript is that it's a proverbial melting pot of experience. Developers from all backgrounds code in it (those who program Java, PHP, Perl, Ruby, .NET, or any other server-side language) and it lends a lot to the style of JavaScript code that you see in the wild. By writing in a single-language stack you miss out on this collaboration with other developers. Java developers now only communicate other Java developers (for example). While this may be fine for some projects it certainly limits your range of experience that you can draw from.

it's usually a melting pot of people putting themselves through a lot of pain, trying desperately to work with a language and with frameworks that they simply do not fully understand or appreciate; the majority of people are way over their heads (see long list above) and getting really rather stressed about it.

i do warn people who come and look at pyjamas for the first time and criticise it, especially those who have never done web programming: pyjamas is what i've settled on after considerable _painful_ exposure to the alternatives.

in that list of alternatives, i include java, javascript, PHP, perl, CSS incompatibilities, browser incompatibilities, and more...

I should mention here that I'm a creator of a JavaScript language abstraction: Processing.js. You can write Canvas visualization using the Processing language (no apparent JavaScript involved).

yes - it's great! in fact, it's so good that someone tried using it, in pyjamas. in fact, it ended up as a pyjamas example. Take a look at Processing.py. It doesn't look like there's any javascript involved at all: the only bit of "javascript" is the "import processing.js" at the top. from then on, the pyjamas compiler produces javascript that sufficiently blends in well enough with quotes "real" quotes javascript that you can actually use Processing.js as if it was python!

one of the reasons for this is in fact that processing.js itself is well-designed. if it wasn't, it would be necessary to use "from __javascript__ import functionnameorwhatever" in order to drop whatever global javascript variable was in the javascript imported file into the python module's namespace. but that's another story...

... the point is: ironically, john resig's own javascript library works extremely well with pyjamas!

lastly - there's something not covered by john nor by the discussions of 124 comments: pyjamas-desktop. pyjamas-desktop is the port of pyjamas AWAY from javascript, and it includes, incredibly, an MSHTML port that uses the same browser engine as IE.

john makes a case to not place any trust in -to-javascript compilers. but - that case is entirely moot when you place pyjamas-desktop into developers' hands. not even GWT with its tens or hundreds of thousands of developers have what pyjamas-desktop offers: the ability to run the same application in all major web browsers and on all major desktop platforms.

unmodified.

just like adobe AIR promises. except, adobe AIR isn't free software...

Pyjamas 0.6 Released

hooraaay and whew :) well, it had to be done. this time, as there was quite a lot involved, i decided to do a "known bugs" list. i can live with 5 bugs, especially when firefox works, IE "mostly" works.

funny though - it's up to everyone _using_ pyjamas to help contribute to it. so, if there are reports of browsers that are broken - well.... people should have helped test them! several people have, and that's absolutely great, i'm delighted with the help and the reports received.

Pyjamas-Desktop MSHTML port

ok - like i said in the update, i got rid of the pywin32 dependency. pywin32 i didn't realise is a 5-6 mb download, whereas comtypes is 200k. IE is installed pretty much by default. that leaves pyjamas itself - a 1.8mb install because of the documentation and the (increasing) number of examples.

what i didn't realise there was that i was using the "timer" module from pywin32, whoops. that worked a treat. no thread mashing / conflicts - all lovely and hunky-dory, just as you'd expect from something that comes deliberately designed for python win32.

except... i don't _want_ pywin32, so i had to do my own timer system. whoops :) so, i started with threading.Timer. great! works? nnnno. the python threads conflict _really_ badly with COM objects. solution? use PostMessage (!!) to send a notification to the main GDI even t loop (which uses GetMessage) - that "wakes it up". prior to "waking it up", you add a function to a list of timers. in the GetMessage loop, on detection that the list is not empty, pop a function and call it.

voila - that avoids the threading problem: the function is called in the main thread, not the python Timer thread.

llooovely.

the MSHTML port is full of dreadful little tricks like that. it's a miracle it all hangs together at all :)

i'm kinda at the point now where i'm twiddling my thumbs, wondering what to do next...

what the MSHTML port is not

someone asked me yesterday why do this when there is titanium? i answered:

"naah, not like titanium at all. titanium uses Silverlight. if you haven't got Silverlight, you're screwed. this uses the MSHTML browser engine _direct_. titanium basically enables <script language="python" /> in a very weird way. so the pyjd MSHTML port, it uses COM to create an ActiveX widget that is put in a windows GDI window; it uses COM to get at the DOM model; it uses pywin32 to do a win32 message-loop"

ok - it _used_ to use pywin32: it now uses ctypes ... :)

16 Aug 2009 (updated 17 Aug 2009 at 12:49 UTC) »

ouaaaahh, i have the MSHTML port of pyjamas-desktop pretty much working, ouaaaaah :)

in case the significance hasn't sunk in: you just need to install some _standard_ python components, on windows (comtypes and pywin32) and you can use IE's browser engine to develop python applications similar to pygtk2 and pyqt4.

not "install a 30mb GTK bundle", then "install pygobject, pycairo and pygtk", and develop python desktop apps.

not "install a multi-megabyte pyqt4 bundle containing c++ that's specifically compiled for python 2.5 or python 2.6" then "maybe install MSVC++ proprietary DLLs as well", and develop python desktop apps.

just:

"install comtypes (200k)" and "install pywin32" (5mb - i'm a bit disappointed by that, and am going to see if i can remove the dependency) and install the pure-python pyjamas API (1.9mb - yes i know. most of that is the examples and the documentation).

hmmm, doesn't sound _so_ great now i actually list the size of the dependencies, does it, ha ha :)

but - i'm really happy that it's working out well. event handling has been a total bitch. onkeydown and onkeyup have disappeared from several objects; onkeypress however _is_ working (argh). this doesn't help, because many widgets react to onkeydown or onkeyup, not onkeypress...

... i'll sort it.

[update: gone. couldn't stand the 5mb dependency. found henk punt's window.py which depends solely on ctypes!]

well - we've had some offers of help, from pyjamas users, with the webkit patch process. already i've discovered that these willing people, who have a vested interested in making sure that pywebkitgtk works (pyjamas-desktop can use it, that's why), feel extremely intimidated by the webkit developers.


"webkit-dev-owner@lists.webkit.org" 	
to me
	
show details
	 10:12 am (10 minutes ago) 
You are not allowed to post to this mailing list, and your
message has
been automatically rejected.  If you think that your
messages are
being rejected in error, contact the mailing list owner at
webkit-dev-owner@lists.webkit.org.




---------- Forwarded message ---------- From: lkcl <luke.leighton@gmail.com> To: webkit-dev@lists.webkit.org Date: Fri, 14 Aug 2009 03:12:00 -0700 (PDT) Subject: [webkit-dev] Completing the goal of Webkit DOM bindings [was Re: GDOM patch spam]

Eric Seidel-6 wrote: > > I just closed out many of the GDOM bugs for patch spam after emailing the > author of these patches. >

eric,

it falls to me, because nobody else is going to do it, to hold up a mirror and to show you the implications of what you've just said.

you have said, "i just aggravated the process of achieving the goal of useful and useable webkit gobject bindings , by calling the combined contributions of many experienced developers 'Stupid Postings And Mailings' ..."

as you work for google, is that an official position of google, to insult martin soto's work, helmut's work, alp toker's work, my work, and many other contributors, by calling their contributions "stupid"?

Eric Seidel-6 wrote: > > > The proper process for getting code into WebKit does not involve uploading > 15 unexplained patches at once. > >

this again does not contribute to the goal. let's track it through.

1) david suggested that #16401 be split into smaller patches.

2) faithfully following the instructions and the process, you and i worked very well together to get several of those patches to an acceptable standard

3) you asked that one of them be split further.

4) faithfully following the process, i split them further, and created a series of patches, citing our conversation at the top of each of them.

5) having created the bugreports, i take the bugreport numbers and place them into the CHANGELOG entries, in order to comply with the process.

6) mark rowe ignores the link to the conversation and closes all of the bugs _before_ the attachments have been uploaded.

how _exactly_ is this _not_ faithfully following the process, eric? this is not a rhetorical question.

Eric Seidel-6 wrote: > > That tends to result only in annoying > reviewers and getting you banned from the bug tracker. :) >

i know you think you are meaning well by adding a smiley, but in this instance it's not a good indication of your state of mind.

firstly, quoting a very wise person i know: "stress is where the mind compares the external view with the internal one; finds that the discrepancy is too great and seeks to place blame on the EXTERNAL world".

so - let's translate what you've put: " That tends to result only in the reviewers comparing their view of how the review process should be, against the way that it is being used in good faith, having a feeling of "annoyance" and the reviewers then BLAMING the submitter. in fact, the reviewers feel that it is SO important that their view of how the review process be rigidly adhered to that if anyone else causes them a feeling of "annoyance", they will BAN that submitter from the bug tracker. i, eric seidel, find this to be very very funny and amusing".

how, exactly, eric, is insulting contributors by calling their contributions "stupid", and by declaring up-front that anyone who does not stick rigidly to a process ( a process which was never designed with such a comprehensive contribution in mind that requires a _dozen_ separate programming skills as well as the infinite patience of a saint ) - anyone not sticking rigidly to the "process" will be BANNED, how exactly is this "funny"? this is not a rhetorical question.

much _much_ more importantly, how does such a statement _contribute_ to the completion of the goal?

Eric Seidel-6 wrote: > > > The GDOM binding patches have some history of trouble. > >

translation: the attitude of some of reviewers has been shockingly disrespectful towards enthusiastic and highly skilled contributors, and they have sought to engineer ways in which the submitters can be blamed, outright, for absolutely everything. the reviewers consider themselves to be unimpeachable and above-board in every way. the contributors have been caught completely off-guard by the hostile attitude, and initially reacted very badly, but after consideration decided to continue - even in the face of absolutely shockingly bad attitudes of some of the reviewers, in the interests of free software and in the interests of completing the goal.

summary: yeah. trouble.

Eric Seidel-6 wrote: > > > I think if anyone > wants to work on GDOM patches they need to talk to one of the > long-standing > Gtk contributers and approach this one patch at a time. > >

great. i will contact them, immediately.

i trust that all contributors will be given the respect that should always go without saying, in working towards a common goal, and i trust that the contributors will afford the same respect towards the reviewers in working with what is an incredibly tedious, ill-defined and ill-equipped process, to complete the goal of providing webkit with free software gobject bindings to its DOM model.

l.

well we stiiiill haven't managed to get through all the testing, nor iron out everything that makes everybody happy enough to do a stable 0.6 release of pyjamas.

coooor, this is turning into a bit of a marathon :)

it was much easier for 0.4 and even 0.5. 0.4 i just ran the tests one day, they passed, i decided "right, that's 0.4 then". 0.5 was almost the same.

but - we've had so many contributions now that this is a leeetle different. i'd like to be able to declare, "yes, you can use this, i checked it - lots of people checked it, they're all happy bunnies, so you should be, too".

bleeeeegh :)

ok, just done the 0.6pre3 release, and instantly found a problem :) oh well, the pre-release series is just to get peoples' attention onto the svn anyway...

pyjamas-desktop xulrunner port basically works. there's an outstanding bug being reviewed with SVG 2D Canvas drawImage, in the mozilla bugtracker, but that's it: the rest is perfectly useable. (except on ubuntu, because they screwed up by not including a common version of python between upgrades, and it's caused _massive_ problems, well beyond just making hulahop unusable)

pyjamas-desktop MSHTML port sooort of works, as long as you don't mind not having any debug reporting on event callbacks. this is because the whole thing's interacting with COM apps. i'll need to do a NamedPipe and shove the python exception stack over to the main GDI loop, and print it out there.... *sigh*...

pyjamas-desktop pywebkitgtk port works, but there are two main issues. 1) exec state is not tracked. so, if you get an exception (try to add a style property that doesn't exist) then ... nothing happens! whoops... 2) the webkit developers are being deliberately obstructive. we'll sort that out, one way or the other.

the debian packaging is basically ready, i'll re-submit it when the 0.6 release is done.

various people are sorting out xulrunner and hulahop for different distros like gentoo and arch... mmmm... that's about it.

"Stress is where the mind compares the external world with the internal one, cannot cope, and places blame on the external world for the discrepancy"

a very sensible person told me this. they also gave a cute example:

I Hate Your Tie!

noo - what you mean to say is, "i have a feeling of hate, and.. and.. i'm going to... blame it on YOUR tie! yes, that's right - it's your tie's fault!"

i told this to a friend one day, when he said to me:

I HATE rush hour traffic

noooo, what you mean is, "i have a feeling of hate, and i'm going to BLAME it on the traffic".

he did laugh at that :)

well, recently, we have this:

luke leighton is causing problems with the webkit development process

nooo, what you mean is, "i can cope with the existing webkit development process, but i cannot cope with the direction and the implications that luke leighton is taking ME, therefore i am going to BLAME luke leighton."

the quotes last straw quotes was when i created an automated bugzilla patch-submission system, because the number of patches and associated bugreports, to get webkit its glib / gobject bindings, has gone up to _thirty_.

the webkit ChangeLog system is very strict, and very good. in the ChangeLog files, you place the filenames that have been modified by the rest of the patch. this means that an automated system can extract those filenames.

what i did was i wrote a patch parser:

http://lkcl.net/webkit/patman

which can understand svn diff files (and probably other diff -u files as well)

by combining the extraction of information from the ChangeLog entry and the "one big monster" diff, a series of smaller diffs can be created, with "proper" ChangeLog entries right at the top, to fit with webkit's submission process.

also in the ChangeLog it's good practice to place the bugreport number, and - again - i decided to make good use of this.

as there are thirty separate patches, i decided to write some further code which can submit attachments to bugzilla, automatically.

the patch exists, the bugreport number's in the ChangeLog, the patch is in an acceptable format - all good so far.

except... wait - the bugreport doesn't exist yet, so it has to be created.

so, that's what i did - create some bugreports. then, taking those bugreport numbers, i placed them into the ChangeLog entries, ready to run the automated uploader.

then, i made some final checks on the uploader.

when i checked the bugreports, to make sure that i would be submitting the right ones, and not go and submit patches that had already been submitted, i found that several of them had been closed.

they'd only been open for about half an hour, twenty minutes - wtf??

there was a particularly petty message in each of them:

"please do not submit empty bugreports".

mark - i'd gone to a lot of trouble to put the dependencies in, and to put a short message indicating the purpose of the bugreport, pending submission of the attachments.

i'm sorry if you feel that that's defined as "empty".

so, rather annoyed, i reopened one, because i wasn't sure how the auto-submitter would react to adding in an attachment to a closed bug.

the tool worked - the attachment was submitted. hooray!

next bug. reopen. submit. aannd.. i got presented with a message:

"Thanks for trying to contribute, but the manner and attitude in which you're doing so does not fit with the approach that the WebKit project takes to development. If you believe your account should be restored, please send email to admin@webkit.org explaining why."

nooo, you mean, "there is a discrepancy between what i can cope with, and what you are trying to achieve, and i am going to blame YOU for that".

so.

anyone reading this, if you would like webkit to become equal to its peers with respect to python, i need your help in dealing with the discrepancy between what webkit is and what it can become.

you can write to: admin@webkit.org expressing your views.

the code's already working: webkit has glib/gobject bindings, and from there it has auto-generated python bindings - it just needs to go into the main webkit repository, by following the webkit review process.

if that process is creaking at the seams and causing "stress" to the developers following it, and you can think of a way that the developers can cope without "blaming luke leighton", please do make it known.

MSHTML port of pyjamas-desktop

... is almost there, already! it's only been about a week, and i'm practically done. IWebBrowser2, IHTMLDocument, IHTMLElement and XMLHttpRequest event handling all work, amazingly.

yes, you heard that right - i managed to work out how to use the COM object XMLHttpRequest, from python, so that you could get asynchronous callbacks to python code just as you would under javascript, by setting the onreadystatechange property!

the key to that one was to create a VARIANT with a COM object (in python) that had an IDispatch interface on it. a little bit of fricking about in the Dispatch_Invoke method (which is what gets called by the XMLHttpRequest handler) and i eventually got successful callbacks.

the code's all a bit of a mess at the moment, but it's there!

http://pyjamas.svn.sourceforge.net/viewvc/pyjamas/trunk/pyjd/

614 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!