Older blog entries for lkcl (starting at number 522)

you know what? i'm bored of advogato not having more stuff posted on it. y'know. original stuff. interesting stuff. stuff not posted by me. seven years, now. last time i checked i can't remember but i think it was fully 10% of the articles on advogato were written by me. well, _that_ increased the quality and sum knowledge of free software a lot, then didn't it :)

why we should use the term "free software" not "open source" .

It took some digging to find that this is not a free software conference.

"Open Source" in the intelligence community means "a source of information and/org intelligence that is beyond our control". in other words, it's a potential nightmare. an information leak. a source of information that is "open". a source of information that cannot be contained. cannot be restricted.

ha ha :)

quad - the reason why machines at ibm/lenovo are sent to "configuration" is because the BIOS is custom-built on a per-machine basis, and will include in it the serial numbers of all the peripherals.

in this way, they ensure that things like the wifi, 3G modem etc. cannot be replaced.

by you.

various reports on how to re-flash your entire BIOS, removing the restrictions, can be found on the internet.

holy cow. i announced pyjamas 0.3 release and the sourceforge page reached 2,000 web hits in about four hours.

wow. you know when sometimes you accidentally encounter something, and it's so simple that it's easy to miss its potential - pyv8 is something like that. who would have thought that something as obtuse as converting python to javascript, with only about 1200 lines of code, would result in such a damn dramatic performance increase in the execution speed of python programs?

iiiinteresting. one of the people on the pyjamas google group has just got V8 to work from python, and - get this: passed the output from the python-to-javascript compiler... to V8.

so, take python. compile to javascript. hand javascript to a blindingly-quick javascript compiler - what do you get?

a 10 times performance increase in the speed of execution when compared to the python interpreter.

django and joomla with jsonrpc

well *phew* thank goodness for django, is all i can say. i spent today updating a joomla site into an experimental joomla-django-pyjamas hybrid.

it's not pretty :)

step 1 - the django model

legacy databases says that your key is "python manage.py inspectdb". that gives you a basic start and, depending on the quality of the design of the database, you get a reasonable approximation.

i found three errors in the joomla database i'm working with: three tables didn't have primary keys, but one _was_ identified as "unique", and they all did have "id" as the first field (being django).

ok - i say three errors - but i would consider not having any foreign keys - at all - to be an absolutely massive error - and serious problem.

this is mysql - there's not a lot that can be done about it...

step 0

whoops - first: install django (from svn), django-evolution (absolutely essential) and libcommonDjango. libcommonDjango contains the essential JSONrpc service.

step 2 - jsonrpc django view

create a django fooapp/events/view.py:

from django.pimentech.network import *
from testapp.models import Events

 eventservice = JSONRPCService()

 @jsonremote(eventservice)
def echo(request, event_id, test_txt):
    try:
        e = Events.objects.get(id=event_id)
        event_desc = e.description
    except Events.DoesNotExist:
        event_desc = ''
    return "hello %s %s" % (test_txt, event_desc)

urls.py gets this:

(r'^event-service/$', 'fooapp.events.views.eventservice'),

step 3 - pyjamas integration in django

this is where it gets tricky. first thing: i'm doing a hybrid, so the pyjamas code will only be loaded on a per-component basis. i.e. on a _specific_ joomla page, which throws up a _specific_ joomla component (in this case, com_event), _only_ then do we want a specific (matching) pyjamas module to be activated.

pyjamas starts off by overriding the javascript body "onload" function, and setting up "modules" which the code looks for in the "meta" tags. so, the only way i could think of getting the modules activated at the right time was to parse the url arguments, looking for the query string "?option=com_event":

/*
 ***** Pyjamas Integration ******
 *
 integrate with pyjamas modules by adding the appropriate
 meta tags, here.  inclusion of the auto-generated pygwt.js
 will go hunting through the meta tags looking for names
 "pygwt:module" and will call the appropriate onModuleLoad
 depending on the content="moduleName".
*/
$args = array();
$args['option'] = '';
$args['view'] = '';
parse_str($_SERVER['QUERY_STRING'], $args);
/* this is gonna be fun.  detect component, also must detect
view */
if ($args['option'] == 'com_events')
{
    echo '<meta name="pygwt:module" content="Test">';
}

as you can see, i'm preparing to load up _different_ pyjamas modules, depending on whether users select different views - so that'll be if ($args['option'] == 'com_events' && args['view'] == 'all') load a different pygwt module.

this bit of awfulness needs to go into your django template's index.php file - right at the top, so that it gets outputted in the <head> section.

then, you're ready to add, to your components/com_events/views/NAMEOFVIEW/tmpl/default.php file the pygwt.js application (which will do the module loading):

<script language="javascript"
src="/js/Events/test/pygwt.js"></script>
<h2>Testing</h2>
<div id="event"> </div>
<h2>End Testing</h2>

you _could_ add this in along with that meta tag thing... but... i decided against it. the div tag will be explained at the end of the next step

step 4 - building the pyjamas app

i've modified build.py accordingly so that the name of the app is included in the per-platform javascript files. see lines 150 to 154 - this results in Test.IE6.cache.html, Test.Mozilla.cache.html and you can thus copy those files into templates/yourapp/ with impunity. i may at some point make it possible to install them in a subdirectory, but if you want to do that yourself manually you'll need to edit pygwt.js.

the test example i chose was the JSONRPC example, renaming it to... Test.py. just a couple of tweaks needed:

class EchoServicePython(JSONProxy):
    def __init__(self):
        JSONProxy.__init__(self, "/event-service/", ["echo",
"reverse", "uppercase", "lowercase"])

/event-service/ matches our django urls.py, above.

import Window
...
...

     else:
        if method == self.METHOD_ECHO:
            loc = Window.getLocation()
            event_id = loc.getSearchVar('id')
            id = self.remote_py.echo(event_id, text, self)

 

alert readers will have noticed that fooapp/events/view.py's "echo" function (step 2, above) had an extra argument - an event_id. here what we're doing is pulling the django-driven "&id=NNN" from the query string, using Window.getLocation() to parse the query string.

last modification:

        RootPanel("event").add(panel)

this is veery special - it results in the AJAX code looking in the HTML DOM model for a tag with an id of "event". this is incredibly, incredibly useful and powerful stuff: it means that you can write HTML and then, when standard HTML is just... too simple, go adding fully-blown AJAX applications into the basic HTML, at the right place.

write your stub template in HTML, get your designer to do it; write your application in python, get your programmer to do _that_, hit "make" and the app gets turned into javascript....

step 5 - installing the pyjamas app

this required that bit of trickery - modifying build.py - so that when it comes to adding more than one pyjamas app to be loaded at different points, the code can be loaded without name clashes (Safari.cache.html for Test.py, Safari.cache.html for Test2.py):

all: build install

 build:
        python ~/pyjamas/src/builder/build.py Test.py

 install:
        mkdir -p ../../../www/js/Events/test
        cp output/pygwt.js ../../../www/js/Events/test
        cp output/*.html ../../../www/

so - the modified build.py generates Test.Mozilla.cache.html and Test.Safari.cache.html not Safari.cache.html and thus output/Test.*.html can all be copied into the main www directory. pygwt.js goes into the js/Events/test subdirectory, matching where it was specified to be in the last part of (step 3).

watch it all fall over

well... surprisingly... it didn't fall over. the sticky-tape technique worked. slight down-side: loading the php page takes the usual amount of time... and then .... mmmmmm BLAM, up comes the javascript app bit, after another second or so.

not a lot that can be done about that - might put "Loading..." into that div or something.

overall, i'm just really relieved. the hybrid between django and joomla turned out to be easier than i thought it might be - thanks to django's "inspectdb" functionality.

i know for a fact that i'm going to have to watch the way that django-evolution is used like a hawk - i've already changed the name of one of the model classes and, perfectly legitimately, django-evolution wiped that table off the mysql map and recreated it (blank) although it had the same "Meta" mysql table name. a backup of the database fixed that one by restoring the table.

this... this should be good.

a css dog is for life, not just for christmas

well, dang me. after promising never to do CSS / HTML cross-browser nastiness ever again, i kinda have to qualify that by saying, "after promising never to INITIATE ...." :)

i've an opportunity to work on a quite exciting web site - exciting in the aims and goals that the site is to be used for, rather than "the technology used" is cool.

for convenience of getting started, and also because it fulfils much of the client's requirements, joomla, the dog from hell, was used. in may ways, this is a good choice: it offers many suitable plugins, it's php, it's well-understood... etc.

where it all goes horribly wrong is the prevalent use of CSS stylesheeting and the fact that many of the plugins are unfinished, tested against certain browsers and not others (prominent use of min-width for example) and generally fall to pieces when you try and do anything "outside of the box".

the seyret video plugin is a good example.

i've wasted three to four days of the client's time and money dealing with the fact that seyret - the only decent youtube video integration plugin for joomla - can realistically only cope with exceptionally wide screens.

one day was spent tracking down the fact that words displayed in a 200px wide column do not "wrap", causing a cascade of CSS style errors up, finally, to the body of the page which was pushed well outside of the 100% screen-width boundary - and - once this was isolated, a further three hours was spent creating a text-wrapper in php to split words down, at white-space boundaries, into invisible DIV tags containing at most eight letters:

    # split words up into 8-group-long floating divs (!!!!!!)
    # this gives the text a chance to "wrap" neatly, should it
    # be required.
    # the "mess" you see before you is because the strings are
    # UTF-8 encoded.

     $wlen = 0;
    $vcrep = "";
    $word = "";
    for ($i=0; $i < mb_strlen($vcomment, 'UTF-8'); $i++)
    {
        $ltr = mb_substr($vcomment, $i, 1, 'UTF-8');
        if ($ltr == ' ' || $ltr == '\t' || $ltr == '\r' ||
$ltr == '\n')
        {
            if (mb_strlen($word) > 0)
            {
                $vcrep.= "<div
style=\"float:left;\">".$word."</div>";
                $word = "";
            }
            if ($ltr == ' ')
                $ltr = " ";
            $vcrep.= "<div
style=\"float:left;\">".$ltr."</div>";
            continue;
        }
        $word .= $ltr;
        if (mb_strlen($word) > 8 || $whitespace)
        {
            $vcrep.= "<div
style=\"float:left;\">".$word."</div>";
            $word = "";
        }
    }
    if (strlen($word) > 0)
    {
        $vcrep.= "<div
style=\"float:left;\">".$word."</div>";
     }
    $vcomment = $vcrep;

that having been solved, it still didn't help - fully - because there were other locations with similar css nastiness.

so, the whole "lovely" css styling had to be ripped out and replaced with a monster table - an 8x7 grid with colspans and rowspans to contain the top-left, top-right etc. "round-cornered" images - rowspan down the left, rowspan down the right - you name it, i done it.

this abominal procedure having been completed, it then highlighted IE6-only limitations, FF3 limitations, FF2 limitations... all different.

it makes me wonder why anyone does css/html at all.

so - i've got through most of them, now, yet it's been four days and i've still effectively "not started".

pyjamas to joomla

... so i'm looking forward to getting out of cross-browser manual-html-hell and into the javascript-only pyjamas loveliness.

one extreeeemly useful pyjamas class, without which i wouldn't even be able to contemplate doing what i intend to do - is HTMLTable. this beautiful life-saving class is capable of embedding widgets into pre-existing HTML. all you have to do is put in a span, with an id, and then when you "add" a widget to an HTMLTable, you specify the span's id.

also, the RootPanel() function can be passed in an "id" argument: you end up automagically getting back a widget which isn't "the entire html screen" - body - but is in fact a wrapper around the div with that unique DOM id. once you've got that, you can start adding stuff.

in this way, i have a means by which i can "tack on" stuff onto the php/joomla nastiness, and thus do what i need to do far quicker and with less stress. not least because the cross-browser tricks - if only in the code that i write - will all be taken care of by the pyjamas underlying AJAX library.

whew :)

the fun bit is going to be whether i decide to use django - and do a joomla-django interface (model) - or whether to risk going mad by writing the back-end (model) code in php (JSONrpc).

will have to think about that one...

we have a bit of an issue.

highlighted recently was some advice i was given, tying in with things i've come to appreciate and understand about myself and what i'm capable of.

to say that i have the classic symptoms of asperger's is missing the point. a wired magazine article i read on autism and aspergers describes the people in it as being "above-average-speed". the article mentioned that the "classic" IQ test is one of "reasoning and articulation" - the focus being on "being able to articulate" - communicate, if you will.

when people with asperger's syndrome and high-functioning autism are tested with the standard IQ test, they typically score very low. this was a serious problem resulting in many people being placed in "remedial schools" etc.

when tested with the more comprehensive, more thorough and less biased IQ test, people with asperger's and high-functioning autism score _extremely_ high.

the reason: their brains are running at much faster speeds than the average person. they are able to think things through far faster.

that's where the reasoning and scope of the wired magazine article ended.

my guess is that the faster "speed" of an asperger's syndrom or high-functioning autistic individual applies to their entire nervous system (and this was borne out by some interesting equipment costing $25,000 i was allowed access to for a couple of hours).

the problem with a faster nervous system is that many parts of the human body - including bits of brain - simply aren't geared to _run_ at the speeds at which other bits can run.

like a radio, mis-tuned ever so slightly, the asperger's individual's psyche cannot run "at full power" due to this mis-match.

thus you get "symptoms" such as "inability to communicate". why? because such individuals find the "average person" so frustrating to communicate with - not just because that person doesn't "get" their ideas, but also because the asperger's individual's own mouth won't respond fast enough.

thus, as with all humans, the ability to communicate .... atrophies. skills that are damn difficult enough to utilise in the first place simply... don't get practiced, don't get developed...

with the emergence of the internet, however, many people with asperger's and high-functioning autism who would otherwise find it difficult to interact with the world around them let alone other people have an alternative communications outlet.

(i type at 170 words per minute, peak, for periods of up to 40 minutes if i really push my luck.)

the machine i had access to for a few hours measures the "speed" of a person's nervous system. the average person is around 5,000 to 10,000, with exceptional individuals coming in at 20,000. a high-functioning autistic will typically be around 40,000. i came in at 54,000.

in many ways, i find this quite upsetting.

the reason is quite straightforward: not only have i known that i think much faster than most people - solving complex problems that other people just don't have the will-power to stick at for months or years - but now i get actual confirmation of _why_ that's the case.

ability to solve significant technical challenges well beyond the capability of most people doesn't leave you with very many friends.

it's like i'm staring across an immense glass wall, seeing people getting on with their lives - enjoying the comfort of well-paid stable jobs to which they are suited, enjoying the banter of communicating with friends and colleagues.

(this time - from the relative comfort of the home that i don't own, eating the food that i didn't pay for, using the internet service that isn't mine...) i watch the interactions and the challenges they and many others face; i see a particular block which, if opened, would benefit vast numbers of people, and i offer my help and advice. some people find it "cool" and interesting to help out - those people i've always cherished and appreciated. others are bewildered by the absolute non-stop approach of eleven to twelve hour near 100% focus days, and the results it gets; the speed at which i acquire new skills, knowledge, information - and they feel lost, belittled and threatened.

the first time this happened was on a successful commercial project - SpecManager, for Pi Technology. eighteen months of work that resulted in a vehicle simulator that was accurate (bar one known bug which was fixed by my successor) to within something like... 0.5% to 1% of real-world fuel economy figures. the client had asked for .... i think... 2.5% accuracy or something.

as you do, i needed to articulate ideas. however, after about a year, i learned to remain silent. the project leader (there were 2.5 of us on the project - me, the project leader and a contractor in visual basic who designed the initial GUI) would respond with the classic question which i now know means "run away so fast you burn your shoes out" - "surely it doesn't have to be as complicated as all that?".

the second time was the samba project. that was three years, spent delving into machine code and disassembly at times (90 days to produce 60 lines of c code), to get the information needed. the end-result was about 100,000 lines of extraordinarily comprehensive code, and it broke the back of the microsoft monopoly; bridged the gap between unix and windows; helped linux "tick the boxes" in absolutely critical areas for acceptance in corporate and civil service adoption criteria; gave the samba team, the DoJ and the EU anti-trust case the critical ammunition they needed to get the IDL files and other documentation out of microsoft.

but... what happened under linuxcare's watch, in 2000 was, by any standards, absolutely disgraceful. (by not actually bothering to pay me - to the extent that i was working for a company but wasn't on a payroll - for nearly three months didn't help matters, either).

the core of what happened is that - again, i came up with the goods, yet, because i had done so, it could not be "believed". because i found it difficult to communicate what i had done. surely the code stood on its own merit? no - because the key samba developers actually wouldn't even look at it, finding literally any excuse possible to not look at it. because they might have found something that proved that i actually had skills that they didn't have.

sadly, i greatly respected those people and the skills that they had. i learned a great deal from them, by being silent, by listening quietly to their descriptions of c systems programming and other topics on which they _were_ experts.

it's a great pity that they did not return the compliment, instead, treating me like a second class unskilled programmer, aged 17.

this fits the classic description i gave at the beginning of this diary entry: ability to comprehend; lack of ability to articulate...

now, we roll forward to today and yesterday. i want the webkit bindings to succeed. to be _useful_ and useable. to put webkit on the map it deserves to be on.

that means taking a different approach - in particular, it means i have to stop - and let people "catch up". i've gone far enough ahead to know what needs to be done - now it's time for other people to follow, to find out that yes, it _can_ be done.

the webkit glib bindings are _so_ significant a strategic free software project, i can't begin to tell you or describe it. without those bindings, webkit remains "just another web browser" - sure, one that can be customised, but... so what? _with_ the bindings, webkit becomes a vast and powerful applications library and toolkit for the next generation of rich media desktop applications - for _every programming language_, not just the c++ in which webkit is written.

so - how do we get from here to there?

i got a second review of the webkit glib bindings, today. i also looked around and found that some of the more prominent free software projects - and some less well-known but extremely good ones - are using webkit.

the list is like... a who's who of heavy-weight free software projects! epiphany, evolution, midori, the olpc browser, pidgeon - missing from the list is kde but i know that's definitely being done (it's in svn.kde.org).

and there's me, going "coooeeeee!" with pyjamas desktop ha ha

yeah, i got my second review of the glib bindings patch, today, which was good. lots of coding conventions comments etc. and - unfortunately, some comments which... bless mark for making them but i really can't face even an extra day of "extra functionality" or "significant changes" work on the webkit glib bindings.

the auto-generator is written in perl, which is not my favourite language to work with - i don't call it a "WORN" language - write once, read never - for nothing.

i don't mind re-visiting stuff, to improve it. but... extending a 6,500 line patch - one of this level of complexity - makes me nervous. i _never_ go two weeks without committing to a repository. i never go 48 _hours_ without doing a commit.

anyway.

another looong day.