Page Templates

Having been part of some significant work using DTML (the mojonation web UI used it as well as some more complex internal UIs we developed) this sounds like a very sane way of dealing with a good set of templating functionality. I know next to nothing about html editors other than vim. Are they truely so good as to actually parse the html and keep all attributes and ordering?

Also, not having tried it, an implementation question. My first instinct to implement a server side PageTemplate processor would be to parse the HTML using a DOM, walking the tree searching for attributes in the "tal" namespace in the appropriate operations order performing whatever simple tasks they require. That said, the "problem" i see with this is that it may not work well for badly formed html (missing closing tags, etc..). Or should that be considered a problem these days? [i'd say no]

Another good comment is that most of the functionality did not seem python specific, which may be good to prevent scaring off the perl fanatics out there from ever considering it.

It is possible to write Tcl code for systems like mod_dtcl so that it looks to the HTML author as if they just have some extra tags. It's very easy to create a procedure like 'navbar' and call it as <? navbar ?>. This makes it fast, easy, simple and unobtrusive. It also lets you change your coding style from quick'n'dirty mixes of html and code, to very clean MVC style web applications. See tcl.apache.org for more information.

I've heard good reports that most mainstream HTML editors, like Dreamweaver, GoLive, and Amaya, retain the extra attributes.

Evan, Guido, and the gang also had the instinct to start with a pure XML to DOM processor. Then they discovered that there is just too much badly formed HTML to ignore. Somehow (I hear a time machine was involved) in a matter of weeks they came up with an implementation that can handle less-than-perfect HTML with ease. There are now basically two parsers, one for XML and one for classic HTML.

Page templates, like XMLC, are careful to not add any new tags. It's difficult for HTML editing software to retain unknown tags, while it's easier to retain unknown attributes. So if you add new tags to a page created by a designer, the page becomes "code" and the designer can't safely change it again. If you only add attributes, you get a clean workflow cycle.

Like splork mentioned, page templates are not Python specific. In fact, there is an extensible expression syntax (called TALES) that lets you add bindings for any language.

I have worked quite a bit with different page generation languages, in the context of a large financial web site, and I can tell you that the "separate content from logic from presentation" strategy is inadequate.

Instead, what you really want is separation of roles. In a typical commercial web site, there may be a large number of such roles: Artist, Infrastructure Developer, Business Rules Developer, Translator/Localizer, Security Specialist, Product Manager, Marketing Manager, Content Developer, etc.

The problem with current technologies is that all of these concerns are deeply tangled within the structure of a typical ASP or JSP page. For example, in one page I examined, there was an input form field which had an "onclick" handler written by the engineers (to validate the data in the input field) and an "onfocus" handler which was written by the HTML developers to provide an attractive roll-over explaining the purpose of the input field. Thus, even within the same HTML tag different roles and concerns can be involved.

What is needed is the HTML equivalent of Xerox's "Aspect-oriented programming" strategy, in which "cross-cutting concerns" are woven together from separate documents via an automated tool.

I've done this. About 4 years ago, I hacked GNUJSP to do exactly what is being described - that is, have a markup language that is valid HTML so that web designers can preview their pages in popular web editing tools. We used it for about 3 months before abandoning it for straight JSP.

This approach seems like a good idea, but it has a number of serious problems. The biggest problem is that it assumes that the static structure of a web page is in any way similar to how that web page is going to look when displayed dynamically.

However, most large web sites use the notion of "components" - that is, re-usable fragments of dynamic HTML which are assembled to form a complete page.

For example, take a typical page that displays a list of top 10 selling movies on a site that sells VHS tapes. The table is, obviously, dynamically generated. In order to "preview" this page in an HTML editor, the artists needs to create a dummy version of that table. That's a lot of work, and it's hard to maintain - every time the code to generate the dynamic table changes, the artist has to change the dummy HTML to stay in synch.

I see from reading the proposal that the Page Template language allows an artist to create a single row of the table as a template and then specify that this template "repeats". However, in the "preview" mode, that is only going to produce a table with a single row. That isn't going to allow the artist to see what the template looks like when it is filled with real data.

Similarly, there might be a "search" component which is used on every web page. This component cannot really be "previewed" in a separate browser window, it can only be artistically evaluated in context, i.e. embedded in a larger page. The HTML for the search box is a fragment - it does not have a HEAD or BODY tag for example, and it "expects" to be inside of a table cell or other container. You can't simply load it into a browser and look at it.

Conversly, the page into which the search box is to be embedded also cannot be previewed, because there is no way in native HTML to "include" an HTML fragment on the client side.

What I have experienced is that HTML pages for industrial-strength web sites like Amazon is made up of a myriad of dynamically generated elements - there is very little static structure to the page at all. Thus, it is only useful to preview the dynamic elements. Previewing the static structure of the page is not in the least bit helpful, and merely serves to make the artists job harder by distracting them with "dummy" content that makes the template more verbose.

What we ended up doing was to give each and every artist their own copy of the web server and application server. It took a bit of training to teach them how to edit pages and to start and stop the server, but once we did this they were very happy with the ability to to preview the dynamic aspects of their art.

Now, I will give you a hint as to what might be really useful and powerful in a markup language: HTML lambda functions. In other words, the ability to take a fragment of HTML (with embedded dynamic elements) and be able to pass a reference to that fragment to some agent or iterator or other processing element, which can re-evaluate that fragment in different contexts. The processing element could bind different variables to the namespace, and then "invoke" the lambda function, which would in turn invoke each of the static and dynamic elements within that fragment in the context of that variable binding.

Moreover, these lambda functions could in turn be passed to other lambda functions as arguments. For example, you could have a standard "decorative border" function which takes a "content producer" function as an argument - the content producer generates the content that is surrounded by the decorative border.

What this allows is the separation of roles idea I talked about earlier - the artists can be responsible for creating the "decorative border" component, while the content developers can work on the "content generator" component.

There was a lot of back and forth about various templating possiblities many months ago on the Webware list, and we thought about several of the possibilities.

While ZPT does not conflict with WYSIWYG tools, I don't think it works well with WYSIWYG tools. I say that, because while a programmer can add ZPT to a page designed by someone, the user (of the editor) does not have any good way to access that logic.

Code kept in HTML attributes is largely inaccessible to the user -- they can't see it, they won't know if they've deleted it, and they'll have a really hard time putting it in. Those users are capable of doing some programming -- the most simple and most common case is simply indicating where certain values should go (like username, status message, etc). In fact, one of ZPT's big detractions, in my mind, is just how difficult it is just to do that simple insertion (<span tal:replace="username">username</span>, if I remember correctly).

One of the great things about WYSIWYG editors is that they are entirely concrete. Now it's not possible to make a dynamic page entirely concrete, but it is nice to at least make the logic visible.

And, of course, ZPT is connected to HTML and XML. But just about every significant application will also have to produce CSS, Javascript, and email messages. Maybe this is part of the transition you've already made from XML to HTML, and eventually to plain text. It's a significant problem, but a solvable one. In the same way, by putting in a non-HTML-attribute syntax, you can potentially make ZPT more concrete. I don't want to seem too negative -- there are some really good parts to ZPT, particularly when changing IMG SRC value, which will piss off the WYSIWYG editor badly in most other systems.

In Cheetah, the system we eventually thought was best, we use a syntax related to WebMacro and Velocity (both Java-based). I think we've made some improvements -- particularly in having Cheetah documents compile to Python class definitions, as opposed to merely having each document be a instance with a fixed interface.

Hopefully we can add some more WYSIWYG-friendly features -- one I've considered is something like <table> <tr> <td> #for $x in $y around tag tr# (text to repeat over) #end for# </td> </tr> </table> for the challenging how-to-loop-over-rows problem. While it's possible in ZPT, it's not friendly there either -- there's no way inside the WYSIWYG to access a TR tag, since it's an abstract part of the layout.

The other half of the WYSIWYG experience that neither ZPT nor Cheetah deals with, is working well with content-producers. It would be nice if such a person could be given the access information (FTP, WebDAV, whatever) and some very basic guidelines, and they'd be able to create a page that fits well into the site. In another project, I've taken content pages and stripped them down to an object with title, keywords, description, and body, and then (more forcefully) inserted them into some other template. That basic pattern seems like a very common one, but not really one that is well supported by any framework I've yet seen. But for all the static pages on a website, it seems very much like the simplist case. METAL deals some with this (and I think addresses some of Talin's objections), but doesn't deal with the basic infrastructure to make this as simple as it should be. (Though it's not really a template issue, but an important publishing issue)

Talin, the designers also wrestled over the question of dynamically generated elements. They came up with METAL, a part of the Page Templates spec, as ianb mentioned. It lets you create site templates using fully valid HTML. Hopefully the next PageTemplates tutorial will explain METAL in detail, but for now here's the Wiki page:

http://dev.zope.org/Wikis/DevSite/Projects/ZPT/METAL

Talin, you're right that a template language isn't everything you need to separate roles, but it is an essential element. Consider the interaction of DTML authors and Zope: although Zope has role management, HTML designers and programmers were forced to step on each other's work. ZPT seeks to better separate the roles. ianb, you argued for allowing a less rigid separation of the roles, and perhaps steps can be made in that direction, but they should be taken carefully.

Talin and ianb, have you taken a close look at Zope? It has role-based security management and FTP/WebDAV authoring.

I have seen rumour of a feature in IE 5.0beta that allowed rich input in a TEXTAREA-like field (called HTMLAREA). Over 50% of cases where I might use something like a template (or about 80% of cases where I use TEXTAREA) could have been replaced with HTMLAREA.

To explain, HTMLAREA allowed for rich text input -- bold, italic, etc. -- and returned that text with HTML markup. As far as I know, there's no even half-way decent way to do this currently. You can have people enter HTML (like here), which is hard to train. You can make a easier markup, like Wiki, which is a bit easier to train but not standard, so it tends to require retraining. You can do uploads, which is far more heavyweight than an HTML form.

It's a total bummer that something like this doesn't exist. In many applications this would significantly help distribute the work among the proper roles, by allowing fine-grained editting of data. Editting a full page is awkward, so you wouldn't want to fire up your HTML WYSIWYG editor just to edit one product description. OTOH, a good website will have rich content, which means TEXTAREA can be kind of lame. Having everyone learn HTML means cutting down the accessibility of the maintenance, which will inevitably lead to some people only working through HTML-savy proxy people.

It's such a small problem, such a simple solution -- which is why it's so damned annoying that we're still stuck with TEXTAREA.

(PS: hathawsh, yes I have used Zope, but ultimately was dissatisfied with it. It didn't let me be a good Python programmer. It demanded I be a good Zope programmer, and I like Python more, and feel Python is better designed than Zope -- hence Webware)

I thinks best way templates should work is seamless for designer.
He mustn't think about permissions or code integrity.
He need just press "Edit page" on site (so get result of scripts) and edit it. Then press "Publish" or soething like this and site-engien must study what changes was made (same way like diff, but character based), find out actual data (which was not changed by designer - only color/style etc.) and apply only thouse changes , which designer was allowed to do.

May be he could be allowed to make information posts this way... or corrections to site data.
It whould be the easies way (of course the easiest still to integrate knowlage about dynamic data content into EDIT tool, but it is not possible for none GPL tools.... May be Mozilla/Editor will help with this (submit bug until 1.0 out)

It might be useful to expound a little bit on how the various "roles" I mentioned interact.

For example, a translator or localizer should be able to modify the text, images, and general appearance of the page, without allowing them access to the underlying functionality. A lot of translation tasks are farmed out to specialized translation houses, which are in turn composed of many individuals located around the world. If you are a company like Wells Fargo, for example, you don't want to have to do a security audit on every translated page that comes back from the translation house! Thus, we need a way to separate the localizable content into a separate document from the functional elements, such as database field validation code, some of which may be in Javascript. (Typically, in large commercial web sites, input fields are validated once on the client using Javascript, for quick, responsive feedback to the user, and once again on the server to maintain database integrity.)

Note that a translator needs access to more than just the text - in some countries, for example, particular background colors (black for example) may be interpreted in ways that are very unlike how a Western person would interpret them.

Another way in a business might want to modify page appearance is "themes", of which there are two commonly occuring types: Promotional themes (such as the various Google holiday themes that you see), as well as user-customized page appearance (as seen in the my.yahoo.com home pages.)

Business Process rules is another interesting area. The company for which I worked hosted a single, web-based product for a number of different banks. When a customer logged in, they would be presented with a version of the product that was "themed" for that bank - a Wells Fargo customer would see stagecoaches and squarish red/yellow borders, whereas a Citibank customer might see rounded teal borders and a citibank logo. These large clients (we called them "brands") not only had editorial control over the appearance of the page, but the functionality as well. Often times they would request changes to specific phrasings of text - we decided to handle that as a translation problem, that there was a "Citibank" catalog which was different from the "Wells Fargo" catalog. This especially made sense, since the actual glossary of the product (i.e. what the various elements were called) was different for each brand.

As another example, one brand might allow the user's login name to be stored as a persistent cookie on the client, but not their password; Others might not allow the username to be stored at all. On this site (which was a shopping site), various "features" of the site would be turned on or off for some brands and not for others. For example, in the "list of stores" page, the store table might have 5, 6, or 7 columns based on whether that bank had elected to subscribe to a product rating service or similar feature. The list of "features" was of course constantly changing, as the agreements between the brands changed, and so what we needed was a way to manage the policies of the various brands.

(All of this was supported using a single underlying application engine which served all of the needs of the various brands. There was a lot of pressure by clients and management to "fork" the application so that brand-specific customizations could be made easier, but the development staff resisted that because they didn't want to have to maintain all of those different forks.)

Security was another interesting role. Because this was a site that dealt with credit cards, we had to be very careful about security, in many cases going beyond the standard security mechanisms - for example, we used NCipher boxes to encrypt the credit cards in the database, as well as many of the browser cookies that were used for session management. Almost every HTML page had some security-specific code embedded in it. Often times a new customer would come to the site, and they would only give us partial information about themselves, or none at all - so we were careful to insure that they could use as much of the features of the site as possible, without exposing any data that might require a complete authentication.

Another interesting security issue that we started dealing with before I left was single-sign-on, the ability to log in at, say, the bank's own web site and then hyperlink to our bank-branded site without having to log in again.

That may seem rather esoteric, but there are a lot of businesses that deal with credit card data who need this kind of security (Did I mention that our ISP was located in a physically impregnable building? :)

As part of this site, we had a downloadable component that sat in your toolbar (Windows and Mac were supported), and there was various Javascript in the HEAD of each page to communicate with this component. This was another role aspect that had to be integrated into the site. (For example, you could log to the site via the component - this was used to support "smart card" authentication, for example.)

Database schema was another interesting area. I always felt that a database scheme should be a separate document - that is, an HTML designer should only specify the positioning of the form elements, and that the database schema should determine the mapping between form elements and database fields.

In my vision, an HTML developer would create a series of "style" objects that could generate checkboxes, radio buttons, input text boxes and other elements in the appropriate visual style. This style object implements a "factory pattern" in that it can create new elements based on an input specification. At page generation time, the "style" object would be selected based on the attributes of the current logged in user (via the session object). The style object would then be passed to the "database schema" object which would then invoke the factory to create the various input fields. The schem would pass as arguments to the style factory any schema-specific information (such as data needed to generate Javascript onclick handlers for data validation.) Upon form submission, the POST results would then be passed back to the database schema object which would then validate the results and update the database as needed.

Thus, if you had an input field for "quanitity ordered", the only thing that the page designer has to do is say "put a 'quantity_ordered' field here". He does not have to create the Javascript rollover for explaining what a quantity orderered is; He does not have to put in the code to insure that the text being entered is a positive integer; He does not have to put any fancy decorations or borders around the input field to be consistent with the look and feel of the site.

The idea is that there are three roles involved: The person who is designing the layout of the specific page (and deciding which data is to be displayed or entered), the person who is designing the graphical "look and feel" of the entire web site, and the person who is deciding the mapping between the exposed data fields and what is actually stored in the database. Each of those people should be working on a separate document.

I did the same within the 0.4 version of Askemos.

To be even more safe from those editors messing, I abused the class attribute to hold a replacement XSLT tage name. (It was probably even valid architecturl form, if there was any declaration for it.) That way we had an example element, which got replaced - to indicate the effect somehow to the designer.

The experience is: it's not worth it. We gradually changed back to write XSLT styles sheets and we still have a superflous feature eating up runtime now.

The WYSIWYG paradigm I believe involves two basic assumptions:

• That the developer of a piece of content is able to see, in real- time, the visual effects of a particular design modification.
• That the operations on the document are all fundamentally visual operations, that is, operations on appearance, layout and style.

In other words, I think that in order to be truly WYSIWYG, the content developer need not concern themselves with the abstractions that lie behind the visual presentation. They operate directly on the visual presentation and never need to penetrate beneath it.

However, in a highly dynamic system, it really isn't possible to visually present any more than a snapshot of the dynamic system at any given moment. In a highly dynamic system, you really do want to expose to the designer the abstract mechanisms behind the facade - because those abstract elements are what defines the dynamics. It means that the designer has to think abstractly, and must operate on that level instead of the level of superficial appearances.

An example of this is building interfaces using Java AWT (or Qt or wxWindows or any GUI toolkit that supports dynamic layout.) Simply dragging and dropping the widgets in to place is insufficient - the various visual editors all have extensive support for examining the properties of the widgets and setting their layout parameters.

This doesn't mean that the abstractions need to be highly technical - for example, most artists don't need much explanation to understand the Java AWT layout algorithm.

So my feeling is that in a page template language, what is really needed is a way to simplify the dynamic abstractions to the point where they are easily understood by an artist.

No-one else seems to have mentiond that Perl has had this for some time (sorry if they have and I missed it on my skim through the comments).

http://search.cpan.org/doc/SAMTREGAR/HTML-Template-2.4/Template.pm

I started working at a new job a few months ago and was introduced to this perl module - and I'm a total convert, it makes it so much easier to make your pages compliant (I've just finished the initial development of an xhtml 1.0 strict compliant slash clone) and having the content engine and database handling seperated from the interface code makes life a lot simpler when debugging etc (sure, I could have achieved this with lots of small functions, but this way seems neater to me).

Regards,
Denny

PS : although the tags look ugly in the example, it is actually possible to embed them in HTML comments, making it possible to validate your templates with weblint or similar tools if you wish.

There's a seemingly fairly popular Perl package called the Template Toolkit.

We use it at work, and I haven't yet found another templating system (including Mason, HTML::Template and several home-grown ones) that I like even nearly as much.

It's basically Turing-complete, and speaks to Perl datastructures without worrying about any kind of marshalling (caveat: passing objects to templates can be bad for security). This means that the programmer can pass a list of, for example, (name,id) records to the template, and the template itself can transform that into a three-column list where each cell is something like <a href="/calatogue/[ID]">[NAME]<a>.

Note that the major difference between ZPT and the Template Toolkit is that ZPT defines no "new" HTML tags. All the templating happens in HTML/XML attributes which makes it reasonable for HTML designers and programmers to work on the same template without stepping on each others toes.

Note also that this is the major difference between ZPT and DTML, Zope's current "in-HTML" scripting language. ZPT will eventually replace DTML for lots of applications. The approach used by Template Toolkit is the same approach used by DTML.

Now, I will give you a hint as to what might be really useful and powerful in a markup language: HTML lambda functions. In other words, the ability to take a fragment of HTML (with embedded dynamic elements) and be able to pass a reference to that fragment to some agent or iterator or other processing element, which can re-evaluate that fragment in different contexts.

ianb said:

In another project, I've taken content pages and stripped them down to an object with title, keywords, description, and body, and then (more forcefully) inserted them into some other template. That basic pattern seems like a very common one, but not really one that is well supported by any framework I've yet seen.

It might be worth looking at WebMake; it's a site-building framework which handles both these cases in a very simple fashion. The former is used as a standard technique for building up catalogs (sitemaps, index pages, etc.), and the latter is the basic building block of a typical site. It's very flexible in general, but those two cases are the most common.

(Disclaimer: I'm the WebMake maintainer.)

While I'm commenting, Talin mentioned this:

The table is, obviously, dynamically generated. In order to "preview" this page in an HTML editor, the artists needs to create a dummy version of that table. That's a lot of work, and it's hard to maintain - every time the code to generate the dynamic table changes, the artist has to change the dummy HTML to stay in synch.

It sounds like it would be handy for the designer to be able to specify the "each line of table" template within the main template (probably delimited by some tags that are ignored by the page-editing tool). Would that make sense?

I've used and written template languages that use <custom-template-tag> constructs, where the language is intended as an extension of HTML. It is, when first looking at the problem, the intuitive thing to do. However, I think it is totally wrong.

In languages like ZPT and XSLT, the structure of the HTML is important to the structure of the language itself. Here it makes sense to make more tags (as in XSLT) or piggy-back on tags (as in ZPT).

However, most template languages actually work on straight ASCII, and just happen to be used together with HTML. In reality, the structure of the template language is completely orthogonal to the structure of the HTML code. The thing that hurts me deep down inside is when I see something like <img src="<var imglocation>">. That's bad HTML -- very bad HTML. If that var tag starts taking HTML-like attributes, with quotes and everything (as it does in DTML), then it becomes very very bad HTML.

And the thing is -- there's no reason you have to use < and > in these languages. Usually [] or {} would work just as well, and they would create valid HTML. You could preview them happily in your browser -- maybe with broken IMG tags, but otherwise approaching valid.

So anyone who's planning to write a templating language, please, do not follow the pack in this one, small, yet important detail!

The thing that hurts me deep down inside is when I see something like <img src="<var imglocation>">. That's bad HTML -- very bad HTML.

Well said! It just doesn't look right, it looks like a bug in your HTML, and editors will barf on it.

IMO, it means the language designers didn't think it through. It should have been obvious that there needs to be a way to use var references inside an attribute. It should also have been obvious that editors (and users!) would barf upon seeing a nested < like that.

And anyway, what's the point of designing the language to use XML-style tags, if they're not valid XML?? For example that example above should really be <var name="imglocation" />.

So anyone who's planning to write a templating language, please, do not follow the pack in this one, small, yet important detail!

(cough) WebMake doesn't ;)

I'm still amuzed - why not to use PHP or Perl?!
Look here.... we could make HTML template with say [-[IMGname]-] variable
Next we just parse it one time to replace:

       $line=str_replace ("[-[","<?",$line);
       $line=str_replace ("]-]","()?>",$line);

And concerning > and < - avoiding them is useless! What it whould do to you if you know that it is correct HTML? Still you can't see image (if there is variable name instead of image) - so WYSIWYG will do you no good.

Developer need to have there example of real data. And this real data should be changed to what you need by Template engine....
So <img name="img/aaa.gif" width=12 height=11> should be parsed and changed to any <img name="img/bbb.gif" width=123 height=32> you need (aaa.gif - real image on developers hdd)

I agree, and in fact I've semi-designed and semi- implemented (but never released) a couple of template processing systems (first in Perl, then C, with a side trip in VBA...) which specifically didn't use < and > because I wanted to be able to use (and see) the new tags in WYSIWIG editors.

Although it would be possible to use, say, { and } or [ and ], one problem with that approach is they are more likely to be used in content and thus you run into problems when parsing the templates correctly (if you take a multi- pass approach anyway).

My tag delimiters of choice are {! and !} which seem to tend not to occur naturally...

Of course the disadvantage to this approach (i.e. multi- character delimiters) is it can complicate the coding, but still I think that's a small price to pay...