Author: Mike Orr

This article discusses what pylons.templating and Buffet do or should do, and how to implement a plugin for your favorite template language. It does not discuss how to use Mako/Genshi/Cheetah/Stan/etc with Pylons – see the other articles for that.

Template plugins

pylons.templating is based on Buffet, a universal template front end originally written for CherryPy but now used especially by TurboGears. Buffet provides a unified front end for any template engine that implements its plugin interface. The Pylons code is slightly modified from Buffet for reasons described below.

The plugin may be distributed as part of the template package itself or as a separate package. Mako has its own plugin, while Cheetah uses TurboCheetah.

Each plugin is a class with the following methods:

def __init__(self, extra_vars_func=None, options=None)
    # Store the two arguments (normally dicts of strings), and do any other initialization required.

def load_template(self, templatename)
    # Optional: not called by Pylons.  Called by TurboGears in special circumstances; see Buffet interface for
    # details.

def render(self, info, format="html", fragment=False, template=None)
    # Return a rendered version of the template.
    # 'info' is a dict of context variables (string : anything).
    # 'format' is the desired output format.  This is engine-specific.
    # 'fragment' is true if the template should be rendered "standalone"; i.e., bypassing any inherited templates
    #     or wrapper templates that would normally be applied.  The plugin may ignore this argument if the engine
    #     does not support this.
    # 'template' is the template name (string).  Buffet requires dotted notation (see below), while Pylons
    #     allows URI notation.
    # The method should call the 'extra_vars_funcs' function given to the constructor, and merge those
    # variables in with 'info'.

def transform(self, info, template)
    # Optional: not called by Pylons.  Called by TurboGears to render the template as a tree of ElementTree
    # nodes rather than as a string.  Kid is the only known engine that supports this.

The plugin must register one or more entry points for itself.The entry point appears as an argument to setup() in your setup.py:

    entry_points="""\
[python.templating.engines]
name = package.module:class
mako = mako.ext.turbogears:TGPlugin
"""

The plugin must use setuptools rather than distutils in order to set up an entry points. This example registers two template engines, "name" and "mako". It means that when the user specifies "mako", Python instantiates mako.ext.turbogears.TGPlugin. Whether these are two different engines or two configurations of the same engine is immaterial: the API treats them as different engines. Genshi registers "genshi" and "genshi-text" for its two template syntaxes.

Pylons templating initialization

(Based on a prerelease of Pylons 0.9.5, SVN revision 1920, April 2007.)

I've written a general Pylons Execution Analysis that goes through the entire life cycle of an application and a request. This section focuses only on template execution and includes material not in that article.

Template options for all engines may be set in the load_environment() function in myapp/config/environment.py. A list of options recognized by various engines is in the "Template Options" section below.

In the application's myapp.config.middleware module is a line:

config.init_app(global_conf, app_conf, package="myapp")

You may specify a default template engine with the template_engine="FOO" arg, where "FOO" is any plugin entry point. The default default is "pylonsmighty" but is expected to change to "mako" in a future version of Pylons. The default engine is invoked if the controller doesn't specify an engine; i.e., 'render_response("/index.html")' vs 'render_response("mako", "/index.html")'. config.init_app calls:

config.add_template_engine(engine_name, root_directory_or_module, options=None, alias=None)

with a hardcoded set of arguments for mako/myghty/kid/genshi. There's a bug if you specify an engine without hardcoded arguments like "cheetah": it initializes no template engine, which causes an exception later in the intialization.

Nevertheless you can call config.add_template_engine yourself in middleware.py to make additional engines available to your controllers. In this case you have to supply your own arguments. To mimic the hardcoded arguments use:

config.add_template_engine("pylonsmighty", None, {SEE SOURCE})
config.add_template_engine("mako", "")
config.add_template_engine("genshi", "myapp.templates")
config.add_template_engine("kid", "myapp.templates")
config.add_template_engine("cheetah", "myapp.templates")    # TurboCheetah 0.9.5 at least.

add_template_engine merely appends a dict to config.template_engines with the chosen arguments for this engine. The keys are slightly different from the method signature, and 'options' has config.template_options merged into it. The first item in template_engines is the default engine. You can modify template_engines yourself, with or without the convenience of add_template_engine.

The next line in the middleware instantiates a pylons.wsgiapp.PylonsApp called 'app'. Its superclass constructor pylons.wsgiapp.PylonsBaseWSGIApp doth do:

# Initialize Buffet and all our template engines, default engine is the
# first in the template_engines list
def_eng = config.template_engines[0]
self.buffet = pylons.templating.Buffet(
    def_eng['engine'], 
    template_root=def_eng['template_root'],
    **def_eng['template_options']
    )
for e in config.template_engines[1:]:
    self.buffet.prepare(e['engine'], template_root=e['template_root'], 
        alias=e['alias'], **e['template_options'])

This calls pylons.templating.Buffet.prepare() for each engine. pylons.templating has already searched the entry points for all template plugins and put a dict of {engine name (string) : plugin (class)} in pylons.templating.available_engines. So .prepare finds the plugin class and instantiates it, or raises Exception if no such engine is known. So 'options' in the plugin constructor comes from add_template_engine's options, and 'extra_vars_func' comes from a "my_template_engine.extra_vars_func" item in the configuration. (I could not find anything in my libraries or Pylons application that sets 'extra_vars_func', so as far as I can tell it's unused.)

Templates in Pylons controllers

Some time later your application receives a web request. Your controller has two global variables render and render_response, imported from pylons.templating via myapp.lib.base. Your controller method might look like this:

def hello(self):
    # For Mako template myapp/templates/tutorial/hello.html containing "Hello, ${c.who}!"
    c.who = "Mako world"
    return render_response("mako", "/tutorial/hello.html")

Or perhaps like this:

def hello(self):
    # For Genshi template myapp/templates/tutorial/hello.html containing "<...>Hello, ${c.who}!</...>"
    c.who = "Genshi world"
    return render_response("genshi", "tutorial.hello")

In any case you set your data values as attributes of the global c object and invoke the template. Buffet takes care of finding the template, setting up an environment it likes, adding Pylons values, and calling it with the right arguments. This provides a least-common-denominator approach to templating because some engines may provide unique features that can't be accessed this way. For instance, Cheetah can search through multiple objects for the required data values, whereas Buffet provides only the single 'info' dict. Pylons puts the c object itself in the 'info' dict, along with g and h and some other stuff, but to the template engine it's one top-level dictionary.

History, dotted notation vs URI notation, and the future

As we've seen above, Mako (and Myghty) use a URI syntax to specify the template in render_response. Genshi (and Kid, and for now TurboCheetah) use a module syntax: "package.modulename". In either case the path is relative to the "templates" directory. The actual template lookup is done entirely in the plugin's .render and .load_template methods, so they can do anything with the template name string they've been given.

Historically, Kid was TurboGears' only template engine and TG used dotted notation. TG/Kid requires _init_.py in the template directory and all subdirectories. Templates are named templates/a/b.kid but referred to as "a.b". I think it stores a compiled template b.py in the same directory and there's an option "kid.precompiled" to bypass the *.kid files.

Users clamored for other templates, especially Cheetah, and TG adopted Buffet as a common front end to several engines. Buffet was originally written for CherryPy but gotten so identified with TurboGears that its plugin spec is at the TG website, and most plugins have "Turbo" or "turbogears" in their name even though they're being used outside TurboGears. The Cheetah plugin (TurboCheetah) was written by TurboGears founder Kevin Dangoor rather than the Cheetah developers, so it reflects the TurboGears philosophy of the time (i.e., it's similar to Kid).

Pylons also adopted Buffet as a universal template front end, but uses its own modified implementation in order to support URI notation. Pylons' original template engine was Myghty, which uses URI notation. But the template engines evolved out from under the frameworks, and today TurboGears is transitioning to Genshi, and Pylons to Mako. Both of these new engines reflect their predecessors, so the Genshi plugin uses dotted notation while Mako uses URI (though Mako attempts to convert dotted notation if given).

This means two different traditions have grown up for referring to templates. This is not necessarily bad, but it's worth stepping back and asking what's best for the Python templating community as a whole, especially since other frameworks and projects will no doubt adopt Buffet too. Shall we standardize on template names and a common set of options, or let every plugin evolve its own way? Lassez-faire inevitably means that some engines will adhere more to TurboGears' conventions (including its esoteric features), and others to Pylons.

We've already seen that TG requires two extra plugin methods: .load_template() and .transform(). .load_template() is used esoterically, and .transform() is applicable only in a Kid-default environment, or at least only when plugging a Kid result into a larger ElementTree structure. Perhaps this idea of "structured DOM output" should be genericized. On the other hand, DOM output is usable only with another DOM of the same type (Kid -> ElementTree, Genshi -> Genshi stream type, text-based engines -> nothing). So perhaps it should be an "optional extra" in the API.

So, back to dotted vs URI notation. Dotted notation:

Buffet spec requires it; Pylons is agnostic.
Requires _init_.py in "templates" directory and all relevant subdirectories. (This may not be a strict requirement but Genshi/Kid/TurboCheetah all require it.)
Nevertheless, the plugins do not simply do "from a import b", which is the ostensible reason for __init__.py. Kid and TurboCheetah do if the "*.precompiled" option is true, but in the ordinary case they stat both the raw template and compiled template and check the mtimes, and then load it iteratively or by path. So __init__.py is not used in that case. Genshi's compiled templates are not Python code so they couldn't be imported via a simple import anyway.
Cheetah does inheritance by importing a Python module (a precompiled template), so dotted notation and __init__. are required for this. It's desirable to use the same notation for render_response and for inheritance, if possible.
Kid also does inheritance by importing a Python module, though it may be able to compile the parent template on the fly?
Genshi does inheritance by loading the raw parent template directly (or by using a cache). The template is specified in pathname notation relative to the child template.

URI notation:

Allows template names to end in *.html.
Allows multiple templates with the same base name for alternate output types: a.html, a.xml, a.txt.
Allows template names containing characters illegal in Python module names. In some scenarios this is necessary, especially when a project has external constraints.
URI notation has a closer resemblance to the URI of the request.
Mako uses URI notation for inheritance, based on the "templates" directory.
Mako caches compiled templates in a separate directory with a .py suffix (myapp/data/templates/a/b.html.py – obviously an illegal module name). Or at least that's how Pylons configures its plugin.

Given the fact that both forms are popular for certain engines, and that there's a significant number of existing TG/Kid and TG/Cheetah sites which mustn't break, it looks like dotted will have to remain a legal format, and those plugins in particular must accommodate it. Likewise there are several key advantages to URI notation, so this is probably the recommended format for future template engines. Except for those whose inheritance model depends on parent templates being Python modules in Python packages – these may have a compelling reason to prefer dotted notation.

However, again, the template lookup is handled entirely by the plugin, so it can do whatever it wants. A future plugin may load templates from a database, using a completely different notation to identify them.

Fragments

Kid and Genshi support the concept of "fragments". If the plugin's .render method is called with the 'fragment' arg true, they will output the template "standalone", without any inherited site template or automatic template wrapper applied. AJAX uses this to update a certain <div> on a page without reloading the entire page.

Cheetah's inheritance model does not allow bypassing parent templates. However, you can make a slight change to turbocheetah.cheetahsupport.TurboCheetah.render to mimic it. Replace return str(tempobj) with:

if fragment:
    return tempobj.fragment()
else:
    return tempobj.respond()

Then you can define a "#def fragment" that does what it wants. For instance, it may output the main content of the template, while the template itself has a "${fragment()}". Note: This patch also improves Unicode handling, avoiding the UnicodeEncodeError str() raises if the string contains non-ASCII characters.

Mako ignores the 'fragment' argument. Myghty respects the 'fragments' argument.

Template options

Again, you can set template options in the load_environment() function in myapp/config/environment.py. Just assign the options to the tmpl_options dict. The runtime configuration is in the local dict app_conf, so you can pull out any values you need.

Pylons default charset is set in pylons.config.response_defaults["charset"] = "UTF-8". You can override it by passing a 'response_settings' arg to the pylons.config.Config constructor call at the end of myapp/config/environment.py.

Mako

Mako recognizes all "mako." options and passes them directly to the mako.lookup.TemplateLookup constructor after stripping the prefix. It also recognizes three options without the prefix: "directories", "filesystem_checks", and "module_directory". Options that are good candidates for standardization among the template engines include:

mako.directories: template search path (list of strings). Pylons init_app overrides this to the "templates" paths set in myapp/config/environment.py.
mako.filesystem_checks=True: true to check if the source template is newer before returning a cached template. Pylons init_app overrides this to True.
mako.module_directory: where to store compiled templates for reuse. Pylons init_app overrides to the 'cache_dir' item in the config file, subdirectory "templates".
collection_size=-1: how many compiled templates to cache in memory. -1 = unlimited.
output_encoding: Pylons init_app overrides to the application default
default_filters=["unicode"]: list of filters to apply to every context value (i.e., placeholder value)

Myghty

pylonsmyghty has a couple dozen options set in myapp/config/environment.py, pylons.config.py, and pylons/templating.py. I won't even begin to detail them.

Kid

TurboKid recognizes these options:

kid.encoding="utf-8": output encoding. Pylons init_app overrides this to the application default.
kid.sitetemplate: automatically apply a site template around every template?
kid.precompiled: look only for compiled templates
kid.assume_encoding: default encoding for source template files? Pylons init_app overrides this to "utf-8".
kid.i18n.run_template_filter: something to do with multilingual templates?

Cheetah

TurboCheetah recognizes only the "cheetah.precompiled" option, which does the same as its Kid counterpart.

Genshi

Both "genshi" and "genshi-text" recognize these options:

genshi.default_encoding="utf-8"
genshi.auto_reload=True: the opposite of kid.precompiled?
genshi.search_path: colon-separated list of directories to search for templates in. If empty use sys.path.
genshi.max_cache_size=25

"genshi" additionally recognizes these options:

genshi.default_doctype: one of html, html-strict, html-transitional, xhtml, xhtml-strict, xhtml-transitional
genshi.default_format="html": one of html, xhtml, xml, text