This is an update of Pylons Execution Analysis
, which was originally written for Pylons 0.9.3. It's based on a Mercurial version of Pylons 0.9.6rc2, which does not contain any legacy (=backward compatibility) code. We're using Python 2.5.1.
Because there's no QuickWiki demo app for 0.9.6 yet, we'll make a simple Pylons application and analyze that.
Apologies for the crude layout of the list numbering; it's the limitations of this Confluence wiki.
Definitions
Pylons is based on WSGI and middleware, so let's define these before proceeding. WSGI is standard way to link Python web components together. In the minimal case you have two components, server and application. server is typically an HTTP server; application is a function implementing a particular web site. The server receives a web request, calls the application, receives a response, and sends the response to the remote user. WSGI defines the application's arguments and return value. You can also build chains longer than two; e.g. server calls middleware1 calls middleware2 calls application. A middleware is like a function decorator: it calls the application provided as its first argument, and provides some value-added service like authentication, error documents, etc. Some Python developers distinguish between "middleware" (which the application doesn't depend on or even know if it exists) and "WSGI application components" (which the application depends on). This article sidesteps that issue and calls all two-ended WSGI components "middleware".
Creating the application
Installation
Here I created a workingenv to create an isolated Python environment that uses only the packages I install into it, not any others I may have that may be incompatible versions. I installed Mercurial inside the workingenv so I could use the Pylons Mercurial repository, but you could instead install a Pylons release or Pylons Subversion if you wish. I had to install Mercurial even though I already had it on my system because the mercurial module wasn't visible through the workingenv wall. (I also got two Mercurial errors below which I don't understand, but they don't seem to be significant.)
You'll now have three subdirectories in your ~/workspace directory:
- wenv is a workingenv.
- pylons is a copy of the latest Pylons development repository.
- Analysis is the Pylons application we just created.
Important: the "(wenv)" prefix indicates you're "inside" the wenv environment. Scripts in ~/workspace/wenv/bin will override those in your normal bin directories; packages in ~/workspace/wenv/lib will be used instead of those in your global "site-packages". To leave the workingenv and return to the normal state, run "deactivate". This Pylons version will then be inaccessible. To reenter the workingenv so we can use Pylons again, run the "source ... activate" command above.
"python setup.py develop" automatically downloaded and installed several Python packages which are required by Pylons. Here are the versions as of this writing (yours may be different):
Customizing
Let's create a simple controller with two methods.
Edit analysis/controllers/main.py to look like this:
1
2
3
4
5
6
7
8
9
10
11
12
13 | import logging
from analysis.lib.base import *
log = logging.getLogger(__name__)
class MainController(BaseController):
def index(self):
return '<h1>Welcome to the Analysis Demo</h1>Here is a <a href="page2">link</a>.'
def page2(self):
return 'Thank you for using the Analysis Demo. <a href="/">Home</a>'
|
Yes, controller methods can return strings in Pylons 0.9.6!
Now edit analysis/config/routing.py. Where it says "CUSTOM ROUTES HERE", add these two lines:
1
2 | map.connect('', controller='main', action='index')
map.connect('page2', controller='main', action='page2')
|
Delete the file analysis/public/index.html.
Optional: activate the transaction log middleware. Edit analysis/config/middleware.py and add the following where it says "CUSTOM MIDDLEWARE HERE":
1
2 | from paste.translogger import TransLogger
app = TransLogger(app, setup_console_handler=True)
|
Running
Point your web browser to http://127.0.0.1:5000/. When bored, press ctrl-C to stop the server.
Differences from Pylons 0.9.5
This is just a quick look at the differences between a Pylons 0.9.6 app and a Pylons 0.9.5 app.
The application's configuration data is now under pylons.config. This simplifies both analysis/lib/middleware.py and analysis/lib/environment.py. They no longer pass the config back and forth. Instead, middleware.py asks environment.py to set up the configuration. environment.py does so, also adds its own configuration constants, and configures the template engine(s). This gets middleware.py closer to containing only middleware.
development.ini has lost all SQLAlchemy/SQLObject variables. Database configuration is now up to the user or to third-party libraries like SAContext. Of course, you can add your own configuration variables for anything you want. Meanwhile development.ini has added several new sections for logging: [loggers], [handlers], [formatters], [logger_root], [handler_console], [formatter_generic]. These are not documented yet???
Analyzing the application
We'll use $LIB as a shortcut for ~/workspace/wenv/lib/python2.5, $BIN for ~/workspace/wenv/bin, and $APP for ~/workspace/Analysis.
Startup (PasteScript)
When you run paster serve development.ini, it runs the "$BIN/paster" program. This is a platform-specific stub created by easy_install. It does this:
1
2
3
4
5 | from pkg_resources import load_entry_point
sys.exit(
load_entry_point('PasteScript==1.3.6dev-r6751', 'console_scripts', 'paster')()
)
|
This says to load a Python object "paster" located in version 1.3.6dev-r6751 of an egg "PasteScript", under the entry point group [console_scripts]. We'll skip the details of how pkg_resources finds the object, but the entry point is at the bottom of $LIB/PasteScript-1.3.6dev_r6751-py2.5.egg/EGG-INFO/entry_points.txt:
1
2 | [console_scripts]
paster=paste.script.command:run
|
This maps the "paster" entry point to the paste.script.command.run function. load_entry_point imports and returns the function, () executes it, and sys.exit() returns the exit value to the operating system. There is also an actual paster script, $LIB/PasteScript-1.3.6dev_r6751-py2.5.egg/EGG-INFO/scripts/paster. I'm not sure whether pkg_resources actually uses it, but it calls the exact same run() function so it doesn't matter. I inserted print statements into paste.script.command to figure out what it does. Here's a simplified description:
1. The run() function parses the command-line options into a subcommand "serve" with arguments ["development.ini"].
2. It calls get_commands(), which loads Paster commands from plugins located at various entry points. (You can add custom commands with the "--plugin" command-line argument.) Paste's standard commands are listed in $LIB/PasteScript-VERSION/EGG-INFO/entry_points.txt:
1
2
3 | [paste.global_paster_command]
serve=paste.script.serve:ServeCommand [Config]
#... other commands like "make-config", "setup-app", etc ...
|
3. It calls invoke(), which essentially does paste.script.serve.ServeCommand(["development.ini"]).run(). This in turn calls {{ServeCommand.command(), which handles daemonizing and other top-level stuff. Since our command line is short, there's no top-level stuff to do. It creates 'server' and 'app' objects based on the configuration, and calls server(app).
Loading the server and the application (PasteDeploy)
This all happens during step 3 of the application startup. We need to find and instantiate the WSGI application and server based on the configuration file. The application is our Analysis application. The server is Paste's built-in multithreaded HTTP server. A simplified version of the code is:
1
2
3
4
5
6 | # Inside paste.script.serve module, ServeCommand.command() method.
from paste.deploy.loadwsgi import loadapp, loadserver
server = loadserver(uri="config:development.ini", name=None,
relative_to="/DIR/CONTAINING/CONFIG/FILE")
app = loadapp(uri="config:development.ini", name=None,
relative_to="/DIR/CONTAINING/CONFIG/FILE")
|
loadserver() and loadapp() are defined in module paste.deploy.loadwsgi. The code here is complex and delves into the details of Python eggs and entry points, so we'll just look at its general behavior. Both functions see the "config:" URI and read our config file. Since there is no server name or app name they both default to "main". Therefore loadserver() looks for a "[server:main]" section in the config file, and loadapp()` looks for "[app:main]". Here's what they find in "development.ini":
1
2
3
4
5
6
7
8 | [server:main]
use = egg:Paste#http
host = 0.0.0.0
port = 5000
[app:main]
use = egg:Analysis
...
|
The "use =" line in each section tells which object to load. The other lines are configuration parameters for that object, or for plugins that object is expected to load. We can also put custom parameters in [app:main] for our application to read directly.
Server loading
1. loadserver()'s args are uri="config:development.ini", name=None, relative_to="$APP".
2. A "config:" URI means to read a config file.
3. A server name was not specified so it defaults to "main". So loadserver() looks for a section "[server:main]". The "server" part comes from the loadwsgi._Server.config_prefixes class attribute in $LIB/PasteDeploy*.egg/paste/deploy/loadwsgi.py).
4. "use = egg:Paste#http" says to load an egg called "Paste".
5. loadwsgi._Server.egg_protocols lists two protocols it supports: "server_factory" and "server_runner".
6. "paste.server_runner" is an entry point group in the "Paste" egg, and it has an entry point "http". The relevant lines in $LIB/Paste*.egg/EGG-INFO/entry_points.txt are:
1
2 | [paste.server_runner]
http = paste.httpserver:server_runner
|
7. There's a server_runner() function in the paste.httpserver module ($LIB/Paste*.egg/paste/httpserver.py).
We'll stop here for a moment and look at how the application is loaded.
Application loading
1. loadapp() looks for a section "[app:main]" in the config file. The "app" part comes from the loadwsgi._App.config_prefixes class attribute (in $LIB/PasteDeploy*.egg/paste/deploy/loadwsgi.py).
2. "use = egg:Analysis" says to find an egg called "Analysis".
3. loadwsgi._App.egg_protocols lists "paste.app_factory" as one of the protocols it supports.
4. "paste.app_factory" is also an entry point group in the egg, as seen in $APP/Analysis.egg-info/entry_points.txt:
1
2
3
4 | [paste.app_factory]
main=analysis:make_app
[paste.app_install]
main=paste.script.appinstall:Installer
|
5. The line "main=analysis:make_app" means to look for a make_app() object in the analysis package. This is a function imported from analysis.config.middleware ($APP/analysis/config/middleware.py).
Instantiating the application (Analysis)
Here's a closer look at our application's make_app function:
1
2
3
4
5
6 | # In $APP/analysis/config/middleware.py
def make_app(global_conf, full_stack=True, **app_conf):
load_environment(global_conf, app_conf)
app = PylonsApp()
app = SomeMiddleware(app, ...) # Repeated for several middlewares.
return app
|
This sets up the Pylons environment (next subsection), creates the application object (following subsection), wraps it in several layers of middleware (listed in "Anatomy of a Request" below), and returns the complete application object.
The [DEFAULT] section of the config file is passed as dict global_conf. The [app:main] section is passed as keyword arguments into dict app_conf. full_stack defaults to True because we're running the application standalone. If we were embedding this application as a WSGI component of some larger application, we'd set full_stack to False to disable some of the middleware.
load_environment & pylons.config
The first thing make_app does is call analysis.config.environment.load_environment():
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17 | def load_environment(global_conf, app_conf):
root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
paths = dict(root=root,
controllers=os.path.join(root, 'controllers'),
static_files=os.path.join(root, 'public'),
templates=[os.path.join(root, 'templates')])
# Initialize config with the basic options
config.init_app(global_conf, app_conf, package='analysis',
template_engine='mako', paths=paths)
config['pylons.g'] = app_globals.Globals()
config['pylons.h'] = analysis.lib.helpers
config['routes.map'] = make_map()
# Customize templating options via this variable
tmpl_options = config['buffet.template_options']
|
config is pylons.config, the global rendezvous point for reading the application configuration from. It's an instance of pylons.configuration.PylonsConfig, a pseudo-dict. Initially it contains a default configuration. config.init_app() adds the keys from the app_conf, global_conf, template configuration, paths, and others.
config['pylons.g'] and config['pylons.h'] are the special g and h globals used in Pylons applications.
config['routes.map'] is a Routes Mapper object created by analysis.config.routing.make_map(). It contains the application's rules for mapping URLs to controller methods, and named routes to URLs.
The tmpl_options variable is just a convenience. Pylons sets default options for several template engines. If you want to customize the options you'd something like tmpl_options["kid.assume_encoding"] = "iso-8859-3".
PylonsApp
The second line of make_app calls PylonsApp(), defined in $LIB/Pylons*.egg/pylons/wsgiapp.py. This creates a Pylons application object based on your configuration. It's called without arguments so you get the default type of application. It creates a PylonsBaseWSGIApp (same module) and puts it at self.app. This has attributes for the various configuration objects we've mentioned above. self.app.buffet is a pylons.templating.Buffet instance, used to access to any template engine implementing the Buffet protocol.
PylonsApp wraps self.app in some middleware, which we'll see in "Anatomy of a request" below.
 | Note
PylonsBaseWSGIApp will apparently be merged into PylonsApp by Pylons 1.0. pylons.templating will be replaced by a new version of Buffet whose API is still in the works. The middleware in wsgiapp.py will be moved to middleware.py. |
Historical note
make_app, load_environment, and pylons.config were changed substantially in Pylons 0.9.6. In earlier versions, paste.deploy.CONFIG was the central configuration repository instead of pylons.config. It contained .app_conf and .global_conf sub-dicts. Some configuration initialization was done in make_app. Partially-completed configurations were passed between make_app and load_environment. This has all been cleaned up and simplified.
Anatomy of a request
Let's say you're running the demo and click the "link" link on the home page. The browser sends a request for "http://localhost:5000/page2". In my Firefox the HTTP request headers are:
The response is:
Here's the processing sequence:
1. server(app) is still running, called by ServeCommand.command() in $LIB/PasteScript-*.egg/paste/script/serve.py.
2. server is actually paste.httpserver.server_runner() in $LIB/Paste-*.egg/paste/httpserver. The only keyword args are 'host' and 'port' extracted from the config file. server_runner de-stringifies the arguments and calls serve(wsgi_app, **kwargs) (same module).
3. serve()'s 'use_threadpool' arg defaults to True, so it creates a WSGIThreadPoolServer instance called (server) with the following inheritance:
4. It calls server.serve_forever(), implemented by the ThreadPoolMixIn superclass. This calls self.handle_request() in a loop until self.running becomes false. That initiates this call stack:
self.socket.accept() blocks, waiting for the next request.
5. The request arrives and self.socket.accept() returns a new socket for the connection. TCPServer.handle_request() continues. It calls ThreadPoolMixIn.process_request(), which puts the request in a thread queue:
1
2
3 | self.thread_pool.put(
lambda: self.process_request_in_thread(request, client_address))
# 'request' is the connection socket.
|
The thread pool is defined in the ThreadPool class. It spawns a number of threads which each wait on the queue for a callable to run. In this case the callable will be a complete Web transaction including sending the HTML page to the client. Each thread will repeatedly process transactions from the queue until they receive a sentinel value ordering them to die.
The main thread goes back to listening for other requests, so we're no longer interested in it.
6. Thread #2 pulls the lambda out of the queue and calls it:
7. The newly-created request handler takes over:
The WSGI environment dict is described in PEP 333, the WSGI specification. It contains various keys specifying the URL to fetch, query parameters, server info, etc. All keys required by the CGI specification are present, as are other keys specific to WSGI or to paricular middleware. The application will calculate a response based on the dict. The application is wrapped in layers of middleware – nested function calls – which modify the dict on the way in and modify the response on the way out.
8. The request handler, still in WSGIHandlerMixin.wsgi_execute(), calls the application thus:
1 | result = app(wsgi_environ_dict, wsgi_start_response)
|
wsgi_start_response is a callable mandated by the WSGI spec. The application will call it to specify the HTTP headers. The return value is an iteration of strings, which when concatenated form the HTML document to send to the browser. Other MIME types are handled analagously.
9. The application, as we remember, was returned by analysis.config.middleware.make_app(). It's wrapped in several layers of middleware, so calling it will execute the middleware in reverse order of how they're listed in $APP/analysis/config/middleware.py and $LIB/Pylons*.egg/pylons/wsgiapp.py:
- Cascade (defined in $LIB/Paste-*.egg/paste/cascade.py) lists a series of applications which will be tried in order:
1. StaticURLParser (defined in $LIB/Paste-*.egg/paste/urlparser) looks for a file URL under $APP/analysis/public that matches the URL. The demo has no static files.
2. If that raises "NotFound", the cascader tries StaticJavascripts (defined in $LIB/Pylons*.egg/pylons/middleware.py), which tries to return a Javascript script from the WebHelpers package (defined in $LIB/WebHelpers*.egg). These support the Pylons h object, and include a port of Ruby on Rails' helpers, Prototype, Scriptaculous. See WebHelpers. MochiKit is included in $LIB/Paste-*.egg/paste/evalexception/mochikit/MochiKit.js, but it's not clear how to access it in an application method.
3. If that fails too, the cascader tries your application. But first there are other middleware to go through...
- RegistryManager (defined in $LIB/Paste-*/paste/registry.py) makes Pylons special globals both thread-local and middleware-local. This includes c, g, cache, request, response, session, and any other StackedObjectProxy listed in $LIB/Pylons*.egg/pylons/_init_.py. h is not included because it's thread safe.
- ErrorDocuments (defined in $LIB/Pylons*.egg/pylons/middleware.py) intercepts any HTTP error status returned by the application (e.g., "Page Not Found", "Internal Server Error") and sends another request to the application to get the appropriate error page to display instead. (Skipped if full_stack argument was false.)
- ErrorHandler (defined in $LIB/Pylons*.egg/pylons/middleware.py) sends an interactive traceback to the browser if the app raises an exception, if "debug" is true in the config file. Otherwise it attempts to email the traceback to the site administrator, and substitutes a generic Internal Server Error for the response. (Skipped if full_stack argument was false.
- User-defined middleware goes here. If you added TransLogger to the demo, it would be called here.
- CacheMiddleware (wsgiapp.py) adds Beaker caching (pylons.cache). (Skipped if the WSGI environment has a key 'cache' – it doesn't in this demo.)
- SessionMiddleware (wsgiapp.py) adds Beaker session support (the pylons.session object). (Skipped if the WSGI environment has a key 'session' – it doesn't in this demo.)
- RoutesMiddleware (wsgiapp.py) compares the request URI against the routing rules in $APP/analysis/config/routing.py and sets 'wsgi.routing_args' to the routing match dict (useful) and 'routes.route' to the Route (probably not useful).
- The innermost middleware calls the PylonsApp instance it was initialized with.
- Surprise! PylonsApp is itself middleware. Its .__call__ method calls the pylons.wsgiapp.PylonsBaseWSGIApp instance in its self.app attribute.
10. PylonsBaseWSGIApp is a middleware too. Its .__call__() method does:
1
2
3
4 | self.setup_app_env(environ, start_response)
controller = self.resolve(environ, start_response)
response = self.dispatch(controller, environ, start_response)
return response
|
.setup_app_env() registers all those special globals.
.resolve() calculates the controller class based on the route chosen by the RoutesMiddleware, and returns the controller class.
.dispatch instantiates the controller class and calls in the WSGI manner. If the controller does not exist (.resolve() returned None), return HTTP "NotFound".
11. analysis.controllers.main.MainController does not have a .__call__() method, so control falls to its parent, analysis.lib.base.BaseController. This trivially calls the grandparent, pylons.controllers.WSGIController. It calls the action method MainController.page2(). The action method may have any number of positional arguments as long as they correspond to variables in the routing match dict. (GET/POST variables are in the request.params dict.) If the method has a **kwargs argument, all other match variables are put there. Any variables passed to the action method are also put on the c object as attributes. If an action method name starts with "_", it's private and HTTPNotFound is raised.
12. If the controller has .__before__() and/or .__after__() methods, they are called before and after the action, respectively. These can perform authorization, lock OS resources, etc. These methods can have arguments in the same manner as the action method. However, if the code is used by all controllers, most Pylons programmers prefer to it in the base controller's .__call__ method instead.
13. The action method returns a string, unicode, Response object, or is a generator of strings. In this trivial case it returns a string. A typical Pylons action would set some c attributes and 'return render('/some/template.html")' . In either case the global response object's body would be set to the string. Legacy Pylons 0.9.5 apps return a Response object, which is used as the global response.
14. WSGIController.__call__() continues, converting the Response object to an appropriate WSGI return value. (First it calls the start_response callback to specify the HTTP headers, then it returns an iteration of strings. The Response object converts unicode to utf-8 encoded strings, or whatever encoding you've specified in the config file.)
15. The stack of middleware calls unwinds, each modifying the return value and headers if desired.
16. The server receives the final return value. (We're way back in paste.httpserver.WSGIHandlerMixin.wsgi_execute() now.) The outermost middleware has called back to server.start_response(), which has saved the status and HTTP headers in .wsgi_curr_headers. .wsgi_execute() then iterates the application's return value, calling .wsgi_write_chunk(chunk) for each encoded string yielded. .wsgi_write_chunk() formats the status and HTTP headers and sends them on the socket if they haven't been sent yet, then sends the chunk. The convoluted header behavior here is mandated by the WSGI spec.
17. Control returns to BaseHTTPRequestHandler.handle(). .close_connection is true so this method returns. The call stack continues unwinding all the way to paste.httpserver.ThreadPoolMixIn.process_request_in_thread(). This calls SocketServer.close_request(), which does nothing.
18. The request lambda finishes and control returns to ThreadPool.worker_thread_callback(). It waits for another request in the thread queue. If the next item in the queue is the shutdown sentinel value, thread #2 dies.
Thus endeth our request's long journey, and this analysis is finished too.
Author: Mike Orr
