My Goal:

I'm trying to get a Pylons-app running as normal as possible, running via paster with full WebError and testing support. While developing this way, I want to have access to the emulated DataStore from the Google SDK, and other features appropriate to GAE. I have tested Datastore; it works. I haven't tested xmpp or images or other GAE services. Yet.

AND, then be able to send the app to GAE for deployment.

warning Ian Bicking (who is the author of Paste among other core Pylons code and knows way more about Appengine than I do) said that this was a bad idea on the mailing list; YMMV.

Other Howtos Here is another howto that may be useful for you: http://countergram.com/articles/pylons-google-app-engine/

Notes:

I'm using Pylons v0.9.7 and GAE v 1.2.7 on Linux. I have not tested these techniques on any other platform or combination.

Note: I'm using [my_gae_id] here to stand-in for the name of your Appengine "Application" name. This is shows up in your domain like this: http://[my_gae_id].appspot.com and also in the dashboard. Your id should not include the square brackets, of course, that's just to remind you what you're looking at.

Install Pylons into a VirtualEnv

1. Make a directory to keep our experiment in:

~$ mkdir sandbox
~$ cd sandbox/

2. Make a virtualenv (mine is called 've' for clarity's sake):

~/sandbox$ virtualenv -p /usr/bin/python2.5 --no-site-packages ve

Note that I am forcing virtualenv to use python2.5, for compatibility with GAE.

3. Activate the virtualenv, install pip, then install pylons:

~/sandbox$ source ve/bin/activate
[notice how the prompt changes to indicate that we're running the ve:]
(ve)~/sandbox$ easy_install pip
(ve)~/sandbox$ pip install pylons

4. Make a Pylons project:

(ve)~/sandbox$ paster create -t pylons MyPylons
[Answer True for Mako, False for SQLAlchemy]

5. Install the Pylons project in develop mode:

(ve)~/sandbox$ cd MyPylons
(ve)~/sandbox/MyPylons$ python setup.py develop

6. Make a controller to test:

(ve)~/sandbox/MyPylons$ paster controller hello

7. Test your app so far:

(ve)~/sandbox/MyPylons$ paster serve development.ini
[hit it with a browser, test the /hello/index route... so far, standard pylons install]

So far we have just done a standard Pylons setup with a virtualenv: nothing has been customized for GAE. Now we'll hook GAE in.

Make an appengine 'app':

1. Unzip the GAE SDK here:

(ve)~/sandbox/MyPylons$ cd ~/sandbox
(ve)~/sandbox$ deactivate
~/sandbox$ unzip ../google_appengine_1.2.7.zip

Your zip file may be located elsewhere; adapt your command accordingly

2. Make a directory for the GAE app:

~/sandbox$ mkdir app

3. We'll need two files installed into app/, first make an "app.yaml" file:

application: [my_gae_id]
version: sandboxtest01
runtime: python
api_version: 1

handlers:
- url: /.*
    script: runpylons.py

skip_files: |
 ^(.*/)?(
 (bin/.*)|
 (include/.*)|
 (.*/docs/.*)|
 (.*/(ez_)?setup.py)|
 (.*/(MANIFEST.in|README.txt|setup.cfg))|
 (lib/pip.*)|
 (lib/nose.*)|
 (lib/[Ww]eb[Tt]est.*)|
 (.*\.yaml)|
 (.*\.egg-link)|
 (#.*#)|
 (.*\~)|
 (.*\.py[co])|
 (.*\.so)|
 (.*/RCS/.*)|
 (\..*)|
 )$

4. And a bootstrap script, called "runpylons.py":

import sys, os

# set up paths of libraries to snarf in...
libdir = "lib"
eggs = ['%s/%s' % (libdir, n) for n in os.listdir(libdir) if n.endswith('.egg')]
sys.path = [libdir] + eggs + sys.path

# this is custom-ish for this app
from mypylons.config.middleware import make_app

from google.appengine.ext.webapp.util import run_wsgi_app

# and get things bootstrapped
APP_ARGS = ({},)
APP_KWARGS = dict()
APP_KWARGS.update({
    'beaker.session.type' : 'google',
    'beaker.session.table_name' : 'beaker_session',
    'beaker.session.key' : 'mylovelykey',
    'beaker.session.secret' : 'mylovelysecret',
    'beaker.cache.type' : 'google',
    'beaker.cache.table_name' : 'beaker_cache',
    'full_stack' : False,
    'static_files' : True,  # @@ probably should let GAE do that some other way
    })
    
# when running on the local dev app server
DEV_APP_ARGS = ({},)
DEV_APP_KWARGS = dict()
DEV_APP_KWARGS['cache_dir'] = '/tmp/gaedev-pylons'

# which one are we?
if os.environ['SERVER_SOFTWARE'].startswith('Development'):
    use_args = DEV_APP_ARGS
    use_kwargs = DEV_APP_KWARGS
else:
    use_args = APP_ARGS
    use_kwargs = APP_KWARGS

# turn some things off for GAE
REMOVE_SYSTEM_LIBRARIES = ['webob']
for system_library in REMOVE_SYSTEM_LIBRARIES:
    for path in list(sys.path):
        if system_library in path:
            sys.path.remove(path)
    for key in list(sys.modules):
        if key == system_library or key.startswith(system_library+'.'):
            del sys.modules[key]

# let her rip
application = make_app(*use_args, **use_kwargs)
run_wsgi_app(application)

Notice the hard-coded name of the app "mypylons" which will need to change on other deployments.

Patch-in GAE support libraries

Now, we wedge the Google SDK libraries into our virtualenv by hacking directly on sys.path in sitecustomize.py (which will have to be created in "sandbox/ve/lib/python2.5/"):

import sys
# get "google" namespace into sys.path
sys.path.append('/home/mjf/sandbox/google_appengine')
# yaml and antlr3
sys.path.append('/home/mjf/sandbox/google_appengine/lib/yaml/lib')
sys.path.append('/home/mjf/sandbox/google_appengine/lib/antlr3')

Note the obvious hard-coded paths. You'll want to fix those in your case.

This will make the google libraries available to the Pylons application when its running from paster. I've tested the DataStorage, and it works, but haven't tried out XMPP, PIL, mail, or anything else. Please let me know if you discover anything.

Modify Some Pylons files:

environment.py

In your pylons app directory, in config/environment.py, we need to fix some things.

1. take out (or comment out) the mako template cache directory line:

module_directory=os.path.join(app_conf['cache_dir'], 'templates'),

2. modify the init_app line to remove legacy template acquisition:

config.init_app(global_conf, app_conf, package='mypylons', paths=paths)

change to:

config.init_app(global_conf, app_conf, package='mypylons', paths=paths, template_engine=None)

3. add some bootstrap stuff to make DataStore work to the end of the file:

# some stuff to get GAE datastore emulation working
if config['debug']:
    os.environ['SERVER_SOFTWARE'] = 'Development/interactive'
    os.environ['APPLICATION_ID'] = config['appid']
    
    from google.appengine.tools import dev_appserver
    from google.appengine.tools.dev_appserver_main import \
        DEFAULT_ARGS, ARG_CLEAR_DATASTORE, ARG_LOG_LEVEL, \
        ARG_DATASTORE_PATH, ARG_HISTORY_PATH
    gae_opts = DEFAULT_ARGS.copy()
    gae_opts[ARG_CLEAR_DATASTORE] = config['datastore_clear']
    gae_opts[ARG_DATASTORE_PATH] = config['datastore_path']
    cfg = dev_appserver.LoadAppConfig('/home/mjf/sandbox/app', {})[0]
    dev_appserver.SetupStubs(cfg.application, **gae_opts)

Note the hard-coded paths in the second to the last line. You'll want to adapt that to your situation.

Note that this last step is just assuming that if you're in "debug" mode (via development.ini) that you'll want these things turned on. If you don't use debug mode during testing, these bits won't be run, and you may have unpredictable results with DataStorage.

development.ini:

Add some config for the datastore to development.ini:

# datastore emulation requirements
appid = [my_gae_id]
datastore_clear = False
datastore_path = %(here)s/gae.datastore

Fix paste:

Fix paste, which hasn't been installed correctly:

~/sandbox/app$ cd ../
~/sandbox$ touch ve/lib/python2.5/site-packages/paste/__init__.py

Install libraries into the app:

install required libraries into the GAE app: GAE ignores libraries that are not installed within the "app/" directory, so the easiest thing to do is to make symlinks:

~/sandbox$ cd app/
~/sandbox/app$ ln -s ../ve/lib/python2.5/site-packages/ lib
~/sandbox/app$ ln -s ../MyPylons/mypylons/ .

Try it out

On the sdk dev server:

~/sandbox$ python2.5 google_appengine/dev_appserver.py app/

And try uploading it:

~/sandbox$ python2.5 google_appengine/appcfg.py update app/

Yay!