Name	Space	Section	Page	Version	Status	Curator	Reviewed	Author(s)
Installing and running Pylons as plain CGI with a no-frills hosting service	Pylons CookBook		Installing and running Pylons as plain CGI with a no-frills hosting service	1.0	Draft	Graham Higgins	False

Installing and running Pylons as plain CGI with a no-frills hosting service

Today I installed a rudimentary Pylons website on a website hosted by WebHostingBuzz. Although I could not compile there, I managed to install it, albeit in a "dirty* way", so I think I should report on what I did in case others face similar problems at other places.

If you can compile on your webhost, please see Installing and running Pylons as plain CGI on a webhost instead.

(* the only part of the installation which was really unacceptably "dirty" was my method for installing MySQLdb; in my opinon the rest of the method applied here seems o.k.)

This document will explain how I dealt with following problems:

installing all packages into my user home directory on the server
installing Pylons, including the MySQLdb, without compiling anything (some RPM binaries were involved, at least for MySQLdb)
running Pylons as plain CGI (not FastCGI or mod_python or anything nice like that)
running under "jailshell" which apparently means that, among other things, you don't have access to any header files ("include" directories)

It could have been worse. The host did have Python, version 2.4 even, and I did have shell access.

Most people looking at this page probably just want to know how to run Pylons as plain CGI; skip to the bottom if so.

These are compiled from notes I took as I was doing it. I have not gone back and re-done the installation from scratch in order to verify that these instructions are correct in every detail; so don't be surprised if you find a typo or mistake.

The user directory in which the following takes place is located at /home/filtere.

The rest of this document is organized into 4 sections:

Step 1: Set up Python so that you can install new site-packages within your user directory
Step 2: install Pylons
Step 3: Install MySQLdb
Step 4: Running Pylons as plain CGI

Step 1: Set up Python so that you can install new site-packages within your user directory

I followed the directions from the EasyInstall web site, here, to do this. In my case, they amounted to, first, fetching the "virtual-python.py" script:

wget http://peak.telecommunity.com/dist/virtual-python.py

virtual-python.py creates a directory structure within your home directory that contains little "lib", "include", and "bin" directories, so that Python packages can be installed into this directory structure instead of the "real" one outside your user directory (which you don't have access to, if you are a client of a webhosting company). It's like a chroot jail.

Just running virtual-python.py straightaway leads to an error, because there are no "include" directories anywhere. So I had to comment out the lines of code in that file which looked in the "include" directories. Specifically, I used a text editor, found the following part of virtual-python.py:

mkdir(inc_dir)
    stdinc_dir = join(prefix, 'include', py_version)
    for fn in os.listdir(stdinc_dir):
        symlink(join(stdinc_dir, fn), join(inc_dir, fn))

and commented out three of those lines:

mkdir(inc_dir)
#    stdinc_dir = join(prefix, 'include', py_version)
#    for fn in os.listdir(stdinc_dir):
#        symlink(join(stdinc_dir, fn), join(inc_dir, fn))

now, I ran the modified script:

python virtual-python.py

and then I tested it:

./bin/python

Note that from now on, you never want to just run Python as "python", because that would run the python interpreter from outside your little sub-universe, and that interpreter won't know about all of the extra packages that you are installing. Instead, you always want to refer to the python interpreter inside "bin" within your user directory (/home/filtere/bin/python in my case).

The next thing to do was to install EasyInstall:

wget http://peak.telecommunity.com/dist/ez_setup.py
./bin/python ez_setup.py

Note that EasyInstall is also now installed only within your user directory; to run, say, easy_install, you want to be sure to type bin/easy_install, and never just easy_install.

Step 2: install Pylons

At this point I attempted to install Pylons:

./bin/easy_install Pylons

Some components installed, but PasteScript, which Pylons depends on, depends on Cheetah. Cheetah tried to compile something (I think it was "_namemapper"?), and that failed, and the whole installation halted with an error.

The reason the compilation didn't work was that there were no "include" directories, and the thing being compiled depended on header files which were supposed to be in those include directories. I suspect this is due to the "jailshell" that I was running in (I should note that it seemed like I was running in bash, not jailshell, but if no processes were running and i typed "fg" at the shell prompt, I got the error "-jailshell: fg: current: no such job").

I couldn't find a non-hacking way to just tell Cheetah not to compile _namemapper, but it's simple enough to change its setup configuration in the code. The first thing you have to do, though, is to get its code; easy_install by default downloads it into a temp directory and then deletes it when it fails. To change this, do:

mkdir tt
./bin/easy_install -b tt Pylons

This tells easy_install to use the "tt" directory that you just created, and to not delete stuff from that directory when it's done.

cd tt/cheetah

use a text editor to edit SetupConfig.py, and change this:

1 2 3 4
if os.name == 'posix': ext_modules=[Extension("Cheetah._namemapper", \[os.path.join("src" ,"_namema\pper.c")] ) ]

to this:

1 2
if os.name == 'posix': ext_modules=[]

Now, in order to turn this modified package into an egg and install it, do this:

export CHEETAH_USE_SETUPTOOLS=1
../../bin/python setup.py -q bdist_egg
../../bin/easy_install dist/Cheetah-2.0rc7-py2.4.egg

replacing the version numbers in the last line with whatever you have.

Now, you can do

cd ../..
bin/easy_install Pylons

and everything else should install.

Step 3: Install MySQLdb

OK, this is the dirtest part of the whole thing. MySQLdb is a DBAPI adaptor, that is, it bridges the gap between Python's unified database API and the idiosyncracies of MySQL. As far as I can tell, most Python modules that use databases use Python's DBAPI. Therefore, if you want to use a Python module to access a MySQL database, you probably need MySQLdb or something else very much like it.

The difficult part is that MySQLdb has to be compiled, and as I mentioned above, I didn't have access to any header files ("include" directories).

So, I resorted to a fudge. Luckily, the web host seemed to be an x86 CentOS based system. They were running something close to Fedora Core 4. So, I was able to find an RPM binary from Fedora for the package MySQL-python, and luckily it was compatible with the other stuff they had on their system.

Specifically, after much trial and error, I was able to do the following. First, create a local RPM db inside your home directory:

cd
mkdir -p var/lock/rpm
rpm --initdb --root /home/filtere/

Next, try to install the package you want:

rpm  -v -i MySQL-python-1.2.0-1.i386.rpm

By looking at the errors, and using a website like rpmfind, determine which other RPMs you need to fulfill dependencies, and add them in, until finally all remaining dependencies are things that are already present on the system:

# rpm  -v -i MySQL-python-1.2.0-1.i386.rpm mx-2.0.6-2.i386.rpm \
             openssl-0.9.7f-7.i686.rpm
warning: MySQL-python-1.2.0-1.i386.rpm: V3 \
         DSA signature: NOKEY, \
         key ID 4f2a6fd2error: Failed dependencies:
        python >= 2.4 is needed by MySQL-python-1.2.0-1.i386
        python(abi) = 2.4 is needed by MySQL-python-1.2.0-1.i386
        python(abi) = 2.4 is needed by mx-2.0.6-2.i386
        libk5crypto.so.3(k5crypto_3_MIT) is needed by openssl-0.9.7f-7.i686
        libkrb5.so.3(krb5_3_MIT) is needed by openssl-0.9.7f-7.i686

in my case, /lib/libk5crypto.so.3 existed, as did /lib/ibkrb5.so.3, as did Python 2.4. So I stopped here. And then I actually installed these things, using --nodeps to ignore failed dependcies, and --root to tell everything to pretend that my home directory is the entire world.

# rpm --nodeps -v --root /home/filtere/ \
      -i MySQL-python-1.2.0-1.i386.rpm mx-2.0.6-2.i386.rpm \
         openssl-0.9.7f-7.i686.rpm
warning: MySQL-python-1.2.0-1.i386.rpm: V3 \
         DSA signature: NOKEY, \
         key ID 4f2a6fd2Preparing packages for installation...
openssl-0.9.7f-7
error: unpacking of archive failed on file /etc/pki/CA: \
       cpio: mkdir failed - No such file or directory
MySQL-python-1.2.0-1
error: unpacking of archive failed on file /usr/lib/python2.4/site-packages/MySQLdb: \
       cpio: mkdir failed - No such file or directory
mx-2.0.6-2
error: unpacking of archive failed on file /usr/include/btr.h;450deb5f: \
       cpio: mkdir failed - No such file or directory

Oops, we need to create some directories:

mkdir -p etc/pki
mkdir -p usr/lib/python2.4/site-packages/ 
mkdir -p usr/include

and then:

rpm --nodeps -v --relocate /=/home/filtere \
    --root /home/filtere \
    -i MySQL-python-1.2.0-1.i386.rpm mx-2.0.6-2.i386.rpm \
                            openssl-0.9.7f-7.10.i386.rpm

actually, I'm not sure that the previous three blocks of commands were all necessary. I suspect only the last one was – but I don't remember.

Finally, for some reason, the symlinks associated with the openssl shared libraries weren't created, so I did that manually:

ln -s /home/filtere/lib/libssl.so.0.9.7f /home/filtere/lib/libssl.so.5
ln -s /home/filtere/lib/libcrypto.so.0.9.7f /home/filtere/lib/libcrypto.so.5

At this point, you can test whether MySQLdb actually works:

# cd /home/filtere; export LD_LIBRARY_PATH=/home/filtere/lib; bin/python
Python 2.4.3 (#1, Jun  4 2006, 18:58:57)
\[GCC 3.4.5 20051201 (Red Hat 3.4.5-2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import MySQLdb
>>>

If things weren't working, there would have been an ImportError when you did that, complaining about shared libraries. The "export LD_LIBRARY_PATH" part is necessary so that the operating system bothers to check your local "lib" directory when looking for shared libraries.

Now, there's one more annoyance. If you browsed the web and your request was handled by some CGI, then that CGI would be loaded by Apache, without any chance to set the LD_LIBRARY_PATH variable in the envirionment first.

So we gotta use a wrapper script. I called mine test.cgi:

#!/bin/sh

exec /usr/bin/env LD_LIBRARY_PATH=/home/filtere/lib/ \
     /home/filtere/bin/python /home/filtere/public_html/test.py

What this does when called is, first, set the environment variable LD_LIBRARY_PATH to /home/filtere/lib/; second, run bin/python public_html/test.py.

I note that the instructions in Step 3 are probably only a temporary solution; someday, the host will upgrade their core shared libraries. The upgraded libraries may be incompatible with the older RPM binaries which are installed in my user directory. And then my RPM binaries will break, and my website will suddenly and unexpectedly stop working, possibly with no easy way for me to fix it.

Step 4: Running Pylons as plain CGI

First, install the "wsgiref" package (if you just skipped down to this part, don't worry about the "bin/" in front):

bin/easy_install wsgiref

Next, be sure that the "debug" config variable is false in your .ini file. The interactive debugger won't run as plain CGI (because wsgi.multiprocess is set), but it will muck things up. When "debug" is set to false, the interactive debugger is never activated.

I note that by setting an "error_log" option in the config file, you can log errors to a file.

Finally, make a file (I called it test.py) containing something like this:

1 2 3 4 5 6
#!/home/filtere/bin/python from paste.deploy import loadapp wsgi_app = loadapp('config:/home/filtere/test.ini') import wsgiref.handlers wsgiref.handlers.CGIHandler().run(wsgi_app)

where "/home/filtere/test.ini" is replaced by the path to your .ini file.

Remember to execute chmod a+x on test.py (and also on test.cgi, if you are following the directions above).

Now, point your browser at the location of this file (or, if you are following the directions above, point the other test.cgi file at this one, and then point your browser to test.cgi) , and there you go.

I have not done any benchmarks so I cannot say if using Pylons as plain CGI will be unacceptably slow.

BayleShanks