Name	Space	Section	Page	Version	Status	Reviewed	Author(s)
Pylons deployment with daemontools	Pylons CookBook		Pylons deployment with daemontools	1.0	Draft	False	James Gardner

Pylons deployment with daemontools

This tutorial describes how to setup a WSGI application (such as a Pylons application) to be monitored by daemontools so that it is automatically restarted if it fails. Using daemontools together with the paste.httpserver server behind a reverse proxy provides a robust, scalable production deployment. Note: pure-python supervisor2 package provides a nice alternative to daemontools, see Manage Pylons application with supervisord for more info.

It is worth reading the daemontools FAQ which forms the main documentation.

Install

First you will need to install daemontools. The process is described on the daemontools install page but there are various patches which need applying depending on your setup. Most Linux distributions will have a daemontools package you can install directly. For example on Ubuntu 6.06 Server if you enablie the multiverse repository you can install daemontools as root with the daemontools-installer package:

> apt-get install daemontools-installer

Follow the instructions and choose the DJB layout and the installer will work away to create a .deb file. Once the file is created it will ask if you want to install it, you do. The installer also adds a line to your inittab so that when your system reboots, daemontools will be started.

Check the line in inittab is correct, there is no reason it shouldn't be but you don't want to be left with a non-functioning system you can't boot!

Graham Stratton adds that to install daemontools under debian, you need to uncomment unstable repositories in /etc/apt/sources.apt and apt-get update, and it works as Ubuntu.

Reboot your system to check daemontools starts.

You now have a /service directory where all the daemontools services to be monitored are installed.

Setting up a virtual Python install

This is described in Creating a Sandboxed Pylons Installation with a Virtual Python Install.

If the Pylons project we wish to serve is in the code directory the lines below will install all the application's dependencies to the virtual Python install:

cd code
    ~/bin/python setup.py develop

The above commands will have added the paster script to our virtual install so we can now create a production.ini configuration file and test it with this command:

~/bin/paster serve production.ini

.. note :: You wouldn't want to use the --reload option for a production deployment so it is left out above.

If the running application behaves correctly we are ready to serve it using daemontools. The end result will be pretty much identical since daemontools will do little more than run the above command and restart the server if it exits for any reason. It is therefore important to ensure your application functions properly first before setting up daemontools.

Setting up daemontools

We want our application to send its output to a log file, run as the user webuser and have webuser able to start, stop and restart it. Create the following directory structure as webuser in their home directory for what we'll call the test project:

service
    service/test
    service/test/log
    service/test/log/main

via:

mkdir -p service/test/log/main

Of course you don't have to call your service directory test but it seems appropriate for our example so we will refer to the test service throughout this tutorial. First let's create the file that will run the application. Create
service/test/run with the following content:

1 2
#!/bin/sh exec setuidgid webuser /home/webuser/bin/python -u server.py

This command uses our virtual Python to execute a custom server.py script which we will write in a minute to serve our application using the production.ini file we created earlier. Obviously you will need to adjust the paths for your setup.

The setuidgid program comes with daemontools and runs the command as the user and group of the user specified, in this case webuser.

Notice how we are using Python in unbuffered mode with the -u option. This is very important because it means Python will output any error messages straight away without buffering them.

We are going to serve our application with the same paste.httpserver module that the paster serve command uses if you are using a recent versions of Pylons. Create server.py to look like this:

1 2 3 4
import paste.deploy wsgi_app = paste.deploy.loadapp('config:/home/webuser/code/production.ini') serve = paste.deploy.loadserver('config:/home/webuser/code/production.ini') serve(wsgi_app)

Of course you don't have to use paste.deploy, you can write any code capable of serving a WSGI application but since this is a Pylons application it makes sense to obtain the server and application configuration from the config file which is what the code above does. Below is an alternative which loads the application from the config file but ignores the server information. This example might be useful for a server administrator who wants to control which ports the users can serve their applications on:

1 2 3 4 5
import paste.deploy import paste.httpserver wsgi_app = paste.deploy.loadapp('config:/home/webuser/code/production.ini') paste.httpserver.serve(wsgi_app, host='0.0.0.0', port=8000)

You don't need to specify a #!/home/webuser/bin/python -u line because server.py is directly executed in run with the unbuffered virtual Python executable being run as webuser.

We also want all log messages from our application to be sent to an error log. Daemontools treats the logging component as a separate service. To enable logging create the file service/test/log/run with the following content:

1 2
#!/bin/sh exec setuidgid webuser multilog t ./main

This will use a rotating log file to log messages from the server to the service/test/log/main directory with the most recent logs being added to a current file which is created once the service is started.

We need to make sure the appropriate files are executable:

chmod 1755 service/test
chmod 1755 service/test/log
chmod 755 service/test/run
chmod 755 service/test/log/run

Then we can add an entry to the real deamontools service directory so it can actually find the our service. Runnung this command effectively installs the service:

ln -s /home/webuser/service/test /service/test

We could have created the filesystem directly in the /service/test directory but daemontools would have constantly being trying to start the application whilst we were writing it. Also daemontools does not look for a log directory once it has started a serivce so if we had built the structure directly in the /service directory, supervise would have started the service before we had written the logging part so no logging would be started. Of course the next time the system rebooted supervise would have found both and logging would then have been enables.

Logs

The main daemontools supervise program logs messages to ps which can be read with this command:

ps -axww | grep readproctitle

This can be very useful for troubleshooting problems with services not starting correctly.

The individual services like the one setup above use whichever logging program you choose, in this case multilog which is part of daemontools. The rotating log files have a timestamp added which needs to be translated using the tai64nlocal program. You can monitor your log file with this command:

tail -f /service/test/log/main/current | tai64nlocal

Or produce the whole thing with:

cat /service/test/log/main/current | tai64nlocal

.. note :: If you have problems with the logs you can use the svstat program to check the logging daemon is actually running:

svstat /service/test/log

It is also very important that all parts of the execution chain use Python with the -u option to ensure that output is not buffered, otherwise it will all be sent to the logging daemon in one go when the service exits.

Controlling the service

Once your service is up and running you can use the svstat program to print the status of your service:

svstat /service/test /service/test/log

You will also find the svc program very useful for controlling the service. These are the commands which you will find most useful. When setup as described above, these can all be executed by webuser.

svstat /service/test
Gives the status

svc -d /service/test
Stops the service

svc -u /service/test
Starts the service

svc -t /service/test
Send a TERM signal (effectively restarts the service)

svstat /service/test/log
Gives the status of the logging daemon