| Name |
Space |
Section |
Page |
Version |
Status |
Reviewed |
Author(s) |
| Manage Pylons application with Supervisor |
Pylons CookBook |
Deployment |
Manage Pylons application with Supervisord |
1.0 |
Draft |
False |
 Max Ischenko |
What's it for?
If you deploy your Pylons application as a long-running process you need to make sure it's automatically started at boot, and restarted if it crashes or gets killed. You also need a convenient way to stop and restart it on demand. There are different ways to achieve this; this page describes one possible setup using Supervisor.
Supervisor
is a pure Python application which monitors one or more applications, restarting them automatically if they fail. Applications under control can be run from different system users and separate logs are kept for each. Supervisor runs applications as children which means the 'process monitoring' it does is very reliable and efficient, using waitpid(2). Although Supervisor is a Python application, it can manage non-Python programs just fine, so you may find yourslf using it for more than just Pylons applications. Supervisor runs on Unix and Macintosh systems. It does not run on Windows due to the differing OS features at this level.
Supervisor consists of two programs. supervisord is the daemon or long-running process that manages child applications. supervisorctl is a command-line utility to start and stop child applications on demand, or to see which ones are running. There's also a primitive web-based console (disabled by default) and an XML-RPC interface providing the same commands. All these front ends go through a common web service built into supervisord. The configuration determines whether the service listens for commands on on a Unix domain socket with filesystem permissions (most secure), or on a TCP port with Basic Authentication (for remote control).
Supervisor2 installation
The fact that Supervisor is written in pure Python means it's easy to install. You don't need a C complier which is often absent on production machines. Supervisor can be installed with setuptools:
1 | $ easy_install Supervisor
|
You'll have to make an /etc/supervisord.conf file listing your Python applications and their runtime options. You can test supervisord by running it in one console window and using supervisorctl in another. When everything is running nicely, you can make supervisord run at boot with an RC script. RC scripts differ on each OS and even each Linux distribution, but a sample script for Ubuntu Linux 7.10 is attached to this article. On Ubuntu you'll need to copy the RC script to /etc/init.d/supervisor, and make a symbolic link to it from /etc/rc2.d/S20supervisor.
Supervisor configuration
Copy the sample configuration file in the Supervisor source distribution to /etc/supervisord.conf. Then follow the
manual to customize the settings. Note that the configuration options have changed significantly between Supervisor1, Supervisor2, and Supervisor3. The following configuration is based on a production system running Supervisor 3.0a4 on Ubuntu Linux. Lines starting with a semicolon are comments.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32 | [supervisord]
logfile = /var/log/supervisor.log
loglevel = info
pidfile = /var/run/supervisord.pid
;environment = KEY1=value1,KEY2=value3
[unix_http_server]
file = /var/run/supervisord.sock
chown = root:wwwadmin
chmod = 0770
[supervisorctl]
; Must match settings in 'unix_http_server'
serverurl = unix:///var/run/supervisord.sock
;serverurl = http://127.0.0.1:9001
;username = USERNAME
;password = PASSWORD
[rpcinterface:supervisor]
; This section is always necessary because supervisor uses RPC internally.
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[program:MYPLONSAPP]
command = /var/www/apps/venv/MYPYLONSAPP/bin/paster serve /var/www/apps/MYPYLONSAPP/production.ini
process_name = mypylonsapp
autostart = true
startsecs = 5
;exitcodes = 0,2
user = www-data
redirect_stderr = true
stdout_logfile = /var/www/apps/MYPYLONSAPP/data/error.log
;environment = KEY1=value1,KEY2=value2
|
This example manages one Pylons application running as www-data (the same user Apache runs as). Only root and users in the 'wwwadmin' group can issue supervisorctl commands because only they have permission on the socket. The Pylons application is running in a virtualenv, so we have to give the full path of the "paster" in the virtualenv, along with the full path of production.ini. (There is a directory option to set the child's current directory.)
This logging configuration is perhaps not ideal. It overwrites the application's error log each time the application is started, making it impossible to see error messages that caused the server to crash previously. I haven't found an option to open the error log in "append" mode. But there are several other log options including rotation and not capturing stdout/stderr which you may have better luck with than I did.
Troubleshooting
If started with -n, supervisord won't daemonize itself which is useful during troubleshooting.
If an application exits several times immediately after starting, Supervisor will not start it again until the sysadmin intervenes. This can happen for a thousand reasons. Look in the application's error log and Supervisor's error log. Try running the application standalone as the same user Supervisor would have, first production.ini and then development.ini. Usually you'll find some permission problem or missing file that's causing the application to crash. If you're accessing the server via ssh or the server does not have a graphical interface, user a text-based web browser such as links to verify the application is running.
Supervisor cannot stop some applications; it will report them as stopped even though they're still running. You'll have to kill these manually (Hint: "ps aux | grep STRING" to find the offending processes, "kill PID" to kill them, and "lsof" to find all processes holding a certain file or socket open.) This hasn't been seen with Pylons applications but it has sometimes been seen with Quixote applications, probably because the SCGI server forks child processes in a way that fools supervisord.
