Some performance test results

This page details some results from a comparison between Nginx and Lighttpd as a front-end for Pylons in various ways (proxy and fastcgi/scgi).

These Apache Bench tests were run from a dual Athlon box against a vanilla 3.4GHz Pentium box with 4G of ram accessing the home page of a Pylons v0.9.4.1 app (which has no db access, just a little Myghty templating) behind nginx v0.5.13 and lighttpd v1.4.11 both with 4 workers.

The overall result is interesting in that the performance was fairly similar regardless of the setup.

Firstly, raw performance serving a static 50k png file (1000 requests - concurrency 10)

Nginx:
Requests per second: 230.04 [num/sec] (mean)
Time per request: 43.470 [ms] (mean)
Time per request: 4.347 [ms] (mean, across all concurrent requests)
Transfer rate: 11408.07 [Kbytes/sec] received

Lighttpd:
Requests per second: 230.08 [num/sec] (mean)
Time per request: 43.463 [ms] (mean)
Time per request: 4.346 [ms] (mean, across all concurrent requests)
Transfer rate: 11407.51 [Kbytes/sec] received

Next, accessing the home page, approx. 10k in size (n=1000,c=1)

Nginx - fastcgi/flup:
Requests per second: 62.28 [num/sec] (mean)
Time per request: 16.056 [ms] (mean)
Time per request: 16.056 [ms] (mean, across all concurrent requests)
Transfer rate: 709.14 [Kbytes/sec] received

Nginx - proxy/Paste httpserver:
Requests per second: 61.44 [num/sec] (mean)
Time per request: 16.277 [ms] (mean)
Time per request: 16.277 [ms] (mean, across all concurrent requests)
Transfer rate: 698.18 [Kbytes/sec] received

Lighttpd - scgi/flup:
Requests per second: 64.20 [num/sec] (mean)
Time per request: 15.576 [ms] (mean)
Time per request: 15.576 [ms] (mean, across all concurrent requests)
Transfer rate: 737.02 [Kbytes/sec] received

Lighttpd - proxy:
Requests per second: 62.73 [num/sec] (mean)
Time per request: 15.942 [ms] (mean)
Time per request: 15.942 [ms] (mean, across all concurrent requests)
Transfer rate: 715.47 [Kbytes/sec] received

Now, hammering the home page (n=500,c=500)

Nginx - fastcgi/flup:
Fell over - locked up

Nginx - proxy/Paste httpserver:
Requests per second: 62.72 [num/sec] (mean)
Time per request: 7972.514 [ms] (mean)
Time per request: 15.945 [ms] (mean, across all concurrent requests)
Transfer rate: 712.70 [Kbytes/sec] received

Percentage of the requests served within a certain time (ms):
50% 3530
66% 3581
75% 3631
80% 3707
90% 7047
95% 7450
98% 7786
99% 7891
100% 7962 (longest request)

Lighttpd - scgi/flup:
Requests per second: 56.04 [num/sec] (mean)
Time per request: 8922.276 [ms] (mean)
Time per request: 17.845 [ms] (mean, across all concurrent requests)
Transfer rate: 647.14 [Kbytes/sec] received

Percentage of the requests served within a certain time (ms):
50% 4416
66% 5773
75% 6726
80% 7116
90% 7829
95% 8074
98% 8118
99% 8349
100% 8708 (longest request)

Lighttpd - proxy:
Requests per second: 63.57 [num/sec] (mean)
Time per request: 7865.190 [ms] (mean)
Time per request: 15.730 [ms] (mean, across all concurrent requests)
Transfer rate: 725.09 [Kbytes/sec] received

Percentage of the requests served within a certain time (ms):
50% 578
66% 709
75% 832
80% 909
90% 3666
95% 6848
98% 7341
99% 7408
100% 7537 (longest request)

Now, pushing the envelope a little on the home page (n=5000,c=100) - via proxy only

Nginx - proxy/Paste httpserver:
Requests per second: 63.79 [num/sec] (mean)
Time per request: 1567.653 [ms] (mean)
Time per request: 15.677 [ms] (mean, across all concurrent requests)
Transfer rate: 724.92 [Kbytes/sec] received

Percentage of the requests served within a certain time (ms):
50% 1502
66% 1540
75% 1562
80% 1577
90% 1623
95% 1735
98% 4571
99% 6996
100% 15587 (longest request)

Lighttpd - proxy:
Requests per second: 61.10 [num/sec] (mean)
Time per request: 1636.701 [ms] (mean)
Time per request: 16.367 [ms] (mean, across all concurrent requests)
Transfer rate: 696.90 [Kbytes/sec] received

Percentage of the requests served within a certain time (ms):
50% 1454
66% 1517
75% 1548
80% 1566
90% 1620
95% 3383
98% 4523
99% 12204
100% 31933 (longest request)

Based on these results, along with the reports of Lighttpd having some issues in its recent versions (see http://bob.pythonmac.org/archives/2006/09/13/nginx-reverse-proxy-panacea/, using Nginx as a frontend proxy to Pylons appears to be a good bet, particularly as it supports load balancing of multiple Pylons instances.