Using NGINX to proxy TileStache

2012-08-30 12:54:18 -0400

I'm working on a re-rendering of OpenStreetMap for hiking, with hill shading and topographic lines and I decided to use TileStache to render the tiles. TileStache has the nice ability to render tiles from a bunch of different sources and then overlay them with different amounts of transparency and layer masks (just like Photoshop). TileStache is a Python WSGI server that you can run in mod_python or GUnicorn to serve tiles directly over HTTP. TileStache can cache map tiles to the file system and serve the static PNGs if they exist or render them from scratch using Mapnik if they don't. Its pretty fast, especially if the tiles are pre-rendered.

However, GUnicorn is a pre-forking server. This means that it needs to fork a different process for each client connection. What happens if a slow client connects is that TileStache processes are rapidly used up to serve that client (typically clients make up to 8 separate HTTP connections for slippy maps, resulting in 8 processes each!). This is the case even if the tiles are being served from cache.

What you need to do is add a reverse proxy in front of GUnicorn, using something like NGINX. The reverse proxy using an evented IO model, which enables it to manage sending data back to a slow client without using an operating system process. NGINX can also directly serve static assets from the filesystem, which means we can serve the cached tiles without even hitting GUnicorn/TileStache.

Getting this to work requires a bit of NGINX HttpRewriteModule voodoo, though. The issue is that TileStache saves cached tiles in a slightly different path than the URI path that comes in via HTTP. Say you have a OpenStreetMap-style URL like this: myserver.com/tiles/layer/$z/$x/$y.png</code>. In this URL, $z</code> is zoom level (usually 1-19), and $x</code> and $y</code> are tile coordinates. For higher zoom levels, you can have 10,000+ by 10,0000+ tiles in the x and y directions. That's way too many PNG files to store in one folder on the filesystem. So, TileStache splits up the x and y paths into two levels. Say you have a URL like /tiles/layer/7/12345/67890.png</code>. TileStache will store that in the filesystem path /tiles/layer/7/012/345/067/890.png</code>. Notice how 12345 is broken into 012/345? That means that there will be at most 1,000 files or folders in each directory—a manageable amount. The issue is we need to get NGINX to rewrite URLs to server these static assets. Here's how I accomplished that:

	location ~ ^\/tiles\/([\w\-]+)\/([\d]+)\/([\d]+)\/([\d]+)\.png {
root /osm/cache;

set $originaluri /$1/$2/$3/$4.png;

# 1 char X
rewrite "^\/tiles\/([\w\-]+)\/([\d]+)\/([\d]{1})\/([\d]{1}).png$" /$1/$2/000/00$3/000/00$4.png break;
rewrite "^\/tiles\/([\w\-]+)\/([\d]+)\/([\d]{1})\/([\d]{2}).png$" /$1/$2/000/00$3/000/0$4.png break;
rewrite "^\/tiles\/([\w\-]+)\/([\d]+)\/([\d]{1})\/([\d]{3}).png$" /$1/$2/000/00$3/000/$4.png break;
rewrite "^\/tiles\/([\w\-]+)\/([\d]+)\/([\d]{1})\/([\d]{1})([\d]{3}).png$" /$1/$2/000/00$3/00$4/$5.png break;
rewrite "^\/tiles\/([\w\-]+)\/([\d]+)\/([\d]{1})\/([\d]{2})([\d]{3}).png$" /$1/$2/000/00$3/0$4/$5.png break;
rewrite "^\/tiles\/([\w\-]+)\/([\d]+)\/([\d]{1})\/([\d]{3})([\d]{3}).png$" /$1/$2/000/00$3/$4/$5.png break;

# 2 char X
rewrite "^\/tiles\/([\w\-]+)\/([\d]+)\/([\d]{2})\/([\d]{1}).png$" /$1/$2/000/0$3/000/00$4.png break;
rewrite "^\/tiles\/([\w\-]+)\/([\d]+)\/([\d]{2})\/([\d]{2}).png$" /$1/$2/000/0$3/000/0$4.png break;
rewrite "^\/tiles\/([\w\-]+)\/([\d]+)\/([\d]{2})\/([\d]{3}).png$" /$1/$2/000/0$3/000/$4.png break;
rewrite "^\/tiles\/([\w\-]+)\/([\d]+)\/([\d]{2})\/([\d]{1})([\d]{3}).png$" /$1/$2/000/0$3/00$4/$5.png break;
rewrite "^\/tiles\/([\w\-]+)\/([\d]+)\/([\d]{2})\/([\d]{2})([\d]{3}).png$" /$1/$2/000/0$3/0$4/$5.png break;
rewrite "^\/tiles\/([\w\-]+)\/([\d]+)\/([\d]{2})\/([\d]{3})([\d]{3}).png$" /$1/$2/000/0$3/$4/$5.png break;

# 3 char X
rewrite "^\/tiles\/([\w\-]+)\/([\d]+)\/([\d]{3})\/([\d]{1}).png$" /$1/$2/000/$3/000/00$4.png break;
rewrite "^\/tiles\/([\w\-]+)\/([\d]+)\/([\d]{3})\/([\d]{2}).png$" /$1/$2/000/$3/000/0$4.png break;
rewrite "^\/tiles\/([\w\-]+)\/([\d]+)\/([\d]{3})\/([\d]{3}).png$" /$1/$2/000/$3/000/$4.png break;
rewrite "^\/tiles\/([\w\-]+)\/([\d]+)\/([\d]{3})\/([\d]{1})([\d]{3}).png$" /$1/$2/000/$3/00$4/$5.png break;
rewrite "^\/tiles\/([\w\-]+)\/([\d]+)\/([\d]{3})\/([\d]{2})([\d]{3}).png$" /$1/$2/000/$3/0$4/$5.png break;
rewrite "^\/tiles\/([\w\-]+)\/([\d]+)\/([\d]{3})\/([\d]{3})([\d]{3}).png$" /$1/$2/000/$3/$4/$5.png break;

# 4 char X
rewrite "^\/tiles\/([\w\-]+)\/([\d]+)\/([\d]{1})([\d]{3})\/([\d]{1}).png$" /$1/$2/00$3/$4/000/00$5.png break;
rewrite "^\/tiles\/([\w\-]+)\/([\d]+)\/([\d]{1})([\d]{3})\/([\d]{2}).png$" /$1/$2/00$3/$4/000/0$5.png break;
rewrite "^\/tiles\/([\w\-]+)\/([\d]+)\/([\d]{1})([\d]{3})\/([\d]{3}).png$" /$1/$2/00$3/$4/000/$5.png break;
rewrite "^\/tiles\/([\w\-]+)\/([\d]+)\/([\d]{1})([\d]{3})\/([\d]{1})([\d]{3}).png$" /$1/$2/00$3/$4/00$5/$6.png break;
rewrite "^\/tiles\/([\w\-]+)\/([\d]+)\/([\d]{1})([\d]{3})\/([\d]{2})([\d]{3}).png$" /$1/$2/00$3/$4/0$5/$6.png break;
rewrite "^\/tiles\/([\w\-]+)\/([\d]+)\/([\d]{1})([\d]{3})\/([\d]{3})([\d]{3}).png$" /$1/$2/00$3/$4/$5/$6.png break;

# 5 char X
rewrite "^\/tiles\/([\w\-]+)\/([\d]+)\/([\d]{2})([\d]{3})\/([\d]{1}).png$" /$1/$2/0$3/$4/000/00$5.png break;
rewrite "^\/tiles\/([\w\-]+)\/([\d]+)\/([\d]{2})([\d]{3})\/([\d]{2}).png$" /$1/$2/0$3/$4/000/0$5.png break;
rewrite "^\/tiles\/([\w\-]+)\/([\d]+)\/([\d]{2})([\d]{3})\/([\d]{3}).png$" /$1/$2/0$3/$4/000/$5.png break;
rewrite "^\/tiles\/([\w\-]+)\/([\d]+)\/([\d]{2})([\d]{3})\/([\d]{1})([\d]{3}).png$" /$1/$2/0$3/$4/00$5/$6.png break;
rewrite "^\/tiles\/([\w\-]+)\/([\d]+)\/([\d]{2})([\d]{3})\/([\d]{2})([\d]{3}).png$" /$1/$2/0$3/$4/0$5/$6.png break;
rewrite "^\/tiles\/([\w\-]+)\/([\d]+)\/([\d]{2})([\d]{3})\/([\d]{3})([\d]{3}).png$" /$1/$2/0$3/$4/$5/$6.png break;

# 6 char X
rewrite "^\/tiles\/([\w\-]+)\/([\d]+)\/([\d]{3})([\d]{3})\/([\d]{1}).png$" /$1/$2/$3/$4/000/00$5.png break;
rewrite "^\/tiles\/([\w\-]+)\/([\d]+)\/([\d]{3})([\d]{3})\/([\d]{2}).png$" /$1/$2/$3/$4/000/0$5.png break;
rewrite "^\/tiles\/([\w\-]+)\/([\d]+)\/([\d]{3})([\d]{3})\/([\d]{3}).png$" /$1/$2/$3/$4/000/$5.png break;
rewrite "^\/tiles\/([\w\-]+)\/([\d]+)\/([\d]{3})([\d]{3})\/([\d]{1})([\d]{3}).png$" /$1/$2/$3/$4/00$5/$6.png break;
rewrite "^\/tiles\/([\w\-]+)\/([\d]+)\/([\d]{3})([\d]{3})\/([\d]{2})([\d]{3}).png$" /$1/$2/$3/$4/0$5/$6.png break;
rewrite "^\/tiles\/([\w\-]+)\/([\d]+)\/([\d]{3})([\d]{3})\/([\d]{3})([\d]{3}).png$" /$1/$2/$3/$4/$5/$6.png break;

# Try to serve the file from the disk. If that doesn't work, pass through the request via the proxy
try_files $uri @tilestache;
}</pre>
This mountain of rewrite lines will rewrite the request URL to the filesystem format, then look for tiles in the filesystem tree starting at /osm/cache</code>. The last line tells NGINX to look for the rewritten URL, then if the file is not found, to send the request to the @tilestache;</code> location block, which looks like this:

	 location @tilestache {

# Rewrite back to standard OSM form
rewrite ^(.*)$ $originaluri break;
add_header X-Static miss;
proxy_pass http://127.0.0.1:8080;
proxy_set_header Host $http_host;
}</pre>
That location block proxies the request to the GUnicorn server listening on localhost:8080.

This seems to be working great. NGINX is far faster in serving static assets, and if all of the worker TileStache processes are busy rendering, the cached zoom levels of the map work fine!