Low-latency caching HTTP proxy with user-viewable cache directory

Question

I have been looking for a caching HTTP proxy for Linux with the following properties:

• Stand-alone: I'd rather not have to mess with e.g. my existing Apache or Squid configuration.

• Low-latency: I.e. it should forward any data as it is retrieved, without waiting for the whole file to be downloaded first. That said, its overall performance is not of particular concern - usually it will only have at most one or two clients, downloading files in the 1-100MB range.

• Caching using a format that allows the cached files to be used and/or modified directly. E.g. http://example.com/a/b/file.txt could be placed in <cachedir>/http/example.com/a/b/file.txt.

• (Optional) A degree of configurability so that e.g. all www*.example.com URLs would map to the same directory to avoid duplicating mirrors. A Python program that one can modify a bit could also be suitable...

I don't need to actually modify headers or file content - the proxy will be mostly used to share software update files to several VMs and physical machines without straining the connection to the Internet unnecessarily.

Answer 1

Polipo

should fit the bill, with one caveat.

Stand-alone: I'd rather not have to mess with e.g. my existing Apache or Squid configuration.

Just install it (no dependencies other than the standard library) and configure as necessary (you'll probably want to change the port if you already have Squid listening on 8123, and you'll want to allow clients other than localhost. Also given your requirements you may want to turn off asynchronous writing.

Low-latency: I.e. it should forward any data as it is retrieved, without waiting for the whole file to be downloaded first. That said, its overall performance is not of particular concern - usually it will only have at most one or two clients, downloading files in the 1-100MB range.

Check. (Unlike wwwoffle, which I quite like but systematically buffers pages until they're fully downloaded.)

Caching using a format that allows the cached files to be used and/or modified directly. E.g. http://example.com/a/b/file.txt could be placed in <cachedir>/http/example.com/a/b/file.txt.

Close. Proxies don't normally store file.txt in a file called file.txt because URLs don't fit well with file syntax: it's possible for foo and foo/ or foo/bar and foo//bar to serve different contents, for example. Polipo does store one file per file but the file name is the MD5 hash of the URL (encoded in Base64), under a directory named like the host.

Furthermore the content of the file includes the headers. If you want the actual content, you'll need to strip the headers, e.g. with sed -e '1,/^\r\?$/d'.

I'm mentioning Polipo despite these limitations because most proxies are likely to work like this, unless they're meant to be used in restricted setups where they can't access all web pages faithfully.

(Optional) A degree of configurability so that e.g. all www*.example.com URLs would map to the same directory to avoid duplicating mirrors. A Python program that one can modify a bit could also be suitable...

Doable with redirected URLs.

Answer 2

I'd wager exactly this exists, but just in case it does not :

CernVM-FS (docs,github) interprets a fixed website as a filesystem with long term caching. If you wanted to access it as a website again, then you could spin up a local HTTP server on that directory.

This solution is not stand-alone, and handles only one website, but maybe still relevant.