httpd: deliver compressed static files (+slowcgi)

Article published the28 May 2020 ; modified the9 June 2021
7 minutes to read

This article has 1477 words.
RAW source of the article:
Commit version: e21600e

Description

OpenBSD has, by default, in basesystem:

a webserver, named httpd, since 5.7
a server CGI, named slowcgi, since 5.4
OpenBSD : 6.6, 6.7

Principle: if web client accepts datas compression in the deflate, gzip, brotli format, then httpd redirects to slowcgi to deliver the compressed static content.

Warning

This is not means compression dynamicly, or “compression on the fly!”

httpd is not able to manage delivery about compressed static content.

The tip is to use the server slowcgi CGI for.

In facts, by CGI script shell, we’ll assume the delivery of those compressed static content:

for the format: atom, css, html, js, json, svg, txt, xml
to both compression: gzip, and brotli.

Installation

You do to download my script sbw.cgi.

When it’s done, you need to put into the cgi-bin folder on web chroot. For this, use the command install as: install -o www -g bin -m 0550 sbw.cgi /var/www/cgi-bin/sbw.cgi

Indeed, we install correctly with minimals/needed rights to the web user www and the group bin.

Dependencies

This script need the installation in the web chroot of several binaries and some libraries, to run correctly:

Copy from:

first: sh, and binaries: cat, date and sha256 — which are all in the /bin system directory.
all others binaries: basename, logger 1 , stat - which are all in the /usr/bin system directory.
the shared libc: /usr/lib/libc.so.xx.0 2
the library to execution: /usr/libexec/ld.so

to the respective folders into web chroot /var/www/.

All need root user and bin group rights:

with 0555 mode for the binaries, e.g.:
- sh: install -o root -g bin -m 0555 /bin/sh /var/www/bin/
- stat: install -o root -g bin -m 0555 /usr/bin/stat /var/www/usr/bin/
and 0444 mode for the libraries, as:
- libc: install -o root -g bin -m 0444 /usr/lib/libc.so.xx.0 /var/www/usr/lib/
- ld.so: install -o root -g bin -m 0444 /usr/libexec/ld.so /var/www/usr/libexec/

Before, you need to create the corresponding directories in the web chroot!

See my dependencies script.

1 about the interest of the binary logger: Normaly, ideally, we don’t need the logger. It is about logging some actions, which in case of failure, have written in the logs /var/log/{daemon,messages}.

Also, if the debug variable is set to 1, on the main function, then the logger will return the values corresponding to the different variables, for analysis: ensure that variable receive one value, and what value‽

2 The libc.so change name, at each version of OpenBSD:

v6.7 : libc.so.96.0
v6.6 : libc.so.95.1

This detail is important, can be hardly scripted. You need to modify the script, at the new version of OpenBSD, to change the name of the library, otherwise, it will not work!

Tip

To known, what are the dependencies from a binary, use the command ldd.

Configuration

httpd

Add at your context server, all following needed location statements:

Code: httpd

location "/*.atom" { include "/etc/httpd.d/sbw.conf" }
location "/*.css" { include "/etc/httpd.d/sbw.conf" }
location "/*.html" { include "/etc/httpd.d/sbw.conf" }
location "/*.js" { include "/etc/httpd.d/sbw.conf" }
location "/*.json" { include "/etc/httpd.d/sbw.conf" }
location "/*.svg" { include "/etc/httpd.d/sbw.conf" }
location "/*.txt" { include "/etc/httpd.d/sbw.conf" }
location "/*.xml" { include "/etc/httpd.d/sbw.conf" }

The file sbw.conf contains the following fastcgi statements:

File: /etc/httpd.d/sbw.conf

root "/cgi-bin/sbw.cgi"
fastcgi param realroot "/htdocs/domaine.tld/www" 
fastcgi param cachecontrol "1814400" 
fastcgi param file404 "/404.html"

Explainations :

It’s important to define, at least:

root: the relative path of script CGI, into the web chroot.
realroot: a parameter for your web root
and optionals paramaters, specially of the cache.

slowcgi

The server slowcgi does not require any configuration. Only, you enable and start with the tool rcctl.

History

Vulnerabilities

In fact, the story of the HTTP protocol reveals us two majors vulnerabilities related to the on-the-fly compression: CRIME and BREACH — the second is forked on the first. Thoses vulnerabilities can even impact TLS .

Among the countermeasures has been the adoption since HTTP 1.1 of block encoding transfer — the famous Transfer-Encoding: chunked header; similarly, it is strongly recommended to implement a Referrer policy to allow delivery of compressed content ONLY from the current domain, and to refuse it from any other domain.

about httpd

Reyk Floeter, the httpd author/developer, denies support for the compressed content. Even, one request had be done to support pre-compressed content, it’s not ready to be integrated.

brotli

brotli is a compression format invented by a team from the company Google. It is considered to be the successor to gzip because it is faster and has a better compression ratio.

For more informations, see:

https://brotli.org/
Does your web client support it: https://caniuse.com/#search=brotli

curl, since v7.57.0, support brotli, by adding the option --compressed, or -H — this manage finely HTTP headers. (see the manpage)*

unsupported clients

On OpenBSD, curl seems not supported brotli.
Egual, on all OS, lynx, w3m does not support it.

Firefox

Since v64, Firefox does not support Atom or RSS feed.

Actually, it’s more subtle than it sounds:

if you deliver Atom or RSS with mime type text/xml, both are XML files, Firefox accepts to read and native display.
if you deliver it with their mime type, respectively application/atom+xml and application/rss+xml, a RFC standard, then Firefox ask you what to do with.

“A nameless aberration!!!”

the little story

For the little story, Xavier Cartron @prx is the original author of this genious idea to deliver compressed static content, by adding the header ETag. This is an id for the delivered ressource.

It is in this context, that the binary sha256 is useful.

My work, based on his first version, was to add several things:

About brotli , so I resumed/continued writing in order to be able to deliver static content previously compressed with this format.

Next, I added code to manage others needed headers:

Content-Length: to send weight of delivered document. It is in this context, that the binary stat is useful.
Last-Modified: to get the modification date of the delivered document; someone considers this header is more relevant than ETag. It is in this context, that the binaries date and stat are useful.
Transfer-Encoding: to send the delivered document with the good compression format, if necessary.

Then, I wrote the necessary code to detect if the useful dependencies were in the web chroot, otherwise the script can’t work. If it’s the case, the serveur send an error 500, with an explicit HTML message.

ATTENTION: the script not installs and can not install the dependancies; because, it on the web chroot, it can not “view” the OS filesystem.

Next, after some research on the web, I understant that the format deflate, that may be requested by some web clients, is managed by the format gzip; then, the script supports too.

But I was confronted with dysfunctions that I couldn’t understand, let alone solve. I stopped the project :(

But, two “things” helped seriously to continue:

Solène Rapenne, from the OpenBSD team, helped to understand I was making the mistake of sending too many line breaks, when I need only one, at the right time, one between the sending of the different headers and the document itself, whether it is compressed or not.
the idea to implement a variable debug and use the binary logger to ensure some differents returns.

One tips that Solène gave me is the local use of this command: env HTTP_ACCEPT_ENCODING=br realroot=/var/www/htdocs/domaine.tld/dev/ PATH_INFO=index.html /var/www/cgi-bin/sbw.cgi | less explaining me that it is possible to query the CGI server locally directly, by sending it the different possible values before the call. Amazing! :D

About this, since we installed the CGI shell script sbw.cgi with the user rights to 0550, we had this erreur: env: /var/www/cgi-bin/sbw2.cgi: Permission denied you need to change the other rights to +x (or, 0551). ;-)

I improved the detection of the mime type by getting it from the file called on the filesystem — this need to use the binary basename - and the management of the mime type about feed Atom or RSS.

Finally, I wrote the necessary code to detect if the user agent was Firefox :

If yes, to obtain his version number. A little hack to deliver feed Atom and RSS to the “false” mime-type text/xml. Also, Firefox accepts to read the feed instead asking to open with another application!

I wrote the first draft of this hack requiring the addition of the binaries grep and awk — but, this functional solution did not satisfy me. When, an user on the forum about the french community “OpenBSD pour tous”, @eol makes me the think to use shell expansion.

“Et, voilà!”

Actualy, @prx rewrote his script in C:

the advantage is that there is no need for any dependency; too, it’s “protected” by the system call security measures pledge(2) and unveil(2).
however, it ONLY supports gzip compression; egual, no header Last-Modified, or brotli, and deflate support, at least not directly|automatically.

Documentation

For more documentation about referrer policy.

manpage

Wikipédia

About the BREACH vulnerability: https://en.wikipedia.org/wiki/BREACH
and the CRIME exploit: https://en.wikipedia.org/wiki/CRIME_(security_exploit)