HTTP2 and 421 errors

I am in the process of migrating an online B2B enterprise built on design patterns from a 1990's dial-up ISP to a more modern infrastructure. It has taken over 20 years of effort to build a network of services which are difficult to manage to and insecure. Replacing this, with no downtime, is something of a challenge. A really important stepping stone to the target architecture is getting every exposed service routed via a proxy. This makes it much, MUCH simpler for us to:

• (re)route services within our network
• Upgrade services to HTTP2
• Provision certificates / manage encryption
• Configure browser-side caching
• Analyse traffic
• ...and more 

Running a handful of BigIp F5s is unfortunately not an option, so my switchboard is a stack of Ubuntu + nginx.

Up till recently the onboarding exercise had gone really well - but then I started encountering 421 errors. My initial reading kept leading me to old bugs in Chrome and other issues with HTTP2. These are neatly summarized by Kevin as follows:

This is caused by the following sequence of events:

1. The server and client both support and use HTTP/2.
2. The client requests a page at foo.example.com.
3. During TLS negotiation, the server presents a certificate which is valid for both foo.example.com and bar.example.com (and the client accepts it). This could be done with a wildcard certificate or a SAN certificate.
4. The client reuses the connection to make a request for bar.example.com.
5. The server is unable or unwilling to support cross-domain connection reuse (for example because you configured their SSL differently and Apache wants to force a TLS renegotiation), and serves HTTP 421.
6. The client does not automatically retry with a new connection (see for example Chrome bug #546991, now fixed). The relevant RfC says that the client MAY retry, not that it SHOULD or MUST. Failing to retry is not particularly user-friendly, but might be desirable for a debugging tool or HTTP library.

However, I was able to reproduce the bug in the first request from a browser instance I had just started. Also I was able to see the issue across sites using distinct certificates. I was confused.

Eventually I looked at the logs for the origin server (Apache 2.4.26). I hadn't considered that before as I knew it did not support HTTP2. But low and behold, there in the logs, a 421 error against my request.

[Fri Jun 12 18:44:47.945706 2020] [ssl:error] [pid 21423:tid 140096556701440] AH02032: Hostname foo.example.net provided via SNI and hostname bar.example.com provided via HTTP have no compatible SSL setup

So disabling SSL session re-use on the connection from the nginx proxy to the origin resolved the issue:

proxy_ssl_session_reuse off;

This does mean slightly more overhead between the proxy and the origin, but since both are on the same LAN, its not really noticeable.

Although its something of a gray area, I don't think this is a bug with nginx - I had multiple sites in nginx pointing at the same backend URL. It would be interesting to check if the issue occurs when I have multiple unique DNS names for the origin - one for each nginx front end - if it still occurs there, then that is probably a bug.

Microsoft - unsafe at any OS

At $WORK we are trying to move from a traditional model of software development and deployment to agile methods. They were already well down the road of using Microsoft DevOps when I joined the party.
This week I was working on setting up the deployment targets. This uses a Microsoft supplied agent running on the target machine which has a constant connection out to Microsoft Azure pipelines.That avoids the whole issue of firewalls, but that is where the thinking ended.

As you would expect, there a massive feature set in the tool chain - with lots of integration. And most of it actually seems to work! But my experience with Microsoft products is that most of the big issues arise in the things that are generally lumped together under the "non-functional requirements".

The installation takes around half a gigabyte. I've seen smaller, full-featured Linux distributions. That might be because it contains a whole lot of DLL files. DLL? On Linux? Is this a dual Linux and MS-Windows installation? Or have they implemented their own runtime linker?

The daemon part is node.js  - That's good. Its a sensible bit of software, although a bit heavy for microservice containers. And there is a large bundle of Javascript files.

But according to the github page,It is "Written for .NET Core in C#."

But the elephant in the room is that it runs as root.

There's absolutely no requirement for this to run as root.

It is just a huge security risk. The Devops user does not control permissions, the system administrator does not control permissions, Microsoft control permissions.

Loving nginx - but still loving apache

I am now running nginx on my production servers.

Having written a book which features the Apache webserver heavily, a lot of people seem surprised when I tell them I am migrating my work environments onto nginx.

Of course the truth is not nearly as black and white as one product always being better than another; nginx is simply a better fit for what I, and the organization I work for need for the foreseeable future. I believe and practice the philosophy of strong opinions weakly held

We are constrained by computing resource. A lot of this arises from bad choices made by the previous admin and lack of governance. There is no way to sugar-coat or justify the practices which were the norm for a very long time. One of several undesirable outcomes is that the systems are not nearly as efficient as they should be. I still believe you can do more with Apache, but that comes at a slightly higher cost in terms of CPU and memory. Certainly if you don't track what resources are assigned for what purpose, you will never achieve any operational efficiency, regardless what tech you use.

Another major consideration is the availability of skills. I am not the only systems admin in the company, I would rather the systems were maintained by people with strong skills in one product than a competency in lots. Maybe one day tech will be as simple as the sales guy like to pretend, but it looks to me that skills will continue to trump tech for a very long time. But the choice of product does have an impact. While Apache can be used as a forward or reverse proxy, this is not its metier - if I had decided to run Apache httpd as the origin webserver I would have been looking at Apache Traffic Server, nginx or even pound at the front end. Being able to use the same product in both places means less cognitive load.

Nginx (along with haproxy for client side and non-HTTP traffic) provides a powerful toolset for distributing traffic around your infrastructure.

If I'm being completely honest, nginx's lack of functionality also looks like a benefit. The Apache config files for the legacy servers were humungous. And more than half of what they contained should never have been there in the first place: thousands of redirect rules for content which had moved, retired or simply put in the wrong place to begin with. Different ssl configurations for every browser released in the last 20 years. Servers writing the same request to four different log files in different formats which nobody ever read. Despite all this config, it was alarming what was missing - no content caching, no keepalives, no process limits!

That's not to say nginx is easy or minimal in its capabilities. With a syntax which looks very much like C or PHP: blocks bounded with curly brackets: statements ending in semi colons: it is easy to assume that the syntax is procedural. That is not the case. It is, as one would expect without looking at the syntax, declarative. Failing to understand this can lead to problems and frustration.

Another misdirection arising from the syntax is an assumption that the hierarchy of the configuration is merely structural - again not true. Many of the directives are very contextual.

But I do miss Apache's server-status page. Running php-fpm means you still need to worry about concurrent connections but nginx's server status metrics don't show the number of back end connections / where they are connected. Creeping up my todo list is to write something to present useful information about these connections and other metrics on the servers.

De-duplication and Opcache

As I now have a lot of Wordpress sites to look after (argh!) I wanted to see if I coiuld set them up to be a bit more efficient in memory. However due to the way Wordpress resolves the location of files, it would have required the wp-settings.php file to be almost completely replaced to set the paths in PHP. I wondered if I could use symlinks on the filesystem to achieve the same goal without hacking the code. The answer appears to be yes - here's the output from my test case:

---

Opcache and symlinks

This script includes the same file via different paths which use symlinks. The objective is determine whether this creates 1 or 2 entries in opcache - and hence whether I can run multiple Wordpress sites from the same files without rewriting the code

Include from linked1 : This is /var/www/html/myvhost/include/testsymlink.php
include from linked2 : This is /var/www/html/myvhost/include/testsymlink.php

/var/www/html/myvhost/include/testsymlink.php
Array
(
    [full_path] => /var/www/html/myvhost/include/testsymlink.php
    [hits] => 9
    [memory_consumption] => 736
    [last_used] => Mon Oct  7 10:31:53 2019
    [last_used_timestamp] => 1570444313
    [timestamp] => 1570443328
)

2 files included resolve to a single entry in opcache - yay!
Note that some caution is required when applying upgrades to the wordpress install!

Source code for this script

<?php
print "<h1>Opcache and symlinks</h1>";
print "<p>This script includes the same file via different paths which use symlinks. The objective is determine whether this creates 1 or 2 entries in opcache - and hence whether I can run multiple Wordpress sites from the same files without rewriting the code</p><p>\n";
print "Include from linked1 : ";
include "linked1/testsymlink.php";
print "include from linked2 : ";
include "linked2/testsymlink.php";

print "<pre>";
$data=opcache_get_status(true);
foreach ($data['scripts'] as $script=>$sd) {
   if ("testsymlink.php"==basename($script)) {
       print $script . "\n";
       print_r($sd);
   }
}
print "</pre>";

print "2 files included resolve to a single entry in opcache - yay!<br />\n";
print "Note that some caution is required when applying upgrades to the wordpress install!<br />\n";    
print "<h2>Source code for this script</h2>";
highlight_file(__FILE__);