HTTP2 and 421 errors

I am in the process of migrating an online B2B enterprise built on design patterns from a 1990's dial-up ISP to a more modern infrastructure. It has taken over 20 years of effort to build a network of services which are difficult to manage to and insecure. Replacing this, with no downtime, is something of a challenge. A really important stepping stone to the target architecture is getting every exposed service routed via a proxy. This makes it much, MUCH simpler for us to:

• (re)route services within our network
• Upgrade services to HTTP2
• Provision certificates / manage encryption
• Configure browser-side caching
• Analyse traffic
• ...and more 

Running a handful of BigIp F5s is unfortunately not an option, so my switchboard is a stack of Ubuntu + nginx.

Up till recently the onboarding exercise had gone really well - but then I started encountering 421 errors. My initial reading kept leading me to old bugs in Chrome and other issues with HTTP2. These are neatly summarized by Kevin as follows:

This is caused by the following sequence of events:

1. The server and client both support and use HTTP/2.
2. The client requests a page at foo.example.com.
3. During TLS negotiation, the server presents a certificate which is valid for both foo.example.com and bar.example.com (and the client accepts it). This could be done with a wildcard certificate or a SAN certificate.
4. The client reuses the connection to make a request for bar.example.com.
5. The server is unable or unwilling to support cross-domain connection reuse (for example because you configured their SSL differently and Apache wants to force a TLS renegotiation), and serves HTTP 421.
6. The client does not automatically retry with a new connection (see for example Chrome bug #546991, now fixed). The relevant RfC says that the client MAY retry, not that it SHOULD or MUST. Failing to retry is not particularly user-friendly, but might be desirable for a debugging tool or HTTP library.

However, I was able to reproduce the bug in the first request from a browser instance I had just started. Also I was able to see the issue across sites using distinct certificates. I was confused.

Eventually I looked at the logs for the origin server (Apache 2.4.26). I hadn't considered that before as I knew it did not support HTTP2. But low and behold, there in the logs, a 421 error against my request.

[Fri Jun 12 18:44:47.945706 2020] [ssl:error] [pid 21423:tid 140096556701440] AH02032: Hostname foo.example.net provided via SNI and hostname bar.example.com provided via HTTP have no compatible SSL setup

So disabling SSL session re-use on the connection from the nginx proxy to the origin resolved the issue:

proxy_ssl_session_reuse off;

This does mean slightly more overhead between the proxy and the origin, but since both are on the same LAN, its not really noticeable.

Although its something of a gray area, I don't think this is a bug with nginx - I had multiple sites in nginx pointing at the same backend URL. It would be interesting to check if the issue occurs when I have multiple unique DNS names for the origin - one for each nginx front end - if it still occurs there, then that is probably a bug.

Microsoft - unsafe at any OS

At $WORK we are trying to move from a traditional model of software development and deployment to agile methods. They were already well down the road of using Microsoft DevOps when I joined the party.
This week I was working on setting up the deployment targets. This uses a Microsoft supplied agent running on the target machine which has a constant connection out to Microsoft Azure pipelines.That avoids the whole issue of firewalls, but that is where the thinking ended.

As you would expect, there a massive feature set in the tool chain - with lots of integration. And most of it actually seems to work! But my experience with Microsoft products is that most of the big issues arise in the things that are generally lumped together under the "non-functional requirements".

The installation takes around half a gigabyte. I've seen smaller, full-featured Linux distributions. That might be because it contains a whole lot of DLL files. DLL? On Linux? Is this a dual Linux and MS-Windows installation? Or have they implemented their own runtime linker?

The daemon part is node.js  - That's good. Its a sensible bit of software, although a bit heavy for microservice containers. And there is a large bundle of Javascript files.

But according to the github page,It is "Written for .NET Core in C#."

But the elephant in the room is that it runs as root.

There's absolutely no requirement for this to run as root.

It is just a huge security risk. The Devops user does not control permissions, the system administrator does not control permissions, Microsoft control permissions.

Loving nginx - but still loving apache

I am now running nginx on my production servers.

Having written a book which features the Apache webserver heavily, a lot of people seem surprised when I tell them I am migrating my work environments onto nginx.

Of course the truth is not nearly as black and white as one product always being better than another; nginx is simply a better fit for what I, and the organization I work for need for the foreseeable future. I believe and practice the philosophy of strong opinions weakly held

We are constrained by computing resource. A lot of this arises from bad choices made by the previous admin and lack of governance. There is no way to sugar-coat or justify the practices which were the norm for a very long time. One of several undesirable outcomes is that the systems are not nearly as efficient as they should be. I still believe you can do more with Apache, but that comes at a slightly higher cost in terms of CPU and memory. Certainly if you don't track what resources are assigned for what purpose, you will never achieve any operational efficiency, regardless what tech you use.

Another major consideration is the availability of skills. I am not the only systems admin in the company, I would rather the systems were maintained by people with strong skills in one product than a competency in lots. Maybe one day tech will be as simple as the sales guy like to pretend, but it looks to me that skills will continue to trump tech for a very long time. But the choice of product does have an impact. While Apache can be used as a forward or reverse proxy, this is not its metier - if I had decided to run Apache httpd as the origin webserver I would have been looking at Apache Traffic Server, nginx or even pound at the front end. Being able to use the same product in both places means less cognitive load.

Nginx (along with haproxy for client side and non-HTTP traffic) provides a powerful toolset for distributing traffic around your infrastructure.

If I'm being completely honest, nginx's lack of functionality also looks like a benefit. The Apache config files for the legacy servers were humungous. And more than half of what they contained should never have been there in the first place: thousands of redirect rules for content which had moved, retired or simply put in the wrong place to begin with. Different ssl configurations for every browser released in the last 20 years. Servers writing the same request to four different log files in different formats which nobody ever read. Despite all this config, it was alarming what was missing - no content caching, no keepalives, no process limits!

That's not to say nginx is easy or minimal in its capabilities. With a syntax which looks very much like C or PHP: blocks bounded with curly brackets: statements ending in semi colons: it is easy to assume that the syntax is procedural. That is not the case. It is, as one would expect without looking at the syntax, declarative. Failing to understand this can lead to problems and frustration.

Another misdirection arising from the syntax is an assumption that the hierarchy of the configuration is merely structural - again not true. Many of the directives are very contextual.

But I do miss Apache's server-status page. Running php-fpm means you still need to worry about concurrent connections but nginx's server status metrics don't show the number of back end connections / where they are connected. Creeping up my todo list is to write something to present useful information about these connections and other metrics on the servers.

De-duplication and Opcache

As I now have a lot of Wordpress sites to look after (argh!) I wanted to see if I coiuld set them up to be a bit more efficient in memory. However due to the way Wordpress resolves the location of files, it would have required the wp-settings.php file to be almost completely replaced to set the paths in PHP. I wondered if I could use symlinks on the filesystem to achieve the same goal without hacking the code. The answer appears to be yes - here's the output from my test case:

---

Opcache and symlinks

This script includes the same file via different paths which use symlinks. The objective is determine whether this creates 1 or 2 entries in opcache - and hence whether I can run multiple Wordpress sites from the same files without rewriting the code

Include from linked1 : This is /var/www/html/myvhost/include/testsymlink.php
include from linked2 : This is /var/www/html/myvhost/include/testsymlink.php

/var/www/html/myvhost/include/testsymlink.php
Array
(
    [full_path] => /var/www/html/myvhost/include/testsymlink.php
    [hits] => 9
    [memory_consumption] => 736
    [last_used] => Mon Oct  7 10:31:53 2019
    [last_used_timestamp] => 1570444313
    [timestamp] => 1570443328
)

2 files included resolve to a single entry in opcache - yay!
Note that some caution is required when applying upgrades to the wordpress install!

Source code for this script

<?php
print "<h1>Opcache and symlinks</h1>";
print "<p>This script includes the same file via different paths which use symlinks. The objective is determine whether this creates 1 or 2 entries in opcache - and hence whether I can run multiple Wordpress sites from the same files without rewriting the code</p><p>\n";
print "Include from linked1 : ";
include "linked1/testsymlink.php";
print "include from linked2 : ";
include "linked2/testsymlink.php";

print "<pre>";
$data=opcache_get_status(true);
foreach ($data['scripts'] as $script=>$sd) {
   if ("testsymlink.php"==basename($script)) {
       print $script . "\n";
       print_r($sd);
   }
}
print "</pre>";

print "2 files included resolve to a single entry in opcache - yay!<br />\n";
print "Note that some caution is required when applying upgrades to the wordpress install!<br />\n";    
print "<h2>Source code for this script</h2>";
highlight_file(__FILE__);


Security Fails

Security Fails

Worse than merely being Security Theatre, a lot of bolt-on "security" products actually undermine your data confidentiality, integrity and availibiliy.

Recently, while perusing my webstats, I noticed http://cp.mcafee.com/... appearing in the referers. The path part of the URL contained rather a lot of data. On opening the URL in a browser, I found it contained a lot of detail about an email, presumably sent to the user of the browser. This report contained a clickable link to my site (hence it appeared in my referers). This information also included the full email address of the email sender.

The technology in question is named "Click Protect" - but it exposes the details of a third party without their consent.

ClickProtect
The site below is rated as Unverified and is categorised by McAfee as XXXXXX/XXXXXXX.

The email was sent to you by XXXX.XXXXX@hotmail.co.uk.

Click the URL only if you understand the risk and wish to continue.

https://www.XXXXXXXX.com/...


Email:  info.security@sainsburys.co.uk

(Original content redacted with XXXXX)

A quick look around the internet and these URLs appear in a lot of different places - there are a lot of sites which publish their stats in a form searchable by Google.

I attempted to contact both McAfee and Sainsburys.co.uk (the webmail provider) to advise them they were leaking information like this but have received no response from either.

Using performance to manipulate behaviour

A darker side to the performance story seems to be emerging. This is the first in a series of 3 posts (there might be more later) about how web performance is being weaponized.

While I, like many of you, spend a lot of time simply trying to make my sites going faster, it seems that other people at finding ways to exploit performance as a way of manipulating user behaviour. This was particularly evident when I recently visited www.forbes.com to read an article about phone biometrics. Not where I would go to for authoritative information – I was just browsing at the time. As is common, it asked me if I wanted to accept its cookies.

Yes, they want to protect their revenue stream so the big green button with white text is easy to see and read, while the smaller grey button is a lot harder to read – and only professes to providing “more information”. Now due to the specifics of the EU's GDPR act, the site needs my “informed consent” to any cookies it drops – so not surprisingly the “more information” button takes me to a dialogue where I can also specify which cookies I will accept.

If I click on the first, big green button, I get an almost immediate acknowledgement. Accepting all three classes of cookies from the “more information” dialogue seems to take slightly longer, but I didn't measure it too closely. But what is interesting is that if I dial back the cookie setting to only “required cookies” the site tells me it has a lot of work to do in order to dial back “the full power of Forbes.com”.

So I have incurred a huge performance penalty for exercising my rights.

This did provoke a torrent of activity in the browser – over a thousand requests – which included a few 404s and several 302's sending my browser back around the internet. I've not looked at all of them, but the 200 responses all contained “no data”, and none of the sites I saw had appeared when I first loaded the page.

This is appears to be a very elaborate piece of theatre.

It took around 60 seconds to reach the 100% point – while helpfully giving me the option to change my mind at any point.

Another interesting feature of the performance was that the counter slowed down as it progressed! If you've read up on progress bars, you'll know that is exactly the opposite of what you should do if you want to convey an impression of speed.

Finally, changing my browser config to send a “Do Not Track” header had no impact at all on the behaviour. Although at the time of writing, this is still a proposal for HTTP.

Usually I don't wear my tin foil hat when browsing the internet – I'm OK that websites need a way to fund the content they publish but I am very disturbed that sites seem to go to such lengths to try to manipulate their users' behaviour.

Random S**t Happens (or sometimes it doesn't)

On Thursday last week, I migrated a wee enterprise application I wrote a number of years ago (2009?) to a its new home on vmware farm. In itself not a big job, but there were a lot of integration points with other systems. Sadly, it went about as well as I expected. After some pain, normal service was restored. Almost. One of the key pages in the application kept pausing.

As you might expect, I am rather fastidious in ensuring the performance of the applications I write. But this seemed strange. Generating the HTML usually took 30-40 ms (measured at the browser). Not earth shattering, but it does do a lot of work and well within the performance budget. But 1 in every 20 or so requests would take much longer - between 6 and 20 seconds!

Since there were no code changes, the obvious candidate for the cause was the infrastructure which had changed:

• other VMs on the same host competing for resource
• I/O contention (this was now on a SAN with a lot of other devices)
• overzealous network security devices filling the network bandiwdth
• congestion crashes on routers

But I hadn't ruled out a problem in the software stack. Mod_deflate buffers, database contention...

Checking the usual metrics (load, CPU usage, disk IO, APC stats) revealed nothing untoward. So the next setp was to inject some profiling in the code. I would have preferred to use XHProf, but the people who own this system are not keen on third party tools in their production systems.

The profiling soon revealed that the pauses were always occurring in the same region of code. This ruled out any environmental issue.

Looking through the region, there was no disk, network or database I/O. It did write some output (and the HTTP response was chunked) but that was a very long delay for a context switch or a garbage collection cycle. And why didn't occur on every request?

All the code seemed to be doing was reading stuff from PHP's memory and injecting it into the HTML page.

Going through the program in some detail (I did mentioned it was a very long time ago when I wrote it originally?) there was an inversion of control - a dependency injection - where a callback was invoked. Dumping the callback led me to an obscure library routine doing encryption. This created an initialization vector:

mcrypt_create_iv (mcrypt_enc_get_iv_size($this->td), MCRYPT_DEV_RANDOM);

This was the smoking gun.

The problem was that I had told mcrypt to read from /dev/random and /dev/random didn't have any randomness. So it blocked until it got some.

The solutions were obvious:

• keep /dev/random topped up (using rngd)
• use a different (weaker?) entropy source - (MCRYPT_DEV_URANDOM, reading from /dev/urandom is available)

Given that I had already peppered the code with profiling, adding a single character seemed the sensible choice. Whether urandom is weaker is debatable. Indeed, vmware (but not RedHat) recommend this as a solution.

Since the encryption in question was using triple-DES (look, it was a really long time ago, OK?) even a bad random number generator wouldn't have helped make it more secure.

In my defence:

1) mcrypt is now deprecated in current versions of PHP, the current Redhat Release (7.3) ships with a version of PHP pre-dating the deprecation - and it certainly was not deprecated at the time I wrote the code. But it wasn't mcrypt doing anything wrong here.

2) The 3DES encryption was an early CSRF protection mechanism for an application which has very restricted access, and subsequently used a more complex system with SHA1 hashes - but the original code was not removed when the new mechanism was added

3) Frankly, base64 encoding the data here would have been overkill given the level of exposure in this application

This was the first time I had come across this problem. I'm going to be involved in moving a lot of other systems into this network - many of which make more extensive (and critical) use of encryption than this one does. Now I know one more thing to look out for.