Some background

The other day, a friend of mine complained that he couldn’t access one of my websites and included a screenshot of the classic “certificate warning” we’ve all grown accustomed to. Suspecting that the server had failed to renew its certificate from Let’s Encrypt, I popped the site’s URL into my own browser without issues and noticed that we were far from the expiry date.

Was the clock out of sync on my friend’s device? Had he forgotten to apply software patches during the last decade? The answer was slightly more exciting: he had accessed the website while being connected to his workplace network and was being subjected to a man-in-the-middle-attack. The perpetrator in this case was not a hoodie-clad individual as seen in countless VPN commercials, but rather a next-generation firewall presumably configured by the network’s legitimate administrators.

Once clicking through the warning (seems like they failed/neglected to roll out the device’s custom certificate authority to prevent us from seeing the error), my friend was presented with a notice that accessing the site was “in violation of internet usage policy (sic)”. Considering the very SFW content of the site, we were both surprised - did they really hate me that much? Clarification was provided once he reached out to the IT department: the site was flagged as “phishing” in the threat feeds provided by a third party.

While this answer made me slightly nervous, I thought it very unlikely that anyone would have compromised a somewhat locked-down box service static content (naive hubris?). Nevertheless, I didn’t want friends/foes to laugh at my incompetence - time for an investigation!

Verifying the claim

From my days managing on-prem mail servers, I knew that a good place to start was to check if the domain was included in the Spamhaus blocklist. Unlike other aggregators of evildoers, they usually provide a helpful explanation why an entry has been added to their database. To my delight, the domain report came back clean!

But who says it was the actual domain that had been flagged - may as well be the IP addresses it points to. I performed the same lookup for the target IPv4 address (seems like something they could’ve done automatically): no issues there either. As a good netizen, I try to expose my systems over IPv6 and this web server was no exception - the domain had an AAAA record configured. Once I entered this address, the culprit was identified!

The page informed me that the addresses had been implicated in malicious activity. As a proof, they listed observed communication been the server’s address and a known botnet command and control node. Scary indeed.

I don’t know why, perhaps it was my spider-sense tingling, but I took the opportunity to look up the IPv6 addresses for some of my other servers hosted by the same provider - many of which showed the very same warning of being in contact with a botnet node.

Had I neglected security patches to the point where these servers (running different operating systems) had all been compromised? Had I in some delirious state posted my private SSH key on GitHub instead of the public one? Even worse: could someone have “ägt upp leverantören i brygga”?

After calming myself with some Incident Response Light, I re-read the proof of malicious activity provided by Spamhaus. Upon closer inspection, it didn’t exactly say that the server’s IPv6 address had been implicated - but rather “almo:stmy:addr:ess::/64”.

Addressing in IPv6

Just like in the old world of IPv4, host and network addresses exist in IPv6 (-ish). To differentiate between host/network addresses, we commonly use CIDR notation - the equivalent of “/32” in IPv4 is “/128” in IPv6.

One of the main benefits of IPv6 is the huge amount of unique address available. They aren’t a particularly scares resource: organizations like RIPE recommend that network providers hand out a whole “/64” to end-customers in an attempt to finally kill NAT and similar abominations. This practice is quite common in server environments as well - give each box its own “/64” (often called “IPv6 prefix”) and you’ll never need to think about “port collisions” on your VMs/containers ever again!*. Even endpoints in corporate networks get a prefix with numerous addresses to play with.

To summarize: in most IPv6 networks, each single host controls a large number of (potential) host addresses.

IPv6 and blocklisting

If we observe that a single IPv6 address is being naughty on the Internet, simply blocking it would likely be an inefficient way to stop abuse - the server controlling it could likely just pick another one at random from the gazillion available in the “/64” and keep on crunching.

For this reason, organizations like Spamhaus don’t include IPv6 host addresses ("/128") in their blocklists - if they detect abuse, they grab the whole “/64” and call it a day. There may be collateral damage (a risk that exist in the IPv4 world as well due to CGNAT and similar things), but the industry best-practices are quite clear as previously noted.

Troubles with OVH

All of my affected servers are hosted by a company called OVH in their public cloud. They’re a European company, the offering is based on OpenStack (great for all that OpenTofu code I’ve got laying around) and the pricing is okay.

When you spawn an instance in their cloud, you can choose between “private” or “public” networking. If you want “easy mode”, select “public” and your server will get a publicly-routable IPv4 and IPv6 address. The “private” mode enables you to do old-school things like configuring a “VPC”.

Note that I didn’t say “publicly-routable IPv4 address and IPv6 prefix”. They assign a single host address to each instance ("/128").

After some experimentation (I have access to another tenant in their public cloud through my company), it seems like each of their availability zones have a pool of IPv6 prefixes that they allocate addresses from.

To summarize: if another customer is flagged for abuse over IPv6, there is a significant risk that your instances will become flagged as well.

Moving forward

My servers may yet be compromised, but I find the explanation above far more likely. I’ve created a support ticket with OVH, asking them to clarify how they share IPv6 prefixes. Mayhaps there is some hidden configuration options available to change this behavior?

Until I’ve figured these things out, it’s probably best to remove the AAAA records from DNS and live with the shame of being without them.