About the Tailscale.com outage on March 7, 2024
On March 7, 2024, tailscale.com was unavailable for approximately 90 minutes due to an expired TLS certificate. We were able to identify and address the issue quickly, and the downtime was mostly limited to our marketing materials and documentation, with a few exceptions we address below. Still, any unexpected downtime is a problem, and we want to take an opportunity to explain exactly what happened, what the impact was, and what steps we’ve taken to ensure it doesn’t happen again.
We rolled out a major website refresh that included a migration to a new hosting provider in December of 2023, just about exactly 90 days before the outage. Keen-eyed readers may recognize that detail as foreshadowing. Our configuration is also a little unusual: because our hosting provider does not natively support IPv6, and because IPv6 is important to us and to our users, we run our own proxy to resolve such requests and list “extra” AAAA records accordingly.
That arrangement is deemed a “misconfiguration” by that provider, and we’ve been receiving alerts about it since rolling it out. We did not realize (and the alerts didn’t specify) that the configuration would prevent automatic certificate renewal from completing. One more bit of bad luck: Although we had probers checking certificate expirations, they were only checking over IPv6. As a result, our probers did not surface the impending certificate expiry because they were hitting the proxy—which had a valid certificate that we were managing independently.
