The TLS certificates for *.ideal-postcodes.co.uk* and *.postcodes.io* from13:12 UTC Saturday 30 May 2020.
What was the issue
At 12:27 UTC a user informed us that they were unable to complete the TLS handshake to *.ideal-postcodes.co.uk. Our third party HTTP and TLS certificate monitoring services reporting no issue.
Did our TLS certificate expire?
No. The certificate for ideal-postcodes.co.uk expires in 2022. One intermediate certificate, on one certification path, did expire on May 30th 2020 11:48 UTC. But that does not invalidate a certificate.
How can an intermediate certificate expire before the domain certificate?
Our TLS certificate contained 3 certification paths. Even if one expires, the other two remain trusted. A TLS handshake needs only one trusted certification path to be successful.
Certificate Authorities may include paths with old intermediate certificates to support very old clients.
How does Ideal Postcodes detect TLS issues
We have 3 types of monitoring to check for TLS certificate issues.
1. Certificate Monitoring. We contract a 3rd party TLS certificate checker to monitor certificates across ideal-postcodes.co.uk and postcodes.io.
Our certificate monitor raised no issue with possible expiration, revocation or misconfiguration on or leading up to May 30th.
2. HTTP Monitoring. We contract three different 3rd party HTTP monitors, configuring dozens of probes to constantly test our API endpoints. The inability to connect over HTTPs pages our support team.
None had reported downtime as a result of not being able to initiate a HTTPs session.
3. Log Monitoring. Our edge webservers log and may subsequently page our support team in the presence of unusual TLS (and non-TLS) webserver errors.
Who was affected
The issue was primarily confined to a group of users using old versions of OpenSSL (<1.1.1, pre September 2018). Old versions of OpenSSL contain a bug that prevents the client from using valid certification paths if one alternate path is no longer viable. When one path expired on May 30th, these versions of OpenSSL did not try the other valid paths causing TLS negotiation to fail.
We are doubtful this has any affect on queries coming directly from browsers. We tested across Operating Systems (Windows 7, Windows 10, Mac OS Catalina, Linux/Ubuntu 18.04) and browsers (latest Chrome, Firefox, down to Internet Explorer 9). We could not replicate this issue on any browser configuration.
How long did it take to rotate the certificates?
We received the issue at 12:27 UTC and rotated the certificates for ideal-postcodes.co.uk at 13:12 UTC. postcodes.io followed at 13:45 UTC.
The intermediate certificate in question expired 10:48 UTC. So it is possible for affected users to observe connection issues between 10:48 UTC and 13:12 UTC.
We were not able to observe connection problems at 10:48 UTC because our internal and third-party probes were able to process the certificate.
Conclusion
Without being able to reproduce the issue across operating systems and browsers we decided to rotate the certificates as the fastest course of action to help affected users.
This resolved the issue for affected users without necessitating an upgrade of system TLS libraries or updating system trust stores.
While our old TLS certificate was valid, we will be updating our security policy because some downstream users may not be able to easily move off old versions of OpenSSL.
Consequently, we plan to reject all certificates that contain intermediates that will expire before the domain.
Furthermore, we will also configure warnings if the above criteria are not met by our current TLS configuration.
Finally, if the above condition cannot be met, we will remove intermediate certificates before they expire in future.
Our active TLS certificate expires in 2020 with no intermediate certificate expiry.
Additional Updates: 31/5/2020
Indications that GnuTLS is also affected by expired intermediate certificates
Indications that only exceptionally old browsers without out of date trust stores affected