Quick reminder from your friendly local SRE: never ever issue certificates that expire on weekends. Make certs expire in the middle of the afternoon on a business day wherever your operators live and work. The cert in question expires at May 30 10:48:38 2020 GMT, which smells suspiciously like a fixed time after the cert was generated, rather than at a well-chosen point in time.
All my applications use a component that watches certs configured (everything in cert and trust store) and returns warning in telemetry from the application if any of the certificates is less than a week from expiration. This is checked periodically while the application runs.
This not only makes sure we don't miss expiration but also ensures we don't forget to configure any of the application.
We had a situation when the cert was replaced but the file was placed in incorrect path and was not actually used by the app. Having the app report on what is actually being in use is the best way to prevent this from ever happening.
Me too! Especially with the short expiration times of LetsEncrypt. But I really don't want to put `nginx -s reload` in the Cron, in case I'm tinkering with the configs and they're suddenly live (which only really happens at staging or at home of course, but still).
That's what I usually do. My problem is that I might be adding a location and nginx reloads between that and adding access restrictions (i.e. because I took a break to google).
After one scrambling emergency with a cert expiring in the middle of the day, a constant check with warnings and alerts a couple of weeks before expiry made a matter of defensive organization into something trivial.
Engineering for failure is important, you should always set yourself up so that you have several lines of defence which can fail. Some lines of defence to make failing "impossible" others to make a fail softer, even when you think failing is impossible.
Great tip. Did you notice that cert in this case was issued 20 years ago? It’s crazy to me that it was still being used to sign certs as recently as last week (according to twitter)
Of course, but that doesn't really excuse them. My first experience with middle-of-Sunday-night SSL certificate expiration was in December 1998, and it was already a well-known doctrine by then. I'd expect a commercial certificate authority to have these kinds of things squared away.
It's actually worse. The new root (good I believe until 2038) uses the same key as the now expired certificate. It has to or it would not be possible to validate the certificates that were issued. And this new one is a root certificate installed in browsers!
What "should" happen is that no certificate should be issued with an expiration date later than the issuing certificate. Then as the issuing certificate gets closer to expiration, a new one, with a new key pair, should be created and this new certificate should sign subordinate certificates.
Sorry to reply to my own comment. But I want to clarify. Two certificates (at least) expired. The root named "AddTrust External CA Root" and a subordinate certificate with a subject of "USERTrust RSA Certification Authority." Both expired around the same time.
The "USERTrust RSA Certification Authority" certificate signed yet another layer of intermediate certificates.
The "USERTrust RSA Certification Authority" certificate was promoted to a self-signed certificate, now in the browser trust stores, using the same key pair as the original certificate that was signed by "AddTrust External CA Root." It has an expiration of 2038 (although that concept is a bit vague in a root certificate).
There's actually a third certificate for "USERTrust RSA Certification Authority", also using the same key pair, signed by a different root called "AAA Certificate Services". It looks like the intended replacement for the expiring one is this one, rather than the one where it's the root itself.
It is explicitly not a replacement, but some kind of legacy fallback that they don't want you to use, but exists for enterprise customers that absolutely can't get trust.
That's what my browser shows me too, but it's just because it's ignoring the cross-signed one that chains to AAA. The server is sending it, per InCommon's setup instructions.
The old TLS (versions 1.0, 1.1, 1.2) specifications said that the certificates supplied are to form a chain, starting from a leaf and leading back towards a root.
Pretty much all clients assume that once they can see a way to a root they trust they'll give up following the provided chain and trust that - but sadly not all of them, so "over-specifying" the chain can cause problems.
Modern clients tend to go further, they still assume the first certificate is a leaf, but all other certificates are just potential hints that might be helpful in working out an acceptable trust path. TLS 1.3 actually specifies that clients must tolerate certificates supplied on this basis rather than a strict "chain".
I'm actually surprised at the number of claimed clients which don't have vaguely modern trust stores but do understand SHA256.
> I'm actually surprised at the number of claimed clients which don't have vaguely modern trust stores but do understand SHA256.
All the clients were limited to SHA-1 have already been forced off https; CAs in the CA/Browser forum weren't permitted to issue SHA-1 certs valid past Jan 1 2017, and you had to have gotten those issued before Jan 1 2016. Browsers were showing warnings on SHA-1 certs depending on expiration throughout 2015, so you had to either put up with a warning (and the customer service burden thereof), ditch your old clients and go sha-2 only, segregate traffic, or build custom software to send sha-1 certs to some people and sha-2 certs to others.
Microsoft added support for sha-2 certs in the OS system stack with XP Service Pack 3, released in 2008, and Microsoft was always pretty slow with support on things, other platforms may have supported this earlier. A CA bundle from like 2005-2008 is going to be fairly limiting today. A lot of CAs back then had a 20 year validity period, which may have started 5-10 years before the bundle date. Of course, a lot of bundles today end in 2038, so we'll be screwed then.
Certificates issued with this CA will have been cross-signed by the newer root certificate, but our CA (Sectigo) was sending the old chain in issuing emails as late as April this year, despite the cross-signed root being available for a long time.
For long-lived certificates they will outlive your calendar tech. The bit rot leave maybe the events fine, but anything fancier like notifications will fad away.
Source: BTDT 3 times in 7 years, and it was all with "Big Enterprise" grade products.
If you were to issue certs for short durations, and also maintain a calendar of cert expiry, those certs could be renewed in a timely manner.
In other scenarios where one would want to issue fresh certificates (receiving Ops control from other orgs, credentials refresh for what ever reason), one can still do so without waiting for the current certificates to expire.
I am just of the opinion that "make it expire in the afternoon" doesn't apply to root certificates that are used across the entire world (i.e. - the topic of discussion)
