May 27, 2020
The Issue
SSL certificates are trusted because they are signed by a certificate chain with a global root Certificate Authority (CA). Our Sectigo Certificate Manager (formerly COMODO) service has produced certificates signed by “AddTrust External CA Root”. But this root certificate is expiring on Saturday, May 30th, 2020.
We do not expect web services to be affected. Recent browsers find and use a better certificate chain, one that will not expire for years. The issue lies with server to server connections. For example, if you use LDAP on your server, your system is a client of the LDAP server and the LDAP server may present an expiring certificate. Some software will not find/use a non-expiring chain and, if so, the connection will fail.
See Sectigo’s knowledge base:
https://support.sectigo.com/Com_KnowledgeDetailPage?Id=kA03l00000117LT
Synopsis
Our Sectigo Certificate Manager account produces SSL certificates, mostly for our web sites, but also for other SSL protected services, like LDAP. Each certificate has an expiry date (aka notAfter date) and are typically valid for 1 or 2 years. Each certificate is signed by a chain of Certificate Authority (CA) certificates. There are certificates in the chains that will expire this May 30th, 2020. It was expected that services would not be affected. We believed that client software would find a chain that did not expire.
But we now expect this will affect some server to server connections. There may be ramifications for some systems with software that does not build non-expiring chain. We will need to update certificate chains on some of our systems.
This issue should have been avoided. Sectigo should not be issuing certificate chains that expire before your host’s certificate does, particularly when our certificates expire in only one or two years. Certificate Authority certificates have much longer terms and the Sectigo chains should have been updated two years ago.
The Solution
If you have a service that provides SSL connected services to Unix or Unix-like servers, the certificate chain file may need to be replaced.
- If Server A fails to connect to Server B because A no longer trusts B’s chain, it is Server B that must adjust its certificate chain.
- If Server A does not find a root CA for Server B’s certificate in A’s Certificate Authority bundle, then Server A must adjust its CA bundle.
The Certificate Authority Chain
Your certificate is signed by a Certificate Authority (CA) certificate. Frequently, that certificate is signed another CA cert, which is signed by another … until one certificate is signed by itself. This last certificate is a root certificate, which is a self-signed certificate that is expected to be already installed on client systems. The chain may be just the global root, but each of our certificates has several certificates in a chain. For example,
- “www.something.utoronto.ca” signed by
- “Sectigo RSA Organization Validation Secure Server CA” signed by
- “USERTrust RSA Certification Authority” signed by
- “AddTrust External CA Root” signed by itself.
The last certificate is “self-signed” and is a Global Root Certificate Authority (CA) certificate. Each client computer has “bundle” of Root CA certificates that it trusts. And we expect all Global Root CA certificates to be installed with every desktop or notebook system: Microsoft Windows, MacOS systems, Android, or other.
Each of the certificates in the chain will have an expiry (aka notAfter ) date and time. The certificate should not be trusted after the expiry time. And for some currently active chains, two of those certificates expire on May 30th, 2020.
Most clients have software that will find a new certificate chain and one that will not expire soon. The popular browsers on Microsoft Windows, MacOS, and GNU/Linux won’t see an issue. We expect no issues with web servers and common browsers. Even when a web server presents a chain with an expiring certificate, we find that web browsers can build a working chain from certificates that are already installed on the desktop/notebook. But perhaps very old desktops/notebooks might have trouble.
We now believe that some UofT servers will have issues when they act as clients to other SSL servers. This isn’t common as most servers are not clients. But we identify two cases that might present issues: (1) LDAP and (2) web tools like curl, wget, and shibd. The trouble lies in the SSL libraries and, perhaps, a system’s certificate bundle file.
The expiring certificate chain has an intermediate certificate “USERTrust RSA Certification Authority” signed by “AddTrust External CA Root”, a global root certificate. Both the intermediate and root certificate are expiring May 30th, 2020. The fix is to replace the chain with the new chain which has “USERTrust RSA Certification Authority” signed by “AAA Certificate Services”.
The Expiring Certificate Chain
subject="Sectigo RSA Organization Validation Secure Server CA" issuer="USERTrust RSA Certification Authority" notAfter="Dec 31 23:59:59 2030 GMT" Fingerprint="40:CE:F3:04:6C:91:6E:D7:AE:55:7F:60:E7:68:42:82:8B:51:DE:53" subject="USERTrust RSA Certification Authority" issuer="AddTrust External CA Root" notAfter="May 30 10:48:38 2020 GMT" Fingerprint="EA:B0:40:68:9A:0D:80:5B:5D:6F:D6:54:FC:16:8C:FF:00:B7:8B:E3" subject="AddTrust External CA Root" issuer="AddTrust External CA Root" notAfter="May 30 10:48:38 2020 GMT" Fingerprint="02:FA:F3:E2:91:43:54:68:60:78:57:69:4D:F5:E4:5B:68:85:18:68"
The New Certificate Chain
subject="Sectigo RSA Organization Validation Secure Server CA" issuer="USERTrust RSA Certification Authority" notAfter="Dec 31 23:59:59 2030 GMT" Fingerprint="40:CE:F3:04:6C:91:6E:D7:AE:55:7F:60:E7:68:42:82:8B:51:DE:53" subject="USERTrust RSA Certification Authority" issuer="AAA Certificate Services" notAfter="Dec 31 23:59:59 2028 GMT" Fingerprint="D8:9E:3B:D4:3D:5D:90:9B:47:A1:89:77:AA:9D:5C:E3:6C:EE:18:4C" subject="AAA Certificate Services" issuer="AAA Certificate Services" notAfter="Dec 31 23:59:59 2028 GMT" Fingerprint="D1:EB:23:A4:6D:17:D6:8F:D9:25:64:C2:F1:F1:60:17:64:D8:E3:49"
Your Server as an SSL Client
You typically think of your servers as just that: a system providing a service. But some of your systems will also be clients. For example, if you have a database behind your web site, your server is a client of the database server. Your system may be a client of an LDAP or Active Directory server. You may have scripts that use curl or wget to fetch information or trigger background work. These are the clients that may be affected by the expiry of the SSL certificate chain. Your client may cease to trust the service.
Admins at CMU, among others, have run some tests. They have advanced system clocks to see how the expiry will affect software. And they have found issues. While some systems and libraries can/will build an acceptable certificate chain from its system Certificate Authority Bundle, some libraries insist on verifying the full chain presented by the server. That chain may expire on May 30.
SSL Certificates
SSL is used to protect client-server protocols and, mostly, it’s the client that wants to verify the identity of the server. From a client’s perspective, the server presents a certificate and a certificate chain. The client checks:
- Who vouches for the server? and
- Whom does the client trust?
If the client trusts the voucher, then the client trusts the server. Take care to separate the two questions and use the appropriate file to answer each.
Who vouches for the server?
This question is answered by the certificate chain. Sectigo vouches for each of our certificates. When you get a Sectigo-signed certificate, you’ll get a notification in email with a set of links, notably,
- one to download the certificate for the “end entity” (e.g. www.something.utoronto.ca) and
- one to download the certificate chain (“Root/Intermediate(s) only”).
The chain file should be installed on your server as the chain file. For example, use it for Apache’s httpd as SSLCertificateChainFile. Your application should have a setting with a similar name.
Whom does the client trust?
This question is answered by the client system’s Certificate Authority Bundle. Microsoft Windows has a system certificate store accessible via certmgr.msc. MacOS has a datastore accessed by the Keychain Access app. Unix and Unix-like systems have OpenSSL or an OpenSSL-compatible libraries and use as a text (PEM) file that holds the global roots, typically as one or more of:
/etc/pki/tls/certs.pem
The path is often a symbolic link to another file. You may have a JVM that has its own certificate datestore. Apache httpd will use SSLCACertificateFile to set the path for this file. The SSL library would have a default if your httpd config does not set a value explicitly.
Your client will trust a service’s certificate if (a) it finds the chain’s Root CA in the client’s CA datastore and (b) no certificate in the chain has expired.