🤦 Dangers of Cert Pinning

The cert pinning saga continues. Previously on this blog, I wrote about TLS Certs and talked about how you must be careful to mind the expiration dates. From that post:

Keep a close eye on cert expiration dates and issue an update so your devices don’t get stuck in a state where they can’t connect to your cloud endpoints

Sounds easy but then another blind spot hit me and it hit me hard. From a practical standpoint certs expire before they expire. Read on to find out what I mean.

What is cert pinning?

“Cert pinning” in this context does not refer to the now-defunct industry security mechanism HPKP (HTTP Public Key Pinning). Cert pinning, for our purposes, means using a hard-coded or otherwise stored public X.509 certificate, the “pinned” cert, to connect to your services (HTTPS/MQTTS). In your IDF code it may look something like this:

const char* rootCACertificate PROGMEM = \
"-----BEGIN CERTIFICATE-----\n" \
"MIIFFjCCAv6gAwIBAgIRAJErCErPDBinU/bWLiWnX1owDQYJKoZIhvcNAQELBQAw\n" \
".... rest of the cert here" \
"-----END CERTIFICATE-----\n";

WiFiClientSecure client;
client.setCACert(rootCACertificate);

Why do IoT devices pin to certs?

As I mentioned in the previous post we typically don’t need to use the entire cert bundle because devices only connect to a fixed set of services. Why store all of those certs in flash if you don’t need them? Cert pinning can save flash space although I showed how you can completely offset that storage cost. with a few menuconfig settings. In other words, you should definitely use the cert bundle.

One additional reason I personally pinned was because of Arduino, which I hate. As of this writing there isn’t a way to easily use the built-in IDF cert bundle in the ESP32 Arduino core library. As a result, anyone using the Arduino library needed another way to specify a cert. While you could specify a custom bundle, it turns out that didn’t actually work at the time.

What went wrong?

In my case I was pinning to the Let’s Encrypt R3 intermediate certificate which has an expiration date of September 15, 2025. I had loads of time before it expired. Over a year! Certainly plenty of time to pin to a new cert and get firmware updates out. Except for one problem: leaf cert rotation. The R3 cert is the cert used to sign the leaf certificates for the Deploy the Fleet service. A service I host via a cloud service provider. Almost every hosting provider in existence today uses ACME to handle auto rotation of X.509 certificates.

Here is what the X.509 cert chain looked like for Deploy the Fleet.

flowchart RL
  C-->B-->A
  A(Let's Encrypt ISRG X1 Root)
  B(Let's Encrypt R3)
  C(ota.deploythefleet.io)

The leaf certificate is generated and signed by the intermediate CA, in this case R3. That leaf certificate is short-lived with an expiration date only 3 months in the future. As such, it is automatically rotated every 2 months by the hosting provider. My IoT devices were expecting the chain of trust for ota.deploythefleet.io to point to the R3 cert.

The R3 intermediate certificate has a lifespan of 5 years from 2020 to 2025. But to stay ahead of rotation schedules, Let’s Encrypt retired it this year and created new intermediate certificates R10 and R11 which have 3 year lifespans. Upon creating the new certificates they retired the R3 and deemed it no longer suitable for signing leaf certificates. This is totally normal practice but something I was unaware of.

So when my hosting provider ran it’s normal cert rotation automation it used the newly commissioned R10 intermediate cert to generate a new leaf cert for the ota.deploythefleet.io domain. As such, every device pinned to the R3 certificate could no longer connect to the domain and get updates. The new cert chain looked like this.

flowchart RL
  C-->B-->A
  A(Let's Encrypt ISRG X1 Root)
  B(Let's Encrypt R10)
  C(ota.deploythefleet.io)

With the R3 cert pinned in firmware, the Mbed TLS library can’t validate the chain of trust for ota.deploythefleet.io because it’s certificate is now signed by R10.

How to prevent issues

So what can we do? The answer is…it depends. For maximum pinned certificate lifespan you should always use the root certificate for TLS validation. In my case, I should have been pinned to the Let’s Encrypt ISRG Root X1 certificate instead of the R3 intermediate certificate. This would have prevented the problem in this case. However, it would have only delayed it.

Isn’t the answer to not pin at all? It’s not that simple. Even if you use things like the IDF certificate bundle you are still pinning. The only difference is that instead of pinning to one certificate you are pinning to a whole bunch. All of which will eventually expire so you need a plan in place to get updated certs to your devices.

The way browsers handle constantly rotating keys is by constantly updating the built-in certificate bundles. Your firmware should do the same but IoT devices, especially low power ones, may be offline for long periods of time between checkins. As cert lifetimes get shorter and shorter this makes it important to have ways to update certificates.

Production Pointers

• Pin to root certs for maximum time between expiry
• Use a certificate bundle to give you built-in failover options
• If you absolutely must pin to a single cert, you absolutely MUST be the controlling certificate authority for the pinned certificate so you control the expiration
• Have a plan in place to ensure all devices get updated certs before expiry

Summary

This problem is only going to get worse as ACME continues to push for shorter and shorter certificate lifespans of all certs, from the root all the way down to the domain-level. This is an area that you need to pay special attention to to ensure your devices remain online and connected to critical cloud services. And remember, even though a cert has a future expiration, behavior at the service provider level could effectively “expire” it much earlier than you expect.

Join the community and get the weekly Production ESP32 newsletter. Concise, actionable content right in your inbox.