storage: mark cluster-sensitive reads for caching backends

[mbardelmeijer]

Why

We run many edge instances that terminate TLS with CertMagic against a shared remote Storage, and want a local disk (or similar) cache in front of it so handshake Loads do not pay remote latency on every in-memory cache miss.

That breaks multi-instance renew coordination today: the renew pre-check (managedCertInStorageNeedsRenewal) and related loads use the same Load path. If they hit a stale local cache, peers queue unnecessary renewals (issue-lock storms) instead of taking the cheap reload-from-storage path—even though shared storage already has the new cert.

Handshake can tolerate a slightly stale-but-valid cert. Cluster decisions (already renewed? already obtained?) cannot.

What

• WithStrongStorageConsistency / StrongStorageConsistency on context
• Set on renew pre-check, reload-from-storage, obtain, and renew
• Handshake / normal managed loads unchanged
• Non-caching backends ignore the flag (no behavior change)
• Caching backends SHOULD read through to durable storage when the flag is set; Store should always write through

Other approaches considered

• Optional LoadConsistent / ExistsConsistent interface (like LockLeaseRenewer): more explicit, also non-breaking; happy to switch if preferred over context.
• Scoped Consistent(fn) mode on Storage: possible, but easy to get wrong under concurrent Loads.
• Changing Storage.Load signatures: breaking for all implementers.
• External invalidation (e.g. pub/sub): works but is more moving parts for the same job.

Happy to rename (e.g. StorageCacheBypass) if that reads better.

[mholt]

This is interesting and I like that it's pretty thin, but it puts a lot of burden on the storage implementation and doesn't generalize well IMO.

I still want to explore the LRU-style eviction we discussed via email. Which I hope to look into more today. But if we are going to do something like this, what about a design like... a LocalCache Storage field in the config that, if configured, is used to help avoid calls to the big, ground-truth Storage that may be remote and expensive. That way it can be any other storage module (likely just FileStorage) and it doesn't need to be custom-built.

Give me a little bit to research LRU "hacks" to gain some efficiency for your case and I'll have more thoughts!

[mbardelmeijer]

LRU could definitely help, although for our use case a local disk cache would be even more valuable for our use-case.

With 100k+ certificates across a fairly large anycast deployment, keeping a large cache in RAM on every edge instance gets expensive quickly. Ideally we'd have a relatively small in-memory cache for the hottest certificates, potentially with LRU eviction, backed by a much larger local disk cache, with the remote Storage remaining the ground truth.

So I do like the LocalCache Storage option you suggested. Being able to use something like FileStorage would be even better.

We can periodically enumerate the certificates in our remote store and pre-warm the local cache, so most handshake-time loads should never need to hit the remote backend in the first place. I think just calling GetCertificate would then warm it up.

I do also like having the option to implement this in our own storage layer like this PR supports, as that gives us a bit more control over the caching behavior. But I understand that a more generalized solution in CertMagic itself would probably be preferable. Or perhaps making the current cache layer pluggable by a user-provided design.

[mholt]

Ok, one other idea we could entertain, is a hook (if one doesn't exist already, I'd have to double-check) for your own code to try to prefetch a cert from a local cache. Before trying to load a cert from Storage (or the LocalCache value first, if set), call the hook. Is that really simpler though? I feel like you'd have to also be managing renewals and such, i.e. basically become a CertificateManager, which we already have implemented.

It sounds like what you're describing is a 3-tier storage system:

• In-memory cache for hottest certs (LRU-esque policy)
• Local disk storage or some pluggable cache logic for semi-hot certs -- not sure if this would make sense to be pluggable though, LocalCache Storage might have to do.
• Expensive, large storage backend as the ground truth for all certs

[mbardelmeijer]

Yes, 3-tier is exactly what we're looking for! Small in-memory cache for the hottest certs, local disk for the semi-hot tail, remote storage as ground truth for the 100k+ rest.

Sketched LocalCache Storage here as a reference: master...mbardelmeijer:certmagic:local-cache-storage-tier (~200 lines outside tests, additive, nil by default). Serving reads come from disk; the renewal pre-check, obtain/renew under the lock and reload-after-a-peer-renewed read Storage directly and refresh the local copy with what they read. Locking never goes local. OCSP staples ride the same path, and there's a WarmLocalCache hook so we can warm disk without filling the in-memory cache.

Happy to open it as a PR if you want to take it further.

[mholt]

Sure, would love to see a PR for that since you've already got a draft working I guess!

[mbardelmeijer]

Done! See #402