A PKI Apocalypse

At work, we are in the middle of an ongoing incident and it is a slow motion disastrophy.

First, when I brought up last year that we would have an intermediate expiring at the end of this month, it was acknowledged. I had my own pressing concerns, so when I heard "someone's going to take care of it," I let it be. Guess that was a mistake because that is now an all-hands effort.

The next problem: people building Docker images pull from our artifact store, and that includes the certificates, so I'll just update those. We have a weird mix of bundles that include .crt and .pem files. I assumed that because we have a lot of Windows machines, the .crts were DER, and the PEMs were... PEM. That's on me.

Turns out both are PEM, update-ca-certificate can't handle DER, and everything broke. It was hard to debug because I still don't have a working Docker setup due to a combination of pathologically hostile desktop setups to include a proxy that is notorious for screwing up connections, a restricted root access process (you have to click an app, fill in a justification, and then you get 30 minutes to do stuff as admin), a generally byzantine infrastructure and tech stack, and a lackluster level of competence across the board [#].

In the process, I rage-wrote a tool that basically replicates the functionality of all the small tools I wrote in goutils. It was a feverish coding session between about 8 PM and 3 AM; been a while since I've done that. It's on Github so it's less scary than my Gitea instance, and I have goreleaser publishing builds via a Github action so I can just hand people a single binary. I have to write docs, but it's already been pretty useful.

I gotta dust off this resume. Maybe Anduril is hiring.

[1]I constantly haha only serious joke that I keep making two judgement errors: the first, overestimating the general competence of staff here; the second, underestimating how dysfunctional everything is. I'm not sure if it's supposed to be a parody of the Office meets Silicon Valley, but that's as best as I can describe it. I have a whole soap box about the "culture." I swear, though, if I hear "top of mind" one more time...

Tags: ,