Wrangling Kubernetes contexts
If you use Kubernetes on a regular basis, you've probably came across the dreaded context.
And if you haven't, it's basically a shorthand name for
(cluster, user, namespace). Or in other words, it's what tells
kubectl delete persistentvolumeclaim/my-database1 whether it's
supposed to tear down your local test environment or the production
database.
It's also pretty hidden. It's just one line in the middle of your
~/.kube/config:
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: [snip]
server: https://127.0.0.1:33097
name: k3d-k3s-default
- cluster:
certificate-authority-data: [snip]
server: https://k8s.example.com
name: prod
contexts:
- context:
cluster: k3d-k3s-default
user: admin@k3d-k3s-default
name: k3d-k3s-default
- context:
cluster: prod
user: admin@prod
name: prod
# The only thing keeping us between safety
# and a bunch of very angry phone calls
current-context: k3d-k3s-default
kind: Config
preferences: {}
users:
- name: admin@k3d-k3s-default
user: [snip]
- name: admin@prod
user: [snip]
Oh dear.
Surely, other people have run into this before?
Now, naturally, a whole cottage industry has sprung up around keeping it under control. There are tools to change it, and tools to remind yourself about it.2
So should mean that we're sorted, right?
Except, well...
Are you actually paying attention to that note on the side of your terminal?
I.. don't really like this kind of state. It's easy to forget about. It's easy
I don't even particularly like the idea of the "current working directory". But at least that's been there since time immemorial.3 And pretty much every single shell on earth knows about it, and gives it prime real estate.4 And it affects pretty much every command, so there's not much opportunity to forget where you are.
And I like typing
vim /home/n/dev/flickboard/app/src/main/java/se/nullable/flickboard/KeyboardService.kt
over and over even less.
It also turns out that a lot of other stuff actually relates to the directory, even if it's not directly about typing file paths, so as a form of harm reduction there are other tools that can reuse it to manage settings per-context.
But the Kubernetes context isn't like that. After all, we need to be able to deploy (more or less) every project to (more or less) every cluster.
Welp.
I guess we do need special cases sometimes.
So, wat do?
I'd much rather specify for each command that I'd like to run it in
a dangerous environment. Something like
run-in-prod kubectl delete persistentvolumeclaim/my-database, maybe?
So regular kubectl does it against the safe cluster that only affects
me, and run-in-prod kubectl afflicts it on everyone.
kubectl actually already provides a --context=prod
flag5
for this. But it gets pretty unwieldy6, and a lot of other tools
don't have it at all.7
But there is another way. The
$KUBECONFIG
environment variable lets us swap out the whole kubeconfig wholesale,
and it works pretty much everywhere.
So instead of having one config for every environment, I just put the
development environment in there, and keep the dangerous environments
separate next to it (~/.kube/config-prod).
And then I can just define shell aliases that set KUBECONFIG to
whatever environment I want:
alias p='KUBECONFIG=/home/n/.kube/config-prod'
alias meta='KUBECONFIG=/home/n/.kube/config-meta'
alias k='kubectl'
And invoke it like so:
$ # dev
$ k get pod
$ # looks good, let's run it in production
$ p !!
(expands to `p k get pod`)
$ # and it works for just about everything else too, since it looks the same
$ # as just setting KUBECONFIG persistently!
$ p helm ls
$ p ./my-custom-deploy-tool.sh
I've been pretty happy with it for the last five years or so.8 Maybe you'd like it, too? Either way, I'd love to hear about it in the comments below!
-
I have to admit that there is a certain joy in telling people that I wrangle PVC for a living.9 ↩
-
And, as always, we can solve the "too many tools" problem with more tools. ↩
-
Yes, even DOS.
C:\, anyone? ↩ -
Sorry, they don't provide anchors for me to link to. ↩
-
--context=k3d-k3s-defaultis already a bit of a mouthful. ↩ -
Or call it something else. Helm has a similar flag, but calls it
--kube-contextinstead. ↩ -
Though, admittedly a fair bit of that time has been on the software vendor side, not really managing production clusters at all. ↩
-
Until I have to explain to them what I actually meant.. :'( ↩