2025-12-05 Notes

Goals + Recap

Yesterday was filled with client meetings
And K3s was one unmitigated disaster after another. The cluster died by the end of the day.
I got lucky with my late night slog, and got back up not just the cluster, but even Traefik, Sealed Secrets and my Kanboard app!
Now I realise the power of declarative files. I did not have to screw around with the command line much last night, with a couple of exceptions. I got things out of my backups¹ and then copied and pasted them around and adapted and simplified them and everything came back up like magic!
And on the bright side of this morbid situation, I do know more about the innards of cluster administration than I did before.
Today will be an attempt and getting old data from an existing install into an app.
And if that works, spend the rest of the day, spinning up and setting up the production cluster on Hetzner, so that I can then test here and then deploy there. Right now I am keeping the two (home and production) separate and apart in different repos, even though Flux does let me have the option of having multiple environments. I am just being my usual paranoid self.
Note to Self: Remember! There is no tearing down this VM once it comes up. Time for play is over!
Also Note to Self: Monday onwards, strict cutoff at 2p. Work meetings, try to schedule as much as possible, in the evening block between 4p-8p

The plan is to redo the cluster again and do my own instance of
- K3s
- Sealed Secrets
- Certmanager
- ~~Letsencrypt~~ (using pre existing Letsencrypt certs)
- Get Traefik Ingress to work
- Figure out a way to get certs automatically into the cluster
And once that is done, figure out an app to move (Miniflux or Hedgedoc?); 2025-12-03: Kanboard it is!
Begin by moving (lifting and shifting in popular parlance) Kanboard to the cluster
- Cert will probably be needed (Wildcard cert works now, just like it does without the cluster)
- Convert a docker-compose to kubernetes manifests
- Learn how to configure an app with code
- Learn how to store data and back it up
- Figure out secrets, if there are any (for now sealed secrets ok, figure out vault and vault injection later)
- Learn how to tunnel through and reverse proxy
- Make Kubernetes manifests work with flux
- Figure out how to automate deployment of manual manifests
- Figure out how to migrate there if there is any in an old app
- Figue out how to automate updation of images in manual manifests
- Once another app is done, then start live deploys. Prototype locally and once it works, migrate to production immediately
- Convert Kubernetes manifests to Helm Charts (optional, based on energy)

Got the cluster back up in its entirety, pretty much the way I want it.
With Kanboard working! Woohooooo! 🎉
Time for bed.

Starting up slowly … still sleepy
2025-12-05 08:40: Everything is booted up and ready. Time for work!
2025-12-05 08:56: Slightly sluggish. Note to self: Do things slowly and carefully!
2025-12-05 09:14: The all Terraform + Flux driven cluster dream of mine probably needs a little compromise. I will have to get Ansible or some automation tool into the mix, because I will have to use at least a couple of things natively on my VM. Syncthing is one thing that comes to mind. Runing something like that on/via a cluster is really overkill to me. It also seems a bit antithetical in terms of values. Syncthing concerns itself entirely with state and data, while Kubernetes veers in the other direction. OS updates are another thing. I don’t want to keep deploying new os images and moving data around. I am not made of money, to run that kind of setup. Will need to figure such a use case to build up the big portfolio project on AWS though.
2025-12-05 09:30: While the dream is always, oh everything is driven by code, there comes a time when stuff cannot be automated completely. So for most deployments folks do VMs and apps on the cloud, where the computer has been abstracted away. In my case, it hasn’t so I can’t treat my single VM as a fungible, disposable thing. Similarly even in the “cloud”, I’m sure there are worker bees, buzzing around machines in datacenters, babysitting them.
2025-12-05 09:35: Break

Back
2025-12-05 10:54: Moved old data into Kanboard. It just works!
Learned to pause flux while I work on something:
flux suspend kustomization <name> and
flux resume kustomization <name>
Same with helmrelease
2025-12-05 11:04: Trying to install extra controllers into flux
2025-12-05 11:13: image-reflector-controller and image-automation-controller installed
2025-12-05 12:18: Break. Tried everything, but the image won’t update. Need to clear my head
2025-12-05 12:19: Lunch

Back
2025-12-05 13:22: Got it to work. I had pointed the automation manifest to the wrong folder
2025-12-05 13:22: Break

Back. Thought up a rule of thumb. Anything exposed out to the web, ought to run in the cluster. Anything local, or exposed only to me, can stay on the node (would prefer it on the cluster, but if not, that’s fine too)
2025-12-05 14:10: And now I have figured out how to convert a docker compose into auto updating Kubernetes manifests, that relieves me of the need to desperately hunt down Helm packages.
2025-12-05 14:21: Starting with Miniflux locally. If all goes well, then off to the production races we go
2025-12-05 14:31: Just realised, Miniflux runs on Postgres. So production work will have to wait. This is going to take a while
2025-12-05 16:07: Struggling to figure out why my letsencrypt certs aren’t updating within the cluster.
2025-12-05 17:00: Fixed it. Cron has limited access to an environment and the KUBECONFIG variable is not set and so no secret or sealed secret was being generated. All I had were empty yaml files. Told the bash script to set KUBECONFIG=/dev/null since I was only generating manifests, not talking to a cluster. And everything started working.

Feedback on this post?
Mail me at feedback at this domain.

Paranoid backups save your ass. Who knew? I did! I have been bitten more times than I can count. Ergo, the habit! ↩︎