After decades of running Linux applications as native packaged installations ( first on bare metal and then in virtual machines ) I’ve taken on the task of migrating the majority of my networked application deployments to containers running in a kubernetes cluster of VMs host on my Fedora Linux workstation.
Why the change ? Frankly, this is a bit overdue. But here are the benefits I hope to realize:
- Simpler re-deployment process across a cluster of devices
- Improved / consolidated monitoring
- Dynamic automated recovery of failed services
- Expanded device management options
While podman, rancher, containerd and docker would have simplified setup on a single workstation significantly, my ultimate goal is to have any compute device in the house seamlessly transition to hosting services without my intervention. Settling on a Fedora CoreOS base node VM under Kubernetes with the CRI-O container runtime seemed appropriate given the broad usage and routine updates supporting the OS and the K8s platform.
Given the kubernetes nodes were initially planned to run on a single high capacity server, I implemented a PXE netboot configuration on a virtual bridge through libvirtd to create a kubernetes control plane and worker node among other targets based on the MAC address pattern binding the ignition configuration through PXE.
This is a learning experience – so I limited the installation to getting a Kubernetes environment running and ready for application installations. The Fedora Project did not quite support seamless installation on CoreOS of Kubernetes. There were a couple issues related to the read only nature of some runtime directories referenced by the kubernetes applications that required finesse to address.
Here are a few tips that might be of assistance:
- This effort is a project – the scripts and butane configurations are maintained as a company ‘admtools’ project in our internal git repository.
- YAML configurations are maintained under a ‘deployment’ folder
- Achieve repeatability –
- Created an environment install bash script invoking virt-install to create and add the nodes to the cluster
- Leverage butane configurations to take advantage of Ignition support in Fedora CoreOS installations.
- The cluster can be initialized running kubeadm init through Ignition startup
- Package installations in Fedora CoreOS are OS extensions installed through rpm-ostree to create an overlay for the read only directories.
- The command line options for kubeadm init are insufficient to address the read only directory issues – a kubeadm configuration file must be used to address altering references to those directories.
- Package installations in Fedora CoreOS are OS extensions installed through rpm-ostree to create an overlay for the read only directories.
- I opted to use Calico as the CNI for performance and security flexibility.
- A simple configuration takes three commands to complete after the control plane is running.
- A bare kubernetes installation can be managed and monitored easily by a knowledgeable admin – I need a few tools:
- Install Dashboard for a web interface presenting cluster status
- Install Prometheus to collect and present performance metrics
- Install helm to support deployment processes with a single YAML config for each application.
- Install nginx to provide reverse proxy support for web applications and keep the pod network isolated.