Picard Family Ramblings

After decades of running Linux applications as native packaged installations ( first on bare metal and then in virtual machines ) I’ve taken on the task of migrating the majority of my networked application deployments to containers running in a kubernetes cluster of VMs host on my Fedora Linux workstation.

Why the change ? Frankly, this is a bit overdue. But here are the benefits I hope to realize:

• Simpler re-deployment process across a cluster of devices
• Improved / consolidated monitoring
• Dynamic automated recovery of failed services
• Expanded device management options

While podman, rancher, containerd and docker would have simplified setup on a single workstation significantly, my ultimate goal is to have any compute device in the house seamlessly transition to hosting services without my intervention. Settling on a Fedora CoreOS base node VM under Kubernetes with the CRI-O container runtime seemed appropriate given the broad usage and routine updates supporting the OS and the K8s platform.

Given the kubernetes nodes were initially planned to run on a single high capacity server, I implemented a PXE netboot configuration on a virtual bridge through libvirtd to create a kubernetes control plane and worker node among other targets based on the MAC address pattern binding the ignition configuration through PXE.

This is a learning experience – so I limited the installation to getting a Kubernetes environment running and ready for application installations. The Fedora Project did not quite support seamless installation on CoreOS of Kubernetes. There were a couple issues related to the read only nature of some runtime directories referenced by the kubernetes applications that required finesse to address.

Here are a few tips that might be of assistance:

• This effort is a project – the scripts and butane configurations are maintained as a company ‘admtools’ project in our internal git repository.
  • YAML configurations are maintained under a ‘deployment’ folder
• Achieve repeatability –
  • Created an environment install bash script invoking virt-install to create and add the nodes to the cluster
  • Leverage butane configurations to take advantage of Ignition support in Fedora CoreOS installations.
• The cluster can be initialized running kubeadm init through Ignition startup
  • Package installations in Fedora CoreOS are OS extensions installed through rpm-ostree to create an overlay for the read only directories.
    
  • The command line options for kubeadm init are insufficient to address the read only directory issues – a kubeadm configuration file must be used to address altering references to those directories.
• I opted to use Calico as the CNI for performance and security flexibility.
  • A simple configuration takes three commands to complete after the control plane is running.
• A bare kubernetes installation can be managed and monitored easily by a knowledgeable admin – I need a few tools:
  • Install Dashboard for a web interface presenting cluster status
  • Install Prometheus to collect and present performance metrics
  • Install helm to support deployment processes with a single YAML config for each application.
• Install nginx to provide reverse proxy support for web applications and keep the pod network isolated.

In order to minimize overhead on enabling R&D while maintaining control of company IP, I wanted to ensure any team member would be able to spin up a R&D environment with zero or marginal delay. We could just spend the money on AWS instances, however I wanted the benefits of maintaining a small on premise cluster of servers to host those environments, such as:

• Lower long term costs with sufficient utilization – start with one server and scale up if necessary
• Low cost archive / restore of past projects
• No-cost access to on-premise servers / services for integration

We wanted to balance cost with simplicity and mostly adhere to common practices for portability. As such, we focused on creating a simple netboot configuration on our internal dhcp server to support network installation on a bare-metal VM referencing a minimal Ubuntu ISO with a cloud-config ISO volume attached to trigger custom VM setup.

Initial Configuration

Some articles discuss classifying network devices and implementing more advanced differentiating configurations on the corporate network dhcp and tftp servers. These features sound great, but we wanted the initial rollout to mostly enable scenarios with zero-configuration effort.

We went opted to start very simply and – given the internal support for dhcp and tftp on network configurations in libvirt – implemented netboot on the libvirt R&D network configuration. We made a minimal edit adding the following two XML config nodes:

• child node under the ip node: <tftp root=’/var/lib/tftp’ />
• child node under the dhcp definition: <bootp file=’pxelinux.0‘ />

To complete configuration – we need to create the tftp root directory, download pxelinux.0 there, and follow the guide provided by Ubuntu to extract the necessary files from the ubuntu live ISO to the tftp directory. The Ubuntu guide is derivative of the pxelinux documentation and can be adjusted with some knowledge of pselinux.Note: Fedora Core provides a convenient package, syslinux-tftpboot, that installs the necessary files in a tftpboot directory on the root filesystem that I found to be a bit more convenient for accessing those files.

We chose to maintain boot configurations for Ubuntu, Fedora, and Fedora CoreOS. We achieved this by maintaining a configuration for each OS in the /var/lib/tftp/pxelinux.cfg directory and linking the configuration to a MAC pattern per the pxelinux documentation. Each configuration passes a cloud-init or ignition configuration to provision admin account access and any other universally available features.

We maintain all referenced images locally on the libvirt host to enable offline support and enhance install performance.

First Boot Experience

The experience on first boot is highly dependent on the new VM configuration, which we have settled on as follows:

• Pass the R&D network as a network (isolated to the VM host)
• (optional) Pass a bridged network as an additional network ( we prefer an externally defined VLAN enabled bridge limited to R&D usage )

This configuration allows immediate access to the VM on our pre-configured R&D network and optional isolated access by VLAN.