KAIROS: The immutable Linux meta-distribution for edge Kubernetes

As the popularity of Edge as a strategic business infrastructure increased, so did the number of K8s production users that use Kubernetes at the edge.

Kubernetes is now the de facto edge infrastructure management layer, with 35% of Kubernetes users employing K8s at the edge today according to reports (2022 State of Production Kubernetes report).


Although Kubernetes works great at the edge, it still comes with challenges in the areas of compliance, security, performance and scaling when operating and scaling the number of edge locations.


Some of these challenges can be addressed with the implementation of an immutable OS in which the system boots up in a read-only mode and where only certain paths of the system are writable after installation.


This significantly reduces the possibility of accidental or malicious modifications endangering your system improving security, stability and performance by guaranteeing the integrity of the core system.


Here is where Kairos comes in to address the need for an immutable, distributed and scalable system with the ability to manage containerized bare metal Kubernetes at the edge.


What is Kairos?

Kairos is an open source and cloud-native project that can be built, managed, and run on Kubernetes. It is an agnostic, immutable Linux meta-distribution for edge Kubernetes.

Kairos serves a very important function by bridging together Edge, Cloud, and bare metal lifecycle management into the same design principles with a unified Cloud Native API.

Let’s break this down:

As a meta-distribution without a base system, Kairos is highly configurable and scalable. Since it sits as a container layer, it can be overlaid on top of other Linux distributions.

Essentially this means any Linux distribution is turned into an immutable system that is distributed via container registries.

Kairos is completely agnostic, here the OS is the container image which is used for new installations and upgrades. Kairos allows you to build custom bootable-OS images for your edge devices and you can do so using any chosen OS or edge Kubernetes distribution. 

Its immutability eliminates issues like configuration drift and snowflake servers since  each node boots from the same image. It also improves security by not allowing additional packages to be installed after the first installation and discarding configuration changes after a reboot. This makes it a lot harder for malicious actors to gain access to the device and at the same time prevents accidental modifications to the system.

Another advantage of an immutable system is that the configuration is driven via cloud-init and has only one source of truth that happens at bootstrap time. There is no need for a configuration management software because everything else is handled natively via Kubernetes.


Kairos stack components listed below are extensible and modular, meaning it can be customized and substituted in the stack and built off either locally or with Kubernetes.

  1. A core OS image release
    in ISO, qcow2, or other format. (currently can pick from openSUSE and Alpine based).
  2. A release with K3s embedded
  3. A set of Kubernetes Native API components (as CRDs)
    • to install into the control-plane node, to manage deployment, artifacts creation, and lifecycle.
    • to install into the target nodes to manage and control the node after deployment.
  4. An agent installed into the nodes
    to be compliant with Kubernetes Native API components mentioned above.

These are Kairos default components:

  • K3s as a Kubernetes distribution.
  • edgevpn (optional) as fabric for the distributed network, node coordination and bootstrap. Provides also embedded DNS capabilities for the cluster. Internally uses libp2p for the P2P mesh capabilities.
  • elemental-toolkit as a fundament to build the Linux derivative.
  • nohang – a low memory handler for Linux.

How it works

In the following image, you can see the four partitions that Kairos creates after its installation:

  1. State partition: 
    • It stores the container images that are going to be booted (active and passive, stored in .img format which are loopback mounted)

  2. Recovery partition: 
    • Stores the container images used for recovery (in .squashfs format). 
    • The Recovery system allows to perform emergency tasks in case of failure of the active and passive images.
    • A fallback mechanism will take place in case of failures such as booting the partitions in this sequence: “A (Active) -> B (Passive) -> Recovery”.
    • The upgrade happens in a transition image and takes place only after all the necessary steps are completed.
    • An upgrade of the active/passive partitions can be done with Kubernetes or manually.
    • An upgrade will create a new pristine image that will be selected as active for the next reboot, where the old one will be flagged as passive.

  3. OEM partition (optional):
    • It stores user configuration and cloud-config files.

  4. Persistent partition
    • Persists the data across reboots.
    • This partition is mounted on boot to /usr/local and additional paths are bound to it.
    • Those configuration aspects are defined in a cloud-config file.
    • Configuration can be overridden when provided in the cloud-config during installation.

With Kairos you can:

  • Configure it to your needs while staying immutable.
  • Configure nodes via a single cloud-init config file.
  • Create multiple-node Kubernetes clusters with K3s (all K3s features are supported). Check out our K3s post to learn more!
  • Bring the convenience of public cloud on-premise.
  • Onboard nodes via QR code, manually, remotely via SSH, interactively, or completely automated with Kubernetes, with zero touch provisioning.
  • Update nodes by pushing a new image to your container registry and let secure, risk-free A/B atomic upgrades do the rest.
  • Handle air gapped upgrades with in-cluster container registries.
  • Extend the image in the runtime or build via Kubernetes native APIs.
  • Optionally connect nodes autonomously via a full-mesh P2P hybrid VPN network and stretch a cluster up to long distance.
  • Create private virtual network segments to enhance your cluster perimeter without any single point of failure (SPOF).

Here is what the P2P Network looks like

You can use Kairos for

  • Building your cloud on-premise, with no vendor lock-in, based on open source.
  • Easily build immutable, bootable Kubernetes and OS images for your edge devices.
  • Node provisioning—bring your own image or use the Kairos releases.
  • Appliances that doesn’t have to be Kubernetes application specific, its design fits multiple use case scenarios

What makes Kairos different from other meta-distributions?

The main difference is that Kairos is distro-agnostic by design. Its OS is distributed as a standard container, you can also “docker run” locally and inspect the OS, and similarly push customizations by pointing nodes to a new image.

Kairos setup is simple and its p2p capabilities allows nodes to self-coordinate, which makes setting up a multi-nodes cluster very straightforward.


Kairos is an exciting and rather young community-driven project that allows you to bootstrap clusters without mesh capabilities, and with different providers besides K3s.

It lets you onboard other Linux distributions and convert to an immutable system which is fully managed via Kubernetes. You can use it to build immutable, bootable Kubernetes and OS images for your edge devices.

Kairos features a distribution-agnostic design and its OS is distributed as a standard OCI container. Its p2p capabilities make it possible to set up distributed, multi-nodes clusters.

Overall, it is a very interesting but young and not yet production-ready project aiming to close the narrow gap for Edge computing with Kubernetes. We’ll be looking closely at its development and keep you posted.

Looking to run Kubernetes at the edge? Get in touch and we’ll architect the best solution for your use-case.