Gardener: Taking Kubernetes as a Service to the next level

If you are a regular reader of our blog, by now you are familiar with Kubernetes, its container orchestration capabilities and how it has positioned itself as the de facto standard for orchestrating containerized workloads. Kubernetes has definitely become an essential tool for modern enterprises over the years as these have started containerizing their applications more and more.

However, as the digital landscape evolves, so does the complexity of managing cloud-native infrastructures. One big challenge is managing multiple clusters across diverse environments, which creates a need for sophisticated management tools.

Here is when a tool like Gardener, which addresses the complexities of multi-cluster and multi-cloud Kubernetes management, comes into play. However, the level of difficulty increases exponentially with the number of clusters that have to be operated, monitored and managed.

In this article, we’ll dive into the significance of Gardener, its workings, applications, and why it’s poised to be an essential tool in the Kubernetes ecosystem.

What is Gardener and what makes it different?

Gardener is an open-source project, primarily maintained by SAP, designed to manage Kubernetes clusters as a service. In the sea of Kubernetes tools, Gardener stands out by offering a consistent interface for managing clusters across some of the biggest cloud providers, but also for on-premises infrastructures. Whether you’re on AWS, Azure, GCP, Alibaba Cloud, Openstack or even VMware vSphere, Gardener ensures you can spin up and manage tens and hundreds of Kubernetes clusters with ease.

Gardener is 100% Kubernetes-native meaning the foundation of Gardener is Kubernetes itself and its main principle is to leverage Kubernetes concepts for all of its tasks. Gardener exposes its own Cluster API to create homogeneous clusters on all supported infrastructures.

Gardener is capable of giving you homogeneous clusters with exactly the same bill of material, configuration and behavior on all supported infrastructures. This is possible because Gardener’s Cluster API harmonizes the process of preparing the clusters themselves instead of only harmonizing “how to get to clusters”.

How does it work?

As you probably know, Kubernetes is the go-to, industry standard solution to manage applications and microservices in the containers today. Even when it comes to the components of Kubernetes itself such as coredns, scheduler, cloud-controller-manager or CNI they can be run and managed as containers inside K8s. Gardener leverages this Kubernetes self-hosting ability to host Kubernetes within Kubernetes.

Running Kubernetes Components inside Kubernetes is called self-hosting. We can say that Gardener applies that principle but with a twist. It has a special pattern that caters to the enterprise needs of provisioning and managing hundreds or even thousands of K8s clusters.

This is done with a cluster-within-a-cluster model that leverages Kubernetes’ inherent scalability and self-healing features.

Gardener distinguishes three types of clusters:

  • Garden Cluster: 
      • An existing Kubernetes cluster
  • Seed Cluster: 
      • A Kubernetes cluster where the control plane (e.g. API server, scheduler, controllers, etcd datastore, etc.) of other Kubernetes clusters (Shoot clusters) are running as deployments and pods.
      • There is one Seed Cluster per IaaS and region.
      • It hosts control planes of many shoot clusters.
  • Shoot Cluster:
    • An end-user Kubernetes cluster managed by Gardener.
    • Consists only of the worker nodes (no control plane at all).
    • Is described via declarative cluster specifications which are observed by the Kubernetes controllers.
    • Have the task of bringing up the clusters, reconciling their state, performing automated updates and making sure they are always up and running.

Gardener introduces new API objects in an existing garden cluster and uses them to manage shoot clusters. It can be seen as an extension API server that comes along with a bunch of custom resources and controllers.

In order for the shoot clusters to accomplish the assigned tasks, Gardener controls the main components of a Kubernetes cluster (etcd, api-server, controller manager, scheduler). The control plane components are hosted in the seed clusters.

The fact that one seed cluster hosts the control planes of multiple shoot clusters, allows you to avoid having dedicated hardware/virtual machines for the shoot cluster control planes. This is one of the main differences of Gardener compared to many other OSS cluster provisioning tools. The control plane is put into pods/containers that are being watched by the “seed” cluster, which means they can be deployed with a replica count of 1 and only need to be scaled out when the control plane gets under pressure, but not necessarily for HA reasons.

This approach reduces the total cost of ownership, simplifies deployments (everything is a standard Kubernetes deployment), updates (standard Kubernetes rolling update feature) and management of hundreds of clusters.

Gardener reuses the identical Kubernetes design to span a scalable multi-cloud and multi-cluster landscape. This helps to accelerate the learning process and consequently the productivity, since the concepts are familiar.


In the image below, you can see Gardener’s Architecture with its components:

Gardener Components

If you take a close look at the previous image, you will notice that the naming of Gardener’s components and Architecture are inspired by Kubernetes. Here is what we mean by that:

Kubernetes API Server

= Gardener API Server

Kubernetes Controller Manager 

= Gardener Controller Manager

Kubernetes Scheduler

= Gardener Scheduler

Kubelet 

= Gardenlet

Node

= Seed cluster

Pod 

= Shoot cluster

However, Gardener has many components that are specific to Gardener and that allow it to offer a seamless experience while providing Kubernetes as a Service.

Let’s have a closer look:

Gardener Dashboard:

  • Is the central operator UI, somewhat similar to K8s dashboard, but for clusters
  • Allows users to create and manage clusters, monitor cluster health, and access cluster-related information.
  • It talks to a dedicated Garden cluster, and uses custom resources managed by an aggregated API server to represent Shoot clusters.
  • In this “Garden” cluster runs the “Gardener”, which is basically a Kubernetes controller that watches the custom resources and acts upon them, i.e. creates, updates/modifies, or deletes “shoot” clusters. 


Gardener API server:

  • Is a Kubernetes-native extension based on its aggregation layer. 
  • It is exposed via an API Service object.
  • Designed to run only in the Kubernetes cluster whose API it extends.
  • With this, the Kubernetes will become aware of the Custom Resource Definitions – CRD objects (like CloudProfile, Project, Seed, Shoot, etc.) being created.


Gardener Controller Manager (GCM):

  • The GCM runs next to the Gardener API server.
  • It runs several control loops that help in reconciling the Project resources and taking necessary actions in case of stale projects.
  • It runs several controllers that do not require talking to any Seed or Shoot cluster.
  • It exposes an HTTP server that is serving several health check endpoints and metrics.


Gardener Scheduler:

  • It is a controller that watches newly created shoots and assigns Seeds to Shoots (similar to how kube-scheduler assigns Pods to Nodes)
  • The scheduling strategy is mentioned in the record of intent of Shoot
  • It can be based on Same Region Strategy or Minimal Distance Strategy
  • Either the scheduling strategy or the shoot cluster purpose hereby determines how the scheduler is operating. 
Gardener Architecture

How to create a Cluster?

The standard procedure for providing a Kubernetes Cluster for use entails the following steps:

  • Select a Seed Cluster:
    • Before creating a Shoot cluster, decide on a Seed cluster where the Shoot’s control plane will run. 
    • Gardener often automates this choice, but in some configurations, users may want to specify it.
  • Secrets:
    • Produce the necessary secrets and credentials, enabling the worker nodes to communicate seamlessly with the control plane.
  • Core infrastructure:
    • Lay out the core infrastructure, predominantly focusing on network configurations, utilizing Terraform.
  • Control plane:
    • Initiate the control plane for the Shoot cluster within its specific namespace in the Seed cluster, which includes the MCM (machine-controller-manager) pod.
  • Machine CRDs:
    • In the Seed cluster, set up machine CRDs that detail the configuration and quantity of worker machines for the Shoot. 
    • The machine-controller-manager will monitor these CRDs and spawn virtual machines based on them.
  • Have some patience:
    • Wait for the Shoot cluster’s API server to attain a responsive state, during which Kubernetes will schedule pods, and orchestrate the creation of persistent volumes and load balancers through the associated Cloud provider.
    • The time it takes depends on the Cloud provider and normally
  • Finally!
    • The setup is concluded by launching the core kube-system services, such as kube-proxy, into the Shoot cluster.
    • If desired, add further add-ons like the dashboard. 
    • Once done, the cluster becomes fully operational.🥳

Who is using Gardener?

Given its open-source nature and a liberal Apache 2.0 license, various organizations and individuals have adopted Gardener for managing their Kubernetes clusters. The number of companies interested in Gardener is constantly growing as the complexity of managing cloud-native infrastructures grows together with ever-rising demand.

Companies use Gardener for its multi-cloud capabilities, efficient cluster management and its  extensibility that allows them to plug in different infrastructure, networks and DNS providers.

Gardener allows these companies to run products and services they offer on top of managed Kubernetes clusters directly to their clients, or even only components that are re-usable from Gardener. Let’s take a look at some of the companies that have adopted Gardener:

SAP
SAP is the main contributor behind Gardener, they use it to deploy and manage Kubernetes clusters at a large scale in a uniform way across multiple infrastructures. Workloads include databases, BigData, Kyma and other cloud native applications along with diverse enterprise workloads.

T-Systems
Uses Gardener not only to manage clusters for their customers but also internally. The flexibility and the architecture of Gardener allows T-Systems to manage clusters around the world with ease as well as to meet industry specific certification standards. They can support and manage multiple application landscapes on-premises and across different hyperscaler infrastructures.

OVHcloud
Gardener can now be run by the customers on the Public Cloud Platform of the leading european Cloud provider OVHcloud.

STACKIT
StackIT is a digital brand of Europes’ biggest retailer, the Schwarz Group, which includes Lidl, Kaufland, but also production and recycling companies. It uses Gardener to offer public and private Kubernetes as a service in its own data centers in Europe and aims to become the cloud provider for German and european small and mid-sized companies.

Conclusion

In the rapidly evolving Kubernetes and Cloud-native landscape, tools that simplify and enhance cluster management play a crucial role. Especially when it comes to enterprise scale with hundreds of clusters that need regular updates and support of multiple Cloud providers. Gardener, with its innovative architecture and wide range of applications, emerges as a significant player in this domain. As organizations continue to scale their Kubernetes deployments and seek efficient, consistent, and flexible management solutions, it’s clear that Gardener will play a pivotal role in shaping the future of Kubernetes management.

One important thing to understand about Gardener is that it does not reinvent the wheel but leverages existing Kubernetes principles. By using Kubernetes’ own constructs and principles to manage other Kubernetes clusters, Gardener epitomizes the “Kubernetes all the way” philosophy.

Today, there are only a few open source alternatives that are able to create and manage multiple Kubernetes clusters, one of those is vCluster, a tool for creating virtual K8s clusters using K8s that you can read about in our blog.

Is Gardener the right technology to offer Kubernetes as a Service in my organization?
Get in touch and we’ll help you to figure it out!