One of the strengths of Kubernetes is its API-driven design, which allows for for the cluster to be managed using API calls programmatically. As outlined in the API Documentation, even Kubernetes itself is composed of many smaller components interacting with each other through API calls.
The main library that developers use to interact with their cluster, kubectl, uses these APIs to manage the cluster. If you’ve used this tool before then you’ll know how powerful it is — anything can be created, updated destroyed, and fetched. This library uses API calls from outside the cluster to make these changes.
However, just like Kubernetes makes API calls from inside the cluster to run itself, programs can be built to run within the cluster and make use of this API as well. A few more advanced examples of this is Helm and cert-manager. Helm combines templates and values specified at run-time to generate various resources, as specified in their architecture documentation. Cert-manager watches ingress resources and secrets, and writes new secrets to store the TLS certificates. These programs are called “Operators” in Kubernetes.
Writing Your Own Operator
This capability provided by the API and shown in Helm and cert-manager is really powerful! It’s also pretty straightforward to do, but documentation and examples are a little lacking. Let’s take a quick look at how to write an operator interacting with the cluster resources.
Since Golang is generally the preferred language in the Kubernetes community, we’ll be using Golang and some Golang packages. There are just a few resources that we will be using:
- We’ll use https://github.com/kubernetes/client-go which is the official library for interacting with the Kubernetes API.
- We’ll also use https://github.com/kubernetes/apimachinery for Kubernetes resource definitions.
Versioning
One thing to note is that the libraries we are using have two sets of compatibility to be concerned with. First is compatibility with your code and program. Second is compatibility with the Kubernetes API. Since Kubernetes moves very fast and development is very active, versioning compatibility is important to be aware of. There is more details on this in the client-go README.
Simple Example
Let’s start with a simple example of listing the Ingress in the cluster (Please excuse my amateur Golang! I am visiting from Ruby land):
Here we are doing the following:
- Initialize a new Kubernetes client.
- Using that client, List Ingress in all namespaces using an empty ListOption, which will find all Ingress resources.
- Iterate through each Ingress and each Ingress’s Rules and print out the Host for that Rule.
Obviously this is a very basic example, but shows how easy it is to interact with the cluster! For a more code example and Docker build/release scripts, check out https://github.com/sgringwe/ingress-validator.
Deploying The Operator
To deploy the Operator above, we can use the CronJob resource type in Kubernetes. This will run the program daily.
RBAC
If you are using RBAC in your cluster (you should be!) then you will notice that the running the above CronJob will give permission errors. We can use the below resources to give the permissions it needs.
Summary
That’s it! Operators are a powerful concept in Kubernetes and can be used to customize the system for your own Company’s needs. What are some other examples of functionality Operators can achieve? Perhaps your traffic is highly predictable and you can auto-scale your deployments based on time of day. This would be fairly straightforward using CronJob and Deployment patching.
One use case that we’ll be exploring at Handshake is a “Maintenance Mode” functionality that works alongside our Seira library. This could be implemented by watching ConfigMap for configuration updates and updating Ingress to point to the Maintenance Mode Service based on the configuration.
Interested in working on Infrastructure and Backend systems like this? Message me!