I recently came across the Kubebuilder tool which provides scaffolding to write your own CustomResourceDefinition (CRD) and Controller to extend the API running in Kubernetes clusters. I worked through the tutorial building a CronJob controller, but wanted to write something from scratch, and got some inspiration for a simple (contrived) example to write a “WebPage” CRD that supports CRUD operations for a static web page:
apiVersion: sandbox.rvmiller.com/v1beta1
kind: WebPage
metadata:
name: sample-web-page
spec:
html: |
<html>
<head>
<title>WebPage CRD</title>
</head>
<body>
<h2>This page served from a Kubernetes CRD!</h2>
</body>
</html>The idea is that our custom controller will create a Deployment with a Pod running an nginx webserver image that uses a mounted ConfigMap volume to display the web page contents. We can then kubectl port-forward to the pod(s) to verify the web page is displayed.
You can find the code for this tutorial here.
Step 1: Set up the environment
Download and install Kubebuilder, Kustomize, and Kind to run your Kubernetes cluster locally. I prefer Kind over Minikube since it starts up faster, but you’re welcome to use any tool to deploy your cluster.
Step 2: Set up the scaffolding
Create a directory to store your Kubebuilder files:
$ mkdir webpage-crd
$ kubebuilder init --domain <any base domain, I used rvmiller.com>
$ kubebuilder create api --group sandbox --version v1beta1 --kind WebPageThis will generate a CRD to create resources with the following manifest:
apiVersion: sandbox.rvmiller.com/v1beta1
kind: WebPageStep 3: Define the Custom Resource fields
We need to update the generated webpage_types.go file with our custom Spec and Status fields. We can also use kubebuilder’s annotations to add OpenAPI validation:
// WebPageSpec defines the desired state of WebPage
type WebPageSpec struct {
// Html field stores the static web page contents
// +kubebuilder:validation:MinLength=1
Html string `json:"html"`
}
// WebPageStatus defines the observed state of WebPage
type WebPageStatus struct {
// Stores the last time the job was successfully scheduled.
// +optional
LastUpdateTime *metav1.Time `json:"lastUpdateTime,omitempty"`
}You can now run make manifests to have Kubebuilder generate the CRD yaml:
spec:
group: sandbox.rvmiller.com
names:
kind: WebPage
...
spec:
properties:
html:
description: Html field stores the static web page contents
minLength: 1
type: string
required:
- html
...
status:
properties:
lastUpdateTime:
description: Stores the last time the job was successfully scheduled.
format: date-time
type: stringStep 4: Define the controller to reconcile state changes
This is where the core logic for your controller lives, reconciling actual state with desired state, based on updates made to WebPage CRDs.
func (r *WebPageReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
ctx := context.Background()
log := r.Log.WithValues("webpage", req.NamespacedName)
log.Info("starting reconcile")
// Get custom resource
var webpage api.WebPage
if err := r.Get(ctx, req.NamespacedName, &webpage); err != nil {
log.Error(err, "unable to fetch WebPage")
return ctrl.Result{}, client.IgnoreNotFound(err)
}
// Desired ConfigMap
cm, err := r.desiredConfigMap(webpage)
if err != nil {
return ctrl.Result{}, err
}
// Desired Deployment
dep, err := r.desiredDeployment(webpage, cm)
if err != nil {
return ctrl.Result{}, err
}
// Patch (create/update) both owned resources
applyOpts := []client.PatchOption{client.ForceOwnership, client.FieldOwner("webpage-controller")}
err = r.Patch(ctx, &cm, client.Apply, applyOpts...)
if err != nil {
return ctrl.Result{}, err
}
err = r.Patch(ctx, &dep, client.Apply, applyOpts...)
if err != nil {
return ctrl.Result{}, err
}
// Set the last update time
webpage.Status.LastUpdateTime = &metav1.Time{Time: time.Now()}
if err = r.Status().Update(ctx, &webpage); err != nil {
log.Error(err, "unable to update status")
}
log.Info("finished reconcile")
return ctrl.Result{}, nil
}These helper functions allow creating/updating a Deployment and ConfigMap with the values we’re interested in. We can then PATCH the objects in Kubernetes to avoid maintaining and updating the state for the whole object. This approach is based on a talk from Kubecon 2019 documented here.
func (r *WebPageReconciler) desiredConfigMap(webpage api.WebPage) (corev1.ConfigMap, error) {
cm := corev1.ConfigMap{
TypeMeta: metav1.TypeMeta{APIVersion: corev1.SchemeGroupVersion.String(), Kind: "ConfigMap"},
ObjectMeta: metav1.ObjectMeta{
Name: webpage.Name + "-config",
Namespace: webpage.Namespace,
},
Data: map[string]string{
"index.html": webpage.Spec.Html,
},
}
// For garbage collector to clean up resource
if err := ctrl.SetControllerReference(&webpage, &cm, r.Scheme); err != nil {
return cm, err
}
return cm, nil
}
func (r *WebPageReconciler) desiredDeployment(webpage api.WebPage, cm corev1.ConfigMap) (appsv1.Deployment, error) {
dep := appsv1.Deployment{
TypeMeta: metav1.TypeMeta{APIVersion: appsv1.SchemeGroupVersion.String(), Kind: "Deployment"},
ObjectMeta: metav1.ObjectMeta{
Name: webpage.Name,
Namespace: webpage.Namespace,
},
Spec: appsv1.DeploymentSpec{
Selector: &metav1.LabelSelector{
MatchLabels: map[string]string{"webpage": webpage.Name},
},
Template: corev1.PodTemplateSpec{
ObjectMeta: metav1.ObjectMeta{
Labels: map[string]string{"webpage": webpage.Name},
},
Spec: corev1.PodSpec{
Containers: []corev1.Container{
{
Name: "nginx",
Image: "nginx",
VolumeMounts: []corev1.VolumeMount{
{
Name: "config-volume",
MountPath: "/usr/share/nginx/html",
},
},
},
},
Volumes: []corev1.Volume{
{
Name: "config-volume",
VolumeSource: corev1.VolumeSource{
ConfigMap: &corev1.ConfigMapVolumeSource{
LocalObjectReference: corev1.LocalObjectReference{
Name: cm.Name,
},
},
},
},
},
},
},
},
}
// For garbage collector to clean up resource
if err := ctrl.SetControllerReference(&webpage, &dep, r.Scheme); err != nil {
return dep, err
}
return dep, nil
}Step 5: Compile, Install, and Run
Now you can use Kubebuilder to generate manifests for your CRD, install the CRD into your running Kubernetes cluster, and run your controller locally (will take up your terminal):
$ make
go build -o bin/manager main.go
$ make install
kustomize build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/webpages.sandbox.rvmiller.com created
$ make run
2020-07-04T22:21:21.748-0400 INFO setup starting manager
...Step 6: View your hard work
You can use the webpage.yaml file from the top of this blog post and apply it to your cluster. You will see a configmap and pod created, and when you port-forward to the running pod, you can view your HTML locally in a web browser!
$ kubectl apply -f webpage.yaml
webpage.sandbox.rvmiller.com/sample-web-page created
$ kubectl get configmaps
NAME DATA AGE
sample-web-page-config 1 10m
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
sample-web-page-nginx 1/1 Running 0 10m
$ kubectl port-forward sample-web-page-nginx 7070:80
Forwarding from 127.0.0.1:7070 -> 80
Forwarding from [::1]:7070 -> 80
You can also edit the CRD, wait a few seconds for reconciliation, and force refresh the page to see the updated HTML:
$ kubectl edit webpage.sandbox.rvmiller.com sample-web-page
...
<h2>Another test...</h2>
....And you can view the status of the custom resource to view its last updated time:
$ kubectl get webpage.sandbox.rvmiller.com sample-web-page -o yaml
...
status:
lastUpdateTime: "2020-07-05T03:14:34Z"Since we “Own” the Deployment and ConfigMap objects, they will be reconciled even if you delete them. Normally deleting a deployment is final, but you will see if you delete the deployment sample-web-page it will be recreated through reconciliation.
Step 7: Tear down
Thanks to the ControllerReference’s set above, you can simply remove the CRD to have all the dependent resources automatically garbage collected:
$ kubectl delete crd webpage.sandbox.rvmiller.com
customresourcedefinition.apiextensions.k8s.io "webpages.sandbox.rvmiller.com" deleted
$ k get pods
NAME READY STATUS RESTARTS AGE
sample-web-page-nginx 0/1 Terminating 0 7m39s