Tag Archives: customresourcedefinition

Tutorial: Writing a Kubernetes CRD and Controller with Kubebuilder

I recently came across the Kubebuilder tool which provides scaffolding to write your own CustomResourceDefinition (CRD) and Controller to extend the API running in Kubernetes clusters. I worked through the tutorial building a CronJob controller, but wanted to write something from scratch, and got some inspiration for a simple (contrived) example to write a “WebPage” CRD that supports CRUD operations for a static web page:

apiVersion: sandbox.rvmiller.com/v1beta1
kind: WebPage
metadata:
  name: sample-web-page
spec:
  html: |
    <html>
      <head>
        <title>WebPage CRD</title>
      </head>
      <body>
        <h2>This page served from a Kubernetes CRD!</h2>
      </body>
    </html>

The idea is that our custom controller will create a Deployment with a Pod running an nginx webserver image that uses a mounted ConfigMap volume to display the web page contents. We can then kubectl port-forward to the pod(s) to verify the web page is displayed.

You can find the code for this tutorial here.

Step 1: Set up the environment

Download and install Kubebuilder, Kustomize, and Kind to run your Kubernetes cluster locally. I prefer Kind over Minikube since it starts up faster, but you’re welcome to use any tool to deploy your cluster.

Step 2: Set up the scaffolding

Create a directory to store your Kubebuilder files:

$ mkdir webpage-crd
$ kubebuilder init --domain <any base domain, I used rvmiller.com>
$ kubebuilder create api --group sandbox --version v1beta1 --kind WebPage

This will generate a CRD to create resources with the following manifest:

apiVersion: sandbox.rvmiller.com/v1beta1
kind: WebPage

Step 3: Define the Custom Resource fields

We need to update the generated webpage_types.go file with our custom Spec and Status fields. We can also use kubebuilder’s annotations to add OpenAPI validation:

// WebPageSpec defines the desired state of WebPage
type WebPageSpec struct {
	// Html field stores the static web page contents
	// +kubebuilder:validation:MinLength=1
	Html string `json:"html"`
}

// WebPageStatus defines the observed state of WebPage
type WebPageStatus struct {
	// Stores the last time the job was successfully scheduled.
	// +optional
	LastUpdateTime *metav1.Time `json:"lastUpdateTime,omitempty"`
}

You can now run make manifests to have Kubebuilder generate the CRD yaml:

spec:
  group: sandbox.rvmiller.com
  names:
    kind: WebPage
...
        spec:
          properties:
            html:
              description: Html field stores the static web page contents
              minLength: 1
              type: string
          required:
          - html
...
        status:
          properties:
            lastUpdateTime:
              description: Stores the last time the job was successfully scheduled.
              format: date-time
              type: string

Step 4: Define the controller to reconcile state changes

This is where the core logic for your controller lives, reconciling actual state with desired state, based on updates made to WebPage CRDs.

func (r *WebPageReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
	ctx := context.Background()
	log := r.Log.WithValues("webpage", req.NamespacedName)

	log.Info("starting reconcile")

	// Get custom resource
	var webpage api.WebPage
	if err := r.Get(ctx, req.NamespacedName, &webpage); err != nil {
		log.Error(err, "unable to fetch WebPage")
		return ctrl.Result{}, client.IgnoreNotFound(err)
	}

	// Desired ConfigMap
	cm, err := r.desiredConfigMap(webpage)
	if err != nil {
		return ctrl.Result{}, err
	}

	// Desired Deployment
	dep, err := r.desiredDeployment(webpage, cm)
	if err != nil {
		return ctrl.Result{}, err
	}

	// Patch (create/update) both owned resources
	applyOpts := []client.PatchOption{client.ForceOwnership, client.FieldOwner("webpage-controller")}

	err = r.Patch(ctx, &cm, client.Apply, applyOpts...)
	if err != nil {
		return ctrl.Result{}, err
	}

	err = r.Patch(ctx, &dep, client.Apply, applyOpts...)
	if err != nil {
		return ctrl.Result{}, err
	}

	// Set the last update time
	webpage.Status.LastUpdateTime = &metav1.Time{Time: time.Now()}
	if err = r.Status().Update(ctx, &webpage); err != nil {
		log.Error(err, "unable to update status")
	}

	log.Info("finished reconcile")

	return ctrl.Result{}, nil
}

These helper functions allow creating/updating a Deployment and ConfigMap with the values we’re interested in. We can then PATCH the objects in Kubernetes to avoid maintaining and updating the state for the whole object. This approach is based on a talk from Kubecon 2019 documented here.

func (r *WebPageReconciler) desiredConfigMap(webpage api.WebPage) (corev1.ConfigMap, error) {
	cm := corev1.ConfigMap{
		TypeMeta: metav1.TypeMeta{APIVersion: corev1.SchemeGroupVersion.String(), Kind: "ConfigMap"},
		ObjectMeta: metav1.ObjectMeta{
			Name:      webpage.Name + "-config",
			Namespace: webpage.Namespace,
		},
		Data: map[string]string{
			"index.html": webpage.Spec.Html,
		},
	}

	// For garbage collector to clean up resource
	if err := ctrl.SetControllerReference(&webpage, &cm, r.Scheme); err != nil {
		return cm, err
	}

	return cm, nil
}

func (r *WebPageReconciler) desiredDeployment(webpage api.WebPage, cm corev1.ConfigMap) (appsv1.Deployment, error) {
	dep := appsv1.Deployment{
		TypeMeta: metav1.TypeMeta{APIVersion: appsv1.SchemeGroupVersion.String(), Kind: "Deployment"},
		ObjectMeta: metav1.ObjectMeta{
			Name:      webpage.Name,
			Namespace: webpage.Namespace,
		},
		Spec: appsv1.DeploymentSpec{
			Selector: &metav1.LabelSelector{
				MatchLabels: map[string]string{"webpage": webpage.Name},
			},
			Template: corev1.PodTemplateSpec{
				ObjectMeta: metav1.ObjectMeta{
					Labels: map[string]string{"webpage": webpage.Name},
				},
				Spec: corev1.PodSpec{
					Containers: []corev1.Container{
						{
							Name:  "nginx",
							Image: "nginx",
							VolumeMounts: []corev1.VolumeMount{
								{
									Name:      "config-volume",
									MountPath: "/usr/share/nginx/html",
								},
							},
						},
					},
					Volumes: []corev1.Volume{
						{
							Name: "config-volume",
							VolumeSource: corev1.VolumeSource{
								ConfigMap: &corev1.ConfigMapVolumeSource{
									LocalObjectReference: corev1.LocalObjectReference{
										Name: cm.Name,
									},
								},
							},
						},
					},
				},
			},
		},
	}

	// For garbage collector to clean up resource
	if err := ctrl.SetControllerReference(&webpage, &dep, r.Scheme); err != nil {
		return dep, err
	}

	return dep, nil
}

Step 5: Compile, Install, and Run

Now you can use Kubebuilder to generate manifests for your CRD, install the CRD into your running Kubernetes cluster, and run your controller locally (will take up your terminal):

$ make
go build -o bin/manager main.go
$ make install
kustomize build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/webpages.sandbox.rvmiller.com created
$ make run
2020-07-04T22:21:21.748-0400    INFO    setup   starting manager
...

Step 6: View your hard work

You can use the webpage.yaml file from the top of this blog post and apply it to your cluster. You will see a configmap and pod created, and when you port-forward to the running pod, you can view your HTML locally in a web browser!

$ kubectl apply -f webpage.yaml 
webpage.sandbox.rvmiller.com/sample-web-page created
$ kubectl get configmaps
NAME                     DATA   AGE
sample-web-page-config   1      10m
$ kubectl get pods
NAME                    READY   STATUS    RESTARTS   AGE
sample-web-page-nginx   1/1     Running   0          10m
$ kubectl port-forward sample-web-page-nginx 7070:80
Forwarding from 127.0.0.1:7070 -> 80
Forwarding from [::1]:7070 -> 80

You can also edit the CRD, wait a few seconds for reconciliation, and force refresh the page to see the updated HTML:

$ kubectl edit webpage.sandbox.rvmiller.com sample-web-page
...
  <h2>Another test...</h2>
....

And you can view the status of the custom resource to view its last updated time:

$ kubectl get webpage.sandbox.rvmiller.com sample-web-page -o yaml
...
status:
  lastUpdateTime: "2020-07-05T03:14:34Z"

Since we “Own” the Deployment and ConfigMap objects, they will be reconciled even if you delete them. Normally deleting a deployment is final, but you will see if you delete the deployment sample-web-page it will be recreated through reconciliation.

Step 7: Tear down

Thanks to the ControllerReference’s set above, you can simply remove the CRD to have all the dependent resources automatically garbage collected:

$ kubectl delete crd webpage.sandbox.rvmiller.com
customresourcedefinition.apiextensions.k8s.io "webpages.sandbox.rvmiller.com" deleted
$ k get pods
NAME                    READY   STATUS        RESTARTS   AGE
sample-web-page-nginx   0/1     Terminating   0          7m39s