Tag Archives: Kubernetes

Tutorial: Writing a Kubernetes CRD and Controller with Kubebuilder

I recently came across the Kubebuilder tool which provides scaffolding to write your own CustomResourceDefinition (CRD) and Controller to extend the API running in Kubernetes clusters. I worked through the tutorial building a CronJob controller, but wanted to write something from scratch, and got some inspiration for a simple (contrived) example to write a “WebPage” CRD that supports CRUD operations for a static web page:

apiVersion: sandbox.rvmiller.com/v1beta1
kind: WebPage
  name: sample-web-page
  html: |
        <title>WebPage CRD</title>
        <h2>This page served from a Kubernetes CRD!</h2>

The idea is that our custom controller will create a Pod running an nginx webserver image and update a mounted ConfigMap volume to display the web page contents. We can then kubectl port-forward to the pod(s) to verify the web page is displayed.

You can find the code for this tutorial here.

Step 1: Set up the environment

Download and install Kubebuilder, Kustomize, and Kind to run your Kubernetes cluster locally. I prefer Kind over Minikube since it starts up faster, but you’re welcome to use any tool to deploy your cluster.

Step 2: Set up the scaffolding

Create a directory to store your Kubebuilder files:

$ mkdir webpage-crd
$ kubebuilder init --domain <any base domain, I used rvmiller.com>
$ kubebuilder create api --group sandbox --version v1beta1 --kind WebPage

This will generate a CRD to create resources with the following manifest:

apiVersion: sandbox.rvmiller.com/v1beta1
kind: WebPage

Step 3: Define the Custom Resource fields

We need to update the generated webpage_types.go file with our custom Spec and Status fields. We can also use kubebuilder’s annotations to add OpenAPI validation:

// WebPageSpec defines the desired state of WebPage
type WebPageSpec struct {
	// Html field stores the static web page contents
	// +kubebuilder:validation:MinLength=1
	Html string `json:"html"`

// WebPageStatus defines the observed state of WebPage
type WebPageStatus struct {
	// Stores the last time the job was successfully scheduled.
	// +optional
	LastUpdateTime *metav1.Time `json:"lastUpdateTime,omitempty"`

You can now run make manifests to have Kubebuilder generate the CRD yaml:

  group: sandbox.rvmiller.com
    kind: WebPage
              description: Html field stores the static web page contents
              minLength: 1
              type: string
          - html
              description: Stores the last time the job was successfully scheduled.
              format: date-time
              type: string

Step 4: Define the controller to reconcile state changes

This is where the core logic for your controller lives, reconciling actual state with desired state, based on updates made to WebPage CRDs.

func (r *WebPageReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
	ctx := context.Background()
	log := r.Log.WithValues("webpage", req.NamespacedName)

	// Get custom resource
	var webpage api.WebPage
	if err := r.Get(ctx, req.NamespacedName, &webpage); err != nil {
		log.Error(err, "unable to fetch WebPage")
		return ctrl.Result{}, client.IgnoreNotFound(err)

	// CreatOrUpdate ConfigMap
	var cm corev1.ConfigMap
	cm.Name = webpage.Name + "-config"
	cm.Namespace = webpage.Namespace
	_, err := ctrl.CreateOrUpdate(ctx, r, &cm, func() error {
		cm.Data = map[string]string{
			"index.html": webpage.Spec.Html,
		// For garbage collector to clean up resource
		return util.SetControllerReference(&webpage, &cm, r.Scheme)
	if err != nil {
		log.Error(err, "unable to CreateOrUpdate configmap")
		return ctrl.Result{}, err

	// Create nginx pod with mounted ConfigMap volume if it does not exist
	if webpage.Status.LastUpdateTime == nil {
		var pod corev1.Pod
		pod.Name = webpage.Name + "-nginx"
		pod.Namespace = webpage.Namespace
		_, err = ctrl.CreateOrUpdate(ctx, r, &pod, func() error {
			pod.Spec.Containers = []corev1.Container{
					Name:  "nginx",
					Image: "nginx",
					VolumeMounts: []corev1.VolumeMount{
							Name:      "config-volume",
							MountPath: "/usr/share/nginx/html",
			pod.Spec.Volumes = []corev1.Volume{
					Name: "config-volume",
					VolumeSource: corev1.VolumeSource{
						ConfigMap: &corev1.ConfigMapVolumeSource{
							LocalObjectReference: corev1.LocalObjectReference{
								Name: webpage.Name + "-config",
			// For garbage collector to clean up resource
			return util.SetControllerReference(&webpage, &pod, r.Scheme)
		if err != nil {
			log.Error(err, "unable to CreateOrUpdate pod")
			return ctrl.Result{}, err

	// Set the last update time
	webpage.Status.LastUpdateTime = &metav1.Time{Time: time.Now()}
	if err = r.Status().Update(ctx, &webpage); err != nil {
		log.Error(err, "unable to update status")

	return ctrl.Result{}, nil

Step 5: Compile, Install, and Run

Now you can use Kubebuilder to generate manifests for your CRD, install the CRD into your running Kubernetes cluster, and run your controller locally (will take up your terminal):

$ make
go build -o bin/manager main.go
$ make install
kustomize build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/webpages.sandbox.rvmiller.com created
$ make run
2020-07-04T22:21:21.748-0400    INFO    setup   starting manager

Step 6: View your hard work

You can use the webpage.yaml file from the top of this blog post and apply it to your cluster. You will see a configmap and pod created, and when you port-forward to the running pod, you can view your HTML locally in a web browser!

$ kubectl apply -f webpage.yaml 
webpage.sandbox.rvmiller.com/sample-web-page created
$ kubectl get configmaps
NAME                     DATA   AGE
sample-web-page-config   1      10m
$ kubectl get pods
NAME                    READY   STATUS    RESTARTS   AGE
sample-web-page-nginx   1/1     Running   0          10m
$ kubectl port-forward sample-web-page-nginx 7070:80
Forwarding from -> 80
Forwarding from [::1]:7070 -> 80

You can also edit the CRD, wait a few seconds for reconciliation, and force refresh the page to see the updated HTML:

$ kubectl edit webpage.sandbox.rvmiller.com sample-web-page
  <h2>Another test...</h2>

And you can view the status of the custom resource to view its last updated time:

$ kubectl get webpage.sandbox.rvmiller.com sample-web-page -o yaml
  lastUpdateTime: "2020-07-05T03:14:34Z"

Step 7: Tear down

Thanks to the ControllerReference’s set above, you can simply remove the CRD to have all the dependent resources automatically garbage collected:

$ kubectl delete crd webpage.sandbox.rvmiller.com
customresourcedefinition.apiextensions.k8s.io "webpages.sandbox.rvmiller.com" deleted
$ k get pods
NAME                    READY   STATUS        RESTARTS   AGE
sample-web-page-nginx   0/1     Terminating   0          7m39s

Kubernetes Secret Environment Variable Gotcha

Playing around with starting a MySQL pod with an environment variable populated from a secret on Kubernetes, I experienced a gotcha with an error message that I couldn’t easily find googling around:

The Error Message

mysqladmin: [ERROR] unknown option '--"'.

The Investigation

Since I had previously started a MySQL pod with a non-secret env variable without any problems, I suspected an issue with my configuration:

apiVersion: v1
kind: Secret
  name: mysql-root-password
type: Opaque
apiVersion: v1
kind: Pod
  name: db
    - name: mysql
      image: mysql
        - secretRef:
            name: mysql-root-password

The value “cGFzc3dvcmQK” comes from base64-encoding the password, in this case the word “password”:

$ echo "password" | base64

But this is actually incorrect, since echo will implicitly add a newline character, which gets base64-encoded into the string! When this string later gets base64-decoded inside Kubernetes, the environment variables in the MySQL container look like this:

$ kubectl exec -it db printenv


That newline character is included, and MySQL fails to start attempting to apply an option for an empty environment variable (‘–“‘), causing that somewhat-confusing error message to appear in the container logs.

The Takeaway

Be sure to base64-encode secrets without the newline character. When generating the secret with echo, you should use the “-n” flag to strip the newline character:

$ echo -n "password" | base64

Using this encoded string will prevent empty environment variables being injected into the MySQL container and allow MySQL to start:

$ kubectl exec -it db printenv

I did come across this issue which describes the gotcha affecting other applications as well, even 4 years after it was originally filed. But since I couldn’t find any posts with this exact MySQL error log, I thought I’d post my experience.