Kubernetes Operators

A Kubernetes operator is a program running in your cluster that manages one application the way an experienced administrator would. It has two halves. A custom resource gives you a new object type to describe what you want (say, "a PostgreSQL cluster with three instances"). A controller watches those objects and keeps doing whatever it takes to make the cluster match: creating Pods and volumes, promoting a replica when the primary dies, running backups. The Kubernetes documentation calls operators software extensions that use custom resources to manage applications, following the same control loop as the built-in controllers.

This chapter shows you each half on its own, then installs a production-grade operator and breaks its database on purpose so you can watch it recover. After that you'll scaffold your own.

The three parts of an operator#

You already know the pattern from Deployments. You write a spec, a controller inside kube-controller-manager compares it with what's running, and it creates or deletes ReplicaSets until the two agree. That comparison runs again every time something changes. An operator copies the mechanism for a resource type Kubernetes never shipped.

• The API. A CustomResourceDefinition (CRD) registers a new kind, with a schema, so the API server will store and validate objects of that kind.
• The loop. A controller, usually a Deployment in its own namespace, watches those objects and the things it creates from them. Each pass is called a reconcile.
• The knowledge. This is the part that earns the name. A Deployment controller knows nothing about databases; a PostgreSQL operator knows which replica has the most recent data and must be promoted, and in what order to restart instances during an upgrade.

Strip out the third part and you have a plain custom controller. Plenty of useful projects are exactly that, and the line between the two is fuzzy. The test this book uses: if the code encodes how to run a specific piece of software through its whole life (install, upgrade, failure, backup), it's an operator.

Before you start#

You need a cluster you can throw away and kubectl pointed at it. The KinD chapter gets you one in a minute; the CloudNativePG quickstart itself uses kind, and the database below needs a default StorageClass, which kind provides. K3s works too. One honest caveat: CloudNativePG 1.30 lists Kubernetes 1.34, 1.35 and 1.36 as supported and 1.37 as tested but not yet supported, so for anything beyond practice, match your cluster to that list.

Step 1: a CRD on its own does nothing#

Start with the API half and nothing else. Save this as website-crd.yaml. It defines a Website kind with an image, a replica count between 1 and 10, and two extra columns for kubectl get:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: websites.ops.example.com     # must be <plural>.<group>
spec:
  group: ops.example.com
  scope: Namespaced
  names:
    plural: websites
    singular: website
    kind: Website
    shortNames:
    - web
  versions:
  - name: v1
    served: true
    storage: true                    # exactly one version stores data
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              image:
                type: string
              replicas:
                type: integer
                minimum: 1
                maximum: 10
                default: 1
    additionalPrinterColumns:
    - name: Image
      type: string
      jsonPath: .spec.image
    - name: Replicas
      type: integer
      jsonPath: .spec.replicas

Apply it and confirm the API server knows about the new type:

kubectl apply -f website-crd.yaml
kubectl get crd websites.ops.example.com
kubectl api-resources --api-group=ops.example.com
kubectl explain website.spec

kubectl explain works because the schema you wrote is published as OpenAPI, which is also how kubectl apply can reject a bad field before it reaches the server. Now create an object of the new kind:

cat <<EOF | kubectl apply -f -
apiVersion: ops.example.com/v1
kind: Website
metadata:
  name: docs
spec:
  image: nginx:1.29
EOF
kubectl get web

The output looks like this, with replicas filled in by the schema default:

NAME   IMAGE        REPLICAS
docs   nginx:1.29   1

Try replicas: 20 and it's rejected against the schema. Then run kubectl get pods. Nothing. No Deployment, no Pod, no error either. The CRD gave you storage and validation, and that's all: the Kubernetes docs say custom resources only become a declarative API when you pair them with a controller. Delete it before moving on, because deleting a CRD also deletes every object of that kind:

kubectl delete -f website-crd.yaml

Step 2: install a real operator#

CloudNativePG runs PostgreSQL on Kubernetes and is a good one to learn on: it's open source, its documentation lives in the same repository as the code, and a database is exactly the kind of stateful workload where an operator pays for itself. The whole operator ships as one manifest:

kubectl apply --server-side -f \
  https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.30/releases/cnpg-1.30.1.yaml

kubectl rollout status deployment -n cnpg-system cnpg-controller-manager

That single file created a namespace (cnpg-system), RBAC, webhooks, a Deployment called cnpg-controller-manager and a set of CRDs. List the CRDs to see the API you've just been given:

kubectl get crd | grep cnpg.io

You'll find clusters.postgresql.cnpg.io, backups.postgresql.cnpg.io, scheduledbackups.postgresql.cnpg.io, poolers.postgresql.cnpg.io and several more. Each one is a noun the operator understands. Compare that with Step 1: same mechanism, but now something is watching.

Step 3: ask for a database#

This is the sample from the CloudNativePG quickstart. Six lines of spec:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: cluster-example
spec:
  instances: 3
  storage:
    size: 1Gi

kubectl apply -f cluster-example.yaml
kubectl wait --for=condition=Ready cluster/cluster-example --timeout=10m
kubectl get cluster cluster-example

The Cluster CRD defines its own printer columns, so once it's healthy the output looks like this:

NAME              AGE   INSTANCES   READY   STATUS                     PRIMARY
cluster-example   4m    3           3       Cluster in healthy state   cluster-example-1

Look at what got created on your behalf. Everything carries the cnpg.io/cluster label:

kubectl get pods,pvc,svc -l cnpg.io/cluster=cluster-example

Three Pods, three PersistentVolumeClaims, and three Services: cluster-example-rw points at the primary, cluster-example-ro at the replicas, cluster-example-r at any instance. Your applications connect to -rw and never need to know which Pod is primary today. Notice there's no StatefulSet: CloudNativePG's storage docs say it doesn't use one, and manages the Pods and volumes itself so it can decide which instance is replaced and how.

Step 4: kill the primary and watch the loop#

Here's where a Helm chart or a plain StatefulSet would leave you on your own. Find the primary through its role label and delete it:

kubectl get pods -l cnpg.io/cluster=cluster-example,cnpg.io/instanceRole=primary
kubectl delete pod cluster-example-1
kubectl get cluster cluster-example -w

Watch the STATUS column. The CloudNativePG failover docs describe what happens: the controller notices the failure in its reconcile loop, shuts down the old primary so the replicas stop streaming from it, elects the most up-to-date replica and promotes it. The -rw Service is updated to point at the new primary. When the deleted Pod comes back, it detects that it's no longer primary and rejoins as a replica. Press Ctrl+C when PRIMARY shows a different instance and READY is back to 3.

Nobody paged a DBA. That sequence (fence, elect, promote, re-point, rejoin) is the operational knowledge from the first section, written down as code.

Operator, controller or Helm chart?#

The three get confused because an operator is often installed with a Helm chart, and every operator contains a controller. They answer different questions:

A quick way to tell what you're looking at in a cluster: run kubectl api-resources and look for API groups you didn't install yourself, then check whether a Deployment somewhere is watching them. A chart with no running controller can't react to anything after install.

How capable is an operator? The five levels#

The Operator Framework publishes a capability model you'll see on OperatorHub listings. Each level is a set of features, and the framework notes they can be independent of each other:

1. Basic Install: provisions and configures the application from a custom resource.
2. Seamless Upgrades: upgrades the operator and the application it manages without much effort from you.
3. Full Lifecycle: backup and restore, complex reconfiguration, resilience features like probes and PodDisruptionBudgets.
4. Deep Insights: metrics, alerts and events about both the operator and the application.
5. Auto Pilot: auto-scaling, auto-healing, auto-tuning and anomaly detection.

CloudNativePG's own docs claim Level V, and the failover you just triggered is listed under it as self-healing. Treat the level as the vendor's self-assessment. It's a useful shortlist filter on OperatorHub.io, not a certification.

Building your own with Kubebuilder#

Any language that can talk to the Kubernetes API can host a controller. In Go, two tools dominate, and they're closer than they look. Kubebuilder, from the kubernetes-sigs organisation, scaffolds a controller-runtime project. Operator SDK uses Kubebuilder under the hood for Go projects, according to its own FAQ, and adds Operator Lifecycle Manager integration, OperatorHub bundles, scorecard tests, and Helm- and Ansible-based operator types. Start with Kubebuilder; switch to Operator SDK if you plan to publish through OLM.

Install the CLI as the Kubebuilder book shows. The v4.16.0 scaffold declares go 1.26.0 in its go.mod and builds against Kubernetes 1.37 client libraries (k8s.io/api v0.37.0, controller-runtime v0.25.0), so use a current Go toolchain:

curl -L -o kubebuilder "https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)"
chmod +x kubebuilder && sudo mv kubebuilder /usr/local/bin/

mkdir memcached-operator && cd memcached-operator
kubebuilder init --domain example.com --repo example.com/memcached
kubebuilder create api --group cache --version v1alpha1 --kind Memcached

Answer y to both prompts. You get api/v1alpha1/memcached_types.go, where the Go structs become your CRD schema, and internal/controller/memcached_controller.go, where the reconcile function lives. Below is a trimmed version of the reconcile function from the book's Memcached tutorial. It keeps the shape and drops the error-path status updates:

// +kubebuilder:rbac:groups=cache.example.com,resources=memcacheds,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=cache.example.com,resources=memcacheds/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete

func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	log := logf.FromContext(ctx)

	// 1. Read the desired state. Gone? Owned objects are garbage-collected.
	memcached := &cachev1alpha1.Memcached{}
	if err := r.Get(ctx, req.NamespacedName, memcached); err != nil {
		if apierrors.IsNotFound(err) {
			return ctrl.Result{}, nil
		}
		return ctrl.Result{}, err
	}

	// 2. Read the actual state and create what's missing.
	found := &appsv1.Deployment{}
	err := r.Get(ctx, types.NamespacedName{Name: memcached.Name, Namespace: memcached.Namespace}, found)
	if apierrors.IsNotFound(err) {
		dep, err := r.deploymentForMemcached(memcached) // sets the owner reference
		if err != nil {
			return ctrl.Result{}, err
		}
		log.Info("Creating a new Deployment", "Deployment.Name", dep.Name)
		if err := r.Create(ctx, dep); err != nil {
			return ctrl.Result{}, err
		}
		return ctrl.Result{RequeueAfter: time.Minute}, nil
	} else if err != nil {
		return ctrl.Result{}, err
	}

	// 3. Close the gap between spec and reality.
	var desired int32
	if memcached.Spec.Size != nil {
		desired = *memcached.Spec.Size
	}
	if found.Spec.Replicas == nil || *found.Spec.Replicas != desired {
		found.Spec.Replicas = &desired
		if err := r.Update(ctx, found); err != nil {
			return ctrl.Result{}, err
		}
		return ctrl.Result{RequeueAfter: time.Minute}, nil
	}

	// 4. Report what you observed.
	meta.SetStatusCondition(&memcached.Status.Conditions, metav1.Condition{
		Type: "Available", Status: metav1.ConditionTrue,
		Reason: "Reconciling", Message: "Deployment matches spec",
	})
	return ctrl.Result{}, r.Status().Update(ctx, memcached)
}

func (r *MemcachedReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&cachev1alpha1.Memcached{}).
		Owns(&appsv1.Deployment{}).
		Named("memcached").
		Complete(r)
}

Three details carry the design. The function never asks "what event happened?"; it reads current state and fixes the difference, so running it twice is harmless (the book calls this out: reconcile must be idempotent). Owns(&appsv1.Deployment{}) means that if someone edits or deletes the Deployment by hand, the Memcached object is queued again and the change is undone. And the +kubebuilder:rbac comments aren't decoration: make manifests turns them into the controller's Role.

The development loop runs against the cluster you already have:

make manifests          # regenerate CRD and RBAC from the Go markers
make install            # apply the CRDs to the cluster
make run                # run the controller locally, in the foreground
kubectl apply -k config/samples/

# when it works, ship it as an image
make docker-build docker-push IMG=<registry>/memcached-operator:v0.1.0
make deploy IMG=<registry>/memcached-operator:v0.1.0

The full tutorial, with finalizers, events and tests, is the Kubebuilder getting-started guide.

Python and other languages#

If Go isn't your language, Kopf lets you write handlers as decorated Python functions (Python 3.11+ per its README). A handler looks like @kopf.on.create('ephemeralvolumeclaims') above a function that receives the object's spec, and you start it with kopf run ephemeral.py --verbose. The Kubernetes operator page also lists kube-rs for Rust, the Java Operator SDK, KubeOps for .NET and shell-operator.

When you shouldn't write one#

An operator is code you'll maintain against every Kubernetes release, with permission to create and delete objects in your cluster. Before writing one, check these:

• Someone already built it. Search OperatorHub and the project's own docs first; databases, message queues and certificate managers are well covered.
• Your "state" is a config file. The Kubernetes docs suggest a ConfigMap when the program reads a file or environment variables and a rolling restart on change is enough.
• You only need day-one install. That's a Helm chart or Kustomize overlay.
• You're composing cloud infrastructure out of existing APIs. Crossplane compositions give you a custom API without writing a controller.

Write one when a person on your team keeps doing the same careful sequence by hand, and getting the order wrong breaks something.

Common mistakes#

• Deleting the operator before its resources. Removing the manifest deletes the CRDs, and with them every object of those kinds. Delete your Clusters first, on purpose.
• Forgetting --server-side on the CloudNativePG install; its docs use server-side apply for the manifest.
• Writing reconcile like an event handler ("on create, do X"). Missed events are normal. Read state, compare, fix.
• Updating status with a stale object, which produces "the object has been modified" conflicts. The Kubebuilder tutorial re-fetches the resource before writing status for this reason.

Cleanup#

Delete in reverse order, and check for leftover volumes:

kubectl delete cluster cluster-example
kubectl get pvc -l cnpg.io/cluster=cluster-example
kubectl delete -f \
  https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.30/releases/cnpg-1.30.1.yaml

If you followed the Kubebuilder part, make undeploy and make uninstall remove the controller and its CRDs. Or delete the kind cluster and start fresh.

Where this fits in the book#

You've already used operators without naming them: the kube-prometheus-stack chart in the Prometheus chapter installs the Prometheus Operator, whose prometheuses.monitoring.coreos.com CRD works exactly like the Cluster object above. Crossplane, in the templating section, is the same idea pointed at cloud resources.

Further watching#

• Video by Anais Urlichs (the original 100 Days of Kubernetes series)
• CloudNativePG quickstart, including the Prometheus and Grafana part this chapter skipped
• Extend the Kubernetes API with CustomResourceDefinitions, for versioning, CEL validation rules and the status subresource