Building Kubernetes Controllers for Multi-Cluster Karmada

Karmada is fully compatible with Kubernetes API, so if you're familiar with writing Kubernetes controllers using Kubebuilder, you're almost good to go. In this post, we'll walk through the key differences and requirements when developing controllers for a multi-cluster Karmada environment.

Example available

Controlled Replica Propagation

Installation: Split the Kustomize Configuration

When developing for Karmada, you’ll need to split the installation into two Kustomize configurations. One to be applied to the Karmada API and one to be applied to the Karmada host cluster (Kubernetes API).

By default, Kubebuilder generates everything in config/default. We’ll instead split it into two folders: config/karmada and config/host.

config/karmada/ folder prepares resources, such as: CRDs, RBAC (service account, roles and role bindings) and namespace.

resources:
  - ../crd
  - ../rbac
  - namespace.yaml

config/host/ folder handles controller deployment and related configurations for the cluster where the controller actually runs.

resources:
  - ../manager

Connecting to the Karmada API from the Host Cluster

By default, a controller uses a Kubernetes service account to access the API of its local cluster. However, when working with Karmada, the controller needs to connect to the Karmada API, not the host cluster's API.

Disable Default Service Account

Remove the service account reference from config/manager/manager.yaml and instead mount a Karmada Kubeconfig:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: controller-manager
  ...
      containers:
      - args:
          - --leader-elect
          - --health-probe-bind-address=:8081
          - --kubeconfig=/karmadaconfig/karmada.config
        ...
        volumeMounts:
          - name: karmadaconfig
            mountPath: /karmadaconfig
            readOnly: true
      volumes:
        - name: karmadaconfig
          secret:
            secretName: karmada-my-controller-config
      # serviceAccountName: controller-manager # <- remove this
  ...

Create the Karmada Kubeconfig

After applying config/karmada, a service account will exist in the Karmada API. You'll need to generate a Kubeconfig from that service account, and then create a Secret in the host cluster so your controller can authenticate.

Here’s a helper script you can place in hack/gen_config.sh:

#!/bin/bash

set -o errexit
set -o nounset
set -o pipefail

context="karmada-apiserver"
cluster_name="karmada-apiserver"
address="$(kubectl --context "$context" config view --minify -o jsonpath='{.clusters[].cluster.server}')"

namespace="controller-namespace"
service_account_name="controller-manager"

ca="$(kubectl --context "$context" get configmap cluster-info --namespace kube-public -o json | yq '.data.kubeconfig' | yq '.clusters[0].cluster.certificate-authority-data')"
# 365 days
token="$(kubectl create token --duration 31536000s --namespace "$namespace" "$service_account_name")"

echo "apiVersion: v1
kind: Config
clusters:
- name: ${cluster_name}
  cluster:
    certificate-authority-data: ${ca}
    server: ${address}
contexts:
- name: ${service_account_name}@${cluster_name}
  context:
    cluster: ${cluster_name}
    namespace: ${namespace}
    user: ${service_account_name}
current-context: ${service_account_name}@${cluster_name}
users:
- name: ${service_account_name}
  user:
    token: ${token}" > sa.karmadaconfig

# This creates a secret with the Karmada Kubeconfig
kubectl --context="karmada-host" --namespace "$namespace" create secret generic karmada-my-controller-config --from-file=karmada.config=sa.karmadaconfig

⚠️ Note: The token expires in 365 days. Automate secret rotation in production.

Using Karmada CRDs in Your Controller

If your controller interacts with Karmada-native resources like PropagationPolicy, you’ll need to:

Register Karmada CRDs to Controller Runtime

In cmd/main.go:

import (
  karmadapolicyv1alpha1 "github.com/karmada-io/api/policy/v1alpha1"
)

func init() {
  utilruntime.Must(clientgoscheme.AddToScheme(scheme))
  utilruntime.Must(karmadapolicyv1alpha1.AddToScheme(scheme))
  utilruntime.Must(mygroupv1alpha1.AddToScheme(scheme))
}

Include Karmada CRDs in Your Test Suite

Add Karmada CRDs under test/vendor_crds, e.g., karmada-io-v1alpha1.yaml.

Update internal/controller/suite_test.go:

testEnv = &envtest.Environment{
  CRDDirectoryPaths: []string{
    filepath.Join("..", "..", "config", "crd", "bases"),
    filepath.Join("..", "..", "test", "vendor_crds", "karmada-io-v1alpha1.yaml"),
  },
  ErrorIfCRDPathMissing: true,
}

Conclusion

Building a Kubernetes controller for Karmada isn't radically different, but it does have a few gotchas. By splitting your install into two parts and using a Kubeconfig to communicate with Karmada API, you can successfully run Karmada-aware controllers.

Need help with your Kubernetes project or want to upgrade your team's skills?
Let's connect