Skip to main content

Leader Election

KubeOps supports different leader election mechanisms through the LeaderElectionType setting. This allows you to control how multiple operator instances coordinate in a cluster.

Leader Election Types​

public enum LeaderElectionType
{
    None = 0,    // No leader election - all instances process events
    Single = 1,  // Single leader election - only one instance processes events
    Custom = 2,  // Escape hatch - user replaces watcher and queue consumer entirely
    Scoped = 3   // Scoped leadership - responsibility partitioned across instances
}

Configuration​

builder.Services
    .AddKubernetesOperator(settings => settings
        .WithLeaderElection(LeaderElectionType.Single)
        .WithLeaderElectionTimings(
            leaseDuration: TimeSpan.FromSeconds(15),
            renewDeadline: TimeSpan.FromSeconds(10),
            retryPeriod: TimeSpan.FromSeconds(2)));

Behavior on Leadership Loss​

With LeaderElectionType.Single, both the resource watcher and the reconciliation queue are gated on leadership. When an instance loses its lease it:

  1. suspends the queue's intake — any further enqueue is dropped, so a still-running reconciler's RequeueAfter and error retries cannot leave work behind;
  2. cancels the queue processing loop and any in-flight reconciliation; and
  3. clears the queue — pending and scheduled entries accumulated by the former leader are discarded.

While the instance is not the leader the intake stays closed. On re-acquiring leadership the intake reopens and the watcher re-lists the current state, so processing resumes from a clean slate.

What this does and does not guarantee. The gate protects the queue side: no stale entry, timed requeue, or error retry from a former leader survives a leadership transition. It does not terminate the process (unlike controller-runtime), so it cannot stop a non-cooperative reconciler that ignores its CancellationToken from performing external side effects while it was the leader. Therefore:

  • Reconcilers must honor cancellation and return promptly once their token is cancelled.
  • Reconcilers must be idempotent. As a second line of defense, concurrent writes to the same object are serialized by the API server via resourceVersion; a stale write fails with HTTP 409 Conflict.

See the Kubernetes leases documentation for leader-election semantics.

Scoped Leader Election​

LeaderElectionType.Scoped partitions responsibility across multiple operator instances instead of electing a single global leader. The default watcher and queue pipeline (including multiple controllers per entity and shared watchers) runs on every instance; before an event is enqueued and before a queued entry is reconciled, the SDK asks a user-provided ILeadershipScope whether this instance is responsible for the entity. The partitioning dimension (namespace, labels, name hashing, region annotations, ...) and the coordination mechanism (per-namespace Lease objects, a wildcard lease, an external assignment service, ...) are entirely up to the scope implementation.

Step 1: Implement the leadership scope

For the common namespace partitioning, derive from NamespacedLeadershipScope:

using KubeOps.Abstractions.LeaderElection;

public sealed class LeaseBasedNamespaceScope : NamespacedLeadershipScope
{
    protected override ValueTask<bool> IsResponsibleForNamespaceAsync(
        string? @namespace, CancellationToken cancellationToken)
    {
        // Answer from cached state - this is called for every watch event and every
        // dequeued entry. Coordinate however fits your deployment, e.g. per-namespace
        // leases, a wildcard lease, or consistent hashing.
        return ValueTask.FromResult(/* ... */ true);
    }

    // Call OnScopeChanged() whenever the assignment changes, e.g. from a background
    // service that renews and observes the leases.
}

For any other partitioning dimension, implement ILeadershipScope directly - the decision is made per entity:

public sealed class RegionLeadershipScope : ILeadershipScope
{
    public event Action? ScopeChanged;

    public ValueTask<bool> IsResponsibleForAsync(
        IKubernetesObject<V1ObjectMeta> entity, CancellationToken cancellationToken) =>
        ValueTask.FromResult(entity.GetAnnotation("example.com/region") == myRegion);
}

Step 2: Register the scope

builder.Services
    .AddKubernetesOperator(settings => settings
        .WithLeaderElection(LeaderElectionType.Scoped))
    .RegisterComponents();

builder.Services.AddSingleton<ILeadershipScope, LeaseBasedNamespaceScope>();

The SDK wires a ScopeAwareResourceWatcher and a ScopeAwareEntityQueueBackgroundService per entity; no elector, custom watcher, or custom queue consumer is required.

Semantics

  • The watch stream runs permanently on every instance; events for entities the instance is not responsible for are skipped before deduplication and enqueueing.
  • Every dequeued entry is re-checked, so entries that were pending when responsibility moved to another instance are skipped instead of reconciled twice.
  • When ScopeChanged is raised, this entity type's deduplication cache is dropped and the watched entities are re-listed (applying the configured namespace, label selector, and field selector) and run through the regular event path, so creations and modifications that happened while another instance was (or nobody was) responsible are picked up. Clearing the cache first guarantees that an entity handed back to this instance is reconciled even when its generation/resourceVersion is unchanged since the previous ownership term — the takeover reconcile would otherwise be suppressed as a duplicate. Reconcilers must therefore be idempotent, since every in-scope entity is reconciled once per scope change.
  • Deletions are not recovered by the resync. The resync is a List of objects that still exist; it cannot reconstruct a tombstone. If an object without a blocking finalizer is deleted during a window in which no instance was responsible (or before the acquiring instance's watch observed the Deleted event), the Deleted event is dropped at the responsibility gate and DeletedAsync runs on no instance. This mirrors standard Kubernetes semantics, where Deleted callbacks are inherently best-effort (an operator that is down during a deletion misses them too). For cleanup that must not be lost across scope handovers, use a finalizer: a blocking finalizer keeps the object alive (with DeletionTimestamp set) until it is processed, so the acquiring instance sees it in the resync List as a modification and runs finalization.
  • Losing responsibility needs no action: pending entries are skipped by the per-entry check. A reconciliation already in flight is not cancelled - reconcilers must be idempotent, and stale writes are rejected by the API server via metadata.resourceVersion (HTTP 409).
  • Unlike Custom, Scoped keeps the full default pipeline: multiple controllers per entity, selectors, and shared watchers keep working.

Custom Leader Election​

The Custom leader election type is the escape hatch for owning the pipeline yourself. It is the right choice only when the decision hook of Scoped is not enough because you must replace watcher or consumer:

  • Own watcher mechanics: events from another source, custom re-list/backoff behavior, or custom deduplication instead of the built-in ResourceWatcher<TEntity> mechanics.
  • Leadership-driven watch lifecycle: Scoped keeps a full watch stream open on every instance and filters client-side; if watch streams must start and stop with responsibility (e.g. per-namespace watches bound to leases, to cut API-server event volume across many shards), the watcher itself has to implement it.
  • Own queue consumption: prioritization, custom parallelism or gating semantics, durable consumption.

For partitioning responsibility across instances, use Scoped instead - it keeps the entire default pipeline (multiple controllers, selectors, shared watchers) and only delegates the responsibility decision.

With Custom, the SDK registers no watcher and no queue consumer - with custom coordination it cannot know how leadership should drive either. AddController registers the reconciler and the entity queue, and you add your own watcher (e.g. derived from ResourceWatcher<TEntity>) plus a queue consumer (e.g. derived from EntityQueueBackgroundService<TEntity>) as hosted services. A watcher without a consumer would enqueue events that nothing ever drains, so the consumer is required.

Enabling WithRegistrationValidation() makes host startup fail fast with an InvalidRegistrationException if a piece is missing, instead of starting an operator that silently processes nothing.

builder.Services
    .AddKubernetesOperator(settings => settings
        .WithLeaderElection(LeaderElectionType.Custom)
        .WithRegistrationValidation())
    .AddController<V1DemoEntityController, V1DemoEntity>();

builder.Services
    .AddHostedService<MyCustomResourceWatcher<V1DemoEntity>>()
    .AddHostedService<EntityQueueBackgroundService<V1DemoEntity>>();

Best Practices​

  1. Start with Single for simple deployments
  2. Use Scoped to shard responsibility by namespace across instances
  3. Use Custom only when you need to own the whole watcher/queue pipeline
  4. Test failover scenarios to ensure seamless transitions
  5. Monitor leader election status in your logs
  6. Set appropriate lease durations based on your workload

Troubleshooting​

  • Verify RBAC permissions for lease resources
  • Check network connectivity between instances
  • Review lease duration settings
  • Monitor logs for leader election events
  • Ensure cluster time and local time are synchronized (time drift can cause lease issues)