Advanced Configuration

This guide covers advanced configuration options for KubeOps operators, including finalizer management, custom leader election, and durable requeue mechanisms.

Configuring the Operator

Pass a configuration callback to AddKubernetesOperator to customise the operator before it starts. Every option has a sensible default, so you only need to configure what you want to change:

builder.Services
    .AddKubernetesOperator(settings => settings
        .WithNamespace("my-namespace")
        .WithLeaderElection(LeaderElectionType.Single));

Each With* method corresponds to one of the options described in the sections below.

Finalizer Management

KubeOps provides automatic finalizer attachment and detachment to ensure proper resource cleanup. These features can be configured through OperatorSettings.

Auto-Attach Finalizers

By default, KubeOps automatically attaches finalizers to entities during reconciliation. This ensures that cleanup operations are performed before resources are deleted.

When AutoAttachFinalizers is enabled:

• Finalizers are automatically added to entities during reconciliation
• You don't need to manually call the EntityFinalizerAttacher delegate
• All registered finalizers for an entity type are automatically attached

When disabled:

builder.Services
    .AddKubernetesOperator(settings => settings.WithAutoAttachFinalizers(false));

You must manually attach finalizers in your controller:

public class V1DemoEntityController(
    ILogger<V1DemoEntityController> logger,
    EntityFinalizerAttacher<DemoFinalizer, V1DemoEntity> finalizerAttacher)
    : IEntityController<V1DemoEntity>
{
    public async Task<ReconciliationResult<V1DemoEntity>> ReconcileAsync(
        V1DemoEntity entity,
        CancellationToken cancellationToken)
    {
        // Manually attach finalizer
        entity = await finalizerAttacher(entity, cancellationToken);

        // Continue with reconciliation logic
        logger.LogInformation("Reconciling entity {Entity}", entity);
        return ReconciliationResult<V1DemoEntity>.Success(entity);
    }

    public Task<ReconciliationResult<V1DemoEntity>> DeletedAsync(
        V1DemoEntity entity,
        CancellationToken cancellationToken)
    {
        return Task.FromResult(ReconciliationResult<V1DemoEntity>.Success(entity));
    }
}

Auto-Detach Finalizers

KubeOps automatically removes finalizers after successful finalization. This can also be configured.

When AutoDetachFinalizers is enabled:

• Finalizers are automatically removed when FinalizeAsync returns success

When disabled:

builder.Services
    .AddKubernetesOperator(settings => settings.WithAutoDetachFinalizers(false));

You must manually manage finalizer removal, which is typically not recommended unless you have specific requirements.

Use Cases

Keep defaults enabled when:

• You want standard finalizer behavior
• Your finalizers follow the typical pattern
• You don't need fine-grained control

Disable auto-attach when:

• You need conditional finalizer attachment
• Different instances should have different finalizers
• You want to attach finalizers based on specific conditions

Disable auto-detach when:

• You need custom finalizer removal logic
• You want to coordinate multiple finalizers manually
• You have external systems that need to confirm cleanup

Custom Leader Election

KubeOps supports different leader election mechanisms through the LeaderElectionType setting. This allows you to control how multiple operator instances coordinate in a cluster.

Leader Election Types

public enum LeaderElectionType
{
    None = 0,    // No leader election - all instances process events
    Single = 1,  // Single leader election - only one instance processes events
    Custom = 2   // Custom implementation - user-defined coordination
}

Configuration

builder.Services
    .AddKubernetesOperator(settings => settings
        .WithLeaderElection(LeaderElectionType.Single)
        .WithLeaderElectionTimings(
            leaseDuration: TimeSpan.FromSeconds(15),
            renewDeadline: TimeSpan.FromSeconds(10),
            retryPeriod: TimeSpan.FromSeconds(2)));

Custom Leader Election

The Custom leader election type allows you to implement your own coordination logic, such as namespace-based leader election.

Example: Namespace-Based Leader Election

In some scenarios, you may want different operator instances to handle different namespaces. This enables horizontal scaling while maintaining isolation.

Step 1: Implement a custom ResourceWatcher

public sealed class NamespacedLeaderElectionResourceWatcher<TEntity>(
    ActivitySource activitySource,
    ILogger<NamespacedLeaderElectionResourceWatcher<TEntity>> logger,
    IFusionCacheProvider cacheProvider,
    ITimedEntityQueue<TEntity> entityQueue,
    OperatorSettings settings,
    IEntityLabelSelector<TEntity> labelSelector,
    IKubernetesClient client,
    INamespaceLeadershipManager namespaceLeadershipManager)
    : ResourceWatcher<TEntity>(
        activitySource,
        logger,
        cacheProvider,
        entityQueue,
        settings,
        labelSelector,
        client)
    where TEntity : IKubernetesObject<V1ObjectMeta>
{
    protected override async Task OnEventAsync(
        WatchEventType eventType,
        TEntity entity,
        CancellationToken cancellationToken)
    {
        // Check if this instance is responsible for the entity's namespace
        if (!await namespaceLeadershipManager.IsResponsibleForNamespace(
            entity.Namespace(),
            cancellationToken))
        {
            // Skip processing - another instance handles this namespace
            return;
        }

        // Process the event
        await base.OnEventAsync(eventType, entity, cancellationToken);
    }
}

Step 2: Implement the leadership manager

public interface INamespaceLeadershipManager
{
    Task<bool> IsResponsibleForNamespace(string @namespace, CancellationToken cancellationToken);
}

public class NamespaceLeadershipManager : INamespaceLeadershipManager
{
    private readonly ILeaderElector _leaderElector;
    private readonly ConcurrentDictionary<string, bool> _namespaceResponsibility = new();

    public async Task<bool> IsResponsibleForNamespace(
        string @namespace,
        CancellationToken cancellationToken)
    {
        // Implement your logic here:
        // - Consistent hashing of namespace names
        // - Lease-based namespace assignment
        // - External coordination service (e.g., etcd, Consul)

        return _namespaceResponsibility.GetOrAdd(
            @namespace,
            ns => CalculateResponsibility(ns));
    }

    private bool CalculateResponsibility(string @namespace)
    {
        // Example: Simple hash-based distribution
        var instanceId = Environment.GetEnvironmentVariable("POD_NAME") ?? "instance-0";
        var instanceCount = int.Parse(
            Environment.GetEnvironmentVariable("REPLICA_COUNT") ?? "1");

        var namespaceHash = @namespace.GetHashCode();
        var assignedInstance = Math.Abs(namespaceHash % instanceCount);
        var currentInstance = int.Parse(instanceId.Split('-').Last());

        return assignedInstance == currentInstance;
    }
}

Step 3: Register the custom watcher

builder.Services
    .AddKubernetesOperator(settings => settings
        .WithLeaderElection(LeaderElectionType.Custom))
    .AddSingleton<INamespaceLeadershipManager, NamespaceLeadershipManager>()
    .AddHostedService<NamespacedLeaderElectionResourceWatcher<V1DemoEntity>>();

Benefits of Custom Leader Election

• Horizontal Scaling: Multiple instances can process different subsets of resources
• Namespace Isolation: Different teams or environments can have dedicated operator instances
• Geographic Distribution: Route requests to instances in specific regions
• Load Balancing: Distribute work across multiple instances

Queue Strategy

KubeOps uses an internal queue to schedule and dispatch reconciliation work. The queue buffers events from the Kubernetes watch stream and feeds them to the reconciler at a controlled rate. The strategy is controlled via OperatorSettings.QueueStrategy.

InMemory (Default)

builder.Services
    .AddKubernetesOperator();
// QueueStrategy defaults to QueueStrategy.InMemory

With InMemory, KubeOps registers TimedEntityQueue<TEntity> as ITimedEntityQueue<TEntity> and starts EntityQueueBackgroundService<TEntity> as a hosted service. All scheduling state lives in the operator pod's heap.

• Advantages: Zero configuration. Minimal latency between watch event and reconciliation call.
• Disadvantages: Pending requeue timers are lost when the pod restarts. In practice this is rarely critical because the Kubernetes watch list provides a full snapshot of existing resources on startup and drives the operator back to convergence automatically.

Custom

Setting QueueStrategy.Custom instructs KubeOps to skip registration of ITimedEntityQueue<TEntity> and EntityQueueBackgroundService<TEntity>. You supply both implementations yourself.

builder.Services
    .AddKubernetesOperator(settings => settings.WithQueueStrategy(QueueStrategy.Custom));

// Register one ITimedEntityQueue<TEntity> per entity type
builder.Services.AddSingleton<ITimedEntityQueue<V1DemoEntity>, MyCustomEntityQueue<V1DemoEntity>>();

// Register one background service per entity type that drains the queue
builder.Services.AddHostedService<MyCustomQueueBackgroundService<V1DemoEntity>>();

note

Register one ITimedEntityQueue<TEntity> implementation and one corresponding background service for each entity type that should use the custom queue.

Use this strategy when you need scheduling behaviour that differs from the built-in timer-based queue — for example, priority queues or implementations that integrate with your own observability stack.

Reconcile Strategy

The reconcile strategy controls which watch events actually trigger a reconciliation cycle. It is set via OperatorSettings.ReconcileStrategy and defaults to ByGeneration.

Background: Generation vs. ResourceVersion

Every Kubernetes resource carries two version counters in metadata:

Field	Incremented by	When
metadata.generation	The API server	On spec changes; never on .metadata changes (labels, annotations); not on status changes when a status subresource is enabled
metadata.resourceVersion	The API server	On every successful write (spec, status, labels, annotations, finalizers, …)

Understanding this distinction is key to choosing the right strategy.

ByGeneration (Default)

builder.Services
    .AddKubernetesOperator(settings => settings
        .WithReconcileStrategy(ReconcileStrategy.ByGeneration));

The watcher caches the last seen metadata.generation for each resource. A watch event triggers reconciliation only when the incoming generation is greater than the cached one.

Consequences:

• Label or annotation changes do not trigger reconciliation (they are .metadata changes and never increment generation).
• Status updates do not trigger reconciliation when the CRD has a status subresource enabled.
• Finalizer additions/removals during deletion bypass the check because DeletionTimestamp is set — deletion does not increment generation, so bypassing is required to handle finalizers correctly.

This matches the pattern used by most Kubernetes controllers (e.g., kube-controller-manager) and is the correct default for the majority of operators.

When to use ByGeneration

Your controller only reacts to .spec changes. Status writes, label updates, and annotation changes are either performed by your own controller (and should not re-trigger it), or performed by other actors and are irrelevant to your business logic.

ByResourceVersion

builder.Services
    .AddKubernetesOperator(settings => settings
        .WithReconcileStrategy(ReconcileStrategy.ByResourceVersion));

The watcher caches the last seen metadata.resourceVersion (a string). A watch event triggers reconciliation whenever the incoming resourceVersion differs from the cached one — which is true for every successful API server write, regardless of which field changed.

Consequences:

• Status updates do trigger reconciliation.
• Label or annotation changes do trigger reconciliation.
• Finalizer removals during deletion trigger reconciliation naturally without any special bypass, because each finalizer removal is a real API write that increments resourceVersion.
• Your controller must be idempotent for status-only changes, since it will be called when another actor updates .status on the same resource.

Infinite reconcile loop risk

Because every write to the resource increments resourceVersion, a reconciler that writes back to the resource — for example to update .status — will trigger a new watch event, which triggers another reconcile, which writes again, and so on.

The official Kubernetes level-driven controller pattern addresses this: a controller must always work from current observed state, not from the event that triggered it. In practice: only write when the value actually changed.

// Compute the desired state from the entity
var desired = ComputeDesired(entity);

// Guard: only write when the resource actually needs to change.
// Any write increments resourceVersion, triggers a new watch event, and
// therefore a new reconcile — omitting this guard causes an infinite loop.
if (NeedsToBeUpdated(entity, desired))
{
    await ApplyUpdate(entity, desired, cancellationToken);
}

Without this guard, ByResourceVersion will produce an infinite reconcile loop for any controller that writes to the resource during reconciliation.

When to use ByResourceVersion

Your controller needs to react to changes outside .spec. Common examples:

• A controller that mirrors label or annotation values into child resources.
• A controller that acts on status conditions written by another controller.
• A controller that must respond to ownership or finalizer changes made by external actors.

Performance consideration

ByResourceVersion can significantly increase the number of reconciliation calls, especially for resources whose .status is updated frequently. Size MaxParallelReconciliations and the error backoff settings accordingly and ensure your controller logic is fast for status-only no-op cases.

Triggering Reconciliation from External Sources

The standard reconciliation loop is driven by the Kubernetes watch stream: every ADDED, MODIFIED, and DELETED event the API server emits flows through the watcher into the queue and eventually reaches your controller. This covers the vast majority of real-world use cases.

However, some architectures require a way for systems outside the operator to request reconciliation — for example:

• A CI/CD pipeline that just deployed a new image and wants the operator to react immediately.
• An event bus consumer that receives domain events and needs to reconcile the corresponding custom resource.
• An admin HTTP endpoint used during development or incident response.

KubeOps makes this straightforward by exposing the EntityQueue<TEntity> delegate. Inject it into any DI-registered component to enqueue an entity for reconciliation on demand.

// Signature (fire-and-forget; enqueue is synchronous)
public delegate void EntityQueue<in TEntity>(
    TEntity entity,
    ReconciliationType type,
    ReconciliationTriggerSource reconciliationTriggerSource,
    TimeSpan queueIn,
    int retryCount,
    CancellationToken cancellationToken)
    where TEntity : IKubernetesObject<V1ObjectMeta>;

Always pass ReconciliationTriggerSource.Operator for externally triggered reconciliations. ReconciliationTriggerSource.ApiServer is reserved for events originating from the Kubernetes watch stream.

HTTP Endpoint

An ASP.NET Core minimal API endpoint is the simplest way to expose an on-demand reconciliation trigger. This pattern is useful for admin tooling, CI/CD pipelines, and testing.

app.MapPost("/reconcile/{namespace}/{name}", async (
    string @namespace,
    string name,
    IKubernetesClient kubernetesClient,
    EntityQueue<V1DemoEntity> entityQueue,
    CancellationToken cancellationToken) =>
{
    var entity = await kubernetesClient.GetAsync<V1DemoEntity>(name, @namespace, cancellationToken);
    if (entity is null)
    {
        return Results.NotFound();
    }

    entityQueue(
        entity,
        ReconciliationType.Modified,
        ReconciliationTriggerSource.Operator,
        TimeSpan.Zero,
        retryCount: 0,
        cancellationToken);

    return Results.Accepted();
});

note

Secure this endpoint appropriately. In a Kubernetes-native deployment, consider restricting access via a NetworkPolicy or an Ingress authentication annotation rather than exposing it publicly.

Message Queue Consumer

For event-driven architectures, a background service can receive messages from an external message broker and enqueue the affected entity for reconciliation. The message bus acts as a notification channel — it signals that something has changed, but the operator always fetches the authoritative resource state from the Kubernetes API before reconciling.

public sealed class DemoEntityReconcileTriggerService(
    ServiceBusClient serviceBusClient,
    IKubernetesClient kubernetesClient,
    EntityQueue<V1DemoEntity> entityQueue,
    ILogger<DemoEntityReconcileTriggerService> logger)
    : BackgroundService
{
    private readonly ServiceBusProcessor _processor = serviceBusClient.CreateProcessor(
        "demo-entity-reconcile-triggers",
        new ServiceBusProcessorOptions { MaxConcurrentCalls = 1, AutoCompleteMessages = false });

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        _processor.ProcessMessageAsync += ProcessMessageAsync;
        _processor.ProcessErrorAsync += ProcessErrorAsync;

        await _processor.StartProcessingAsync(stoppingToken);
        await Task.Delay(Timeout.Infinite, stoppingToken).ConfigureAwait(ConfigureAwaitOptions.SuppressThrowing);
    }

    private async Task ProcessMessageAsync(ProcessMessageEventArgs args)
    {
        var trigger = args.Message.Body.ToObjectFromJson<ReconcileTrigger>();

        var entity = await kubernetesClient.GetAsync<V1DemoEntity>(
            trigger.Name,
            trigger.Namespace,
            args.CancellationToken);

        if (entity is null)
        {
            logger.LogWarning("Entity {Name}/{Namespace} no longer exists — skipping trigger.",
                trigger.Name, trigger.Namespace);
            await args.CompleteMessageAsync(args.Message, args.CancellationToken);
            return;
        }

        entityQueue(
            entity,
            ReconciliationType.Modified,
            ReconciliationTriggerSource.Operator,
            TimeSpan.Zero,
            retryCount: 0,
            args.CancellationToken);

        await args.CompleteMessageAsync(args.Message, args.CancellationToken);
    }

    private Task ProcessErrorAsync(ProcessErrorEventArgs args)
    {
        logger.LogError(args.Exception, "Error processing reconcile trigger.");
        return Task.CompletedTask;
    }

    public override async Task StopAsync(CancellationToken cancellationToken)
    {
        await _processor.StopProcessingAsync(cancellationToken);
        _processor.ProcessMessageAsync -= ProcessMessageAsync;
        _processor.ProcessErrorAsync -= ProcessErrorAsync;
        await _processor.DisposeAsync();
        await base.StopAsync(cancellationToken);
    }
}

public sealed record ReconcileTrigger(string Name, string Namespace);

Register the background service alongside your operator:

builder.Services
    .AddSingleton<ServiceBusClient>(_ =>
        new ServiceBusClient(configuration["ServiceBus:ConnectionString"]))
    .AddKubernetesOperator()
    .RegisterComponents();

builder.Services.AddHostedService<DemoEntityReconcileTriggerService>();

note

The message bus carries only a pointer to the resource (name and namespace), not a full resource snapshot. The operator always re-fetches the current state from the Kubernetes API. This keeps the operator's view of the world consistent with the API server — the single source of truth for all custom resource data.

Parallel Reconciliation

EntityQueueBackgroundService controls how many entities are reconciled simultaneously and what happens when two events for the same entity UID arrive at the same time. Both settings live on OperatorSettings.ParallelReconciliationOptions.

Maximum Concurrency

MaxParallelReconciliations sets the upper bound on concurrently running reconciliations across all entity types. The default is Environment.ProcessorCount * 2.

builder.Services
    .AddKubernetesOperator(settings => settings
        .WithParallelReconciliation(new() { MaxParallelReconciliations = 16 }));

A global SemaphoreSlim is acquired before an entry is dequeued, which means the queue acts as a natural buffer and back-pressure mechanism: producers (the watcher) can continue enqueuing while all reconciler slots are occupied.

Conflict Strategy

When a second reconciliation event arrives for an entity that is already being reconciled, the ConflictStrategy determines what happens:

Strategy	Behaviour	Use when
WaitForCompletion (default)	Blocks until the running reconciliation finishes, then processes the new request immediately	Ordering matters; no events should be dropped
Discard	Drops the incoming request	Reconciler always reads fresh state from the API; duplicate events are safe to ignore
RequeueAfterDelay	Places the incoming request back into the queue after RequeueDelay (default 5 s)	Events must not be dropped, but immediate blocking is undesirable

builder.Services
    .AddKubernetesOperator(settings => settings
        .WithParallelReconciliation(new()
        {
            MaxParallelReconciliations = 8,
            ConflictStrategy = ParallelReconciliationConflictStrategy.RequeueAfterDelay,
            RequeueDelay = TimeSpan.FromSeconds(3),
        }));

note

The per-UID lock is independent of the global semaphore. Even with MaxParallelReconciliations = 1, two events for different entities are never serialised by the UID lock; only events for the same UID are affected by ConflictStrategy.

Best Practices

Finalizer Management

1. Keep defaults enabled for most use cases
2. Monitor finalizer attachment in your logs
3. Test finalizer behavior in development before production
4. Handle finalizer failures gracefully with proper error messages

Leader Election

1. Start with Single for simple deployments
2. Use Custom only when you need advanced coordination
3. Test failover scenarios to ensure seamless transitions
4. Monitor leader election status in your logs
5. Set appropriate lease durations based on your workload

Parallel Reconciliation

1. Leave defaults for most operators — WaitForCompletion with ProcessorCount * 2 is a safe starting point
2. Use Discard only when your reconciler always re-fetches entity state from the API server
3. Use RequeueAfterDelay when events must not be dropped but you want to avoid blocking a worker slot
4. Raise MaxParallelReconciliations if reconciliation is I/O-bound and CPU is not a constraint
5. Lower MaxParallelReconciliations if the Kubernetes API server is rate-limiting your requests

Queue Strategy

1. Use InMemory for development and most production operators — the Kubernetes watch stream guarantees convergence on restart.
2. Use Custom only when you need scheduling behaviour that the built-in queue cannot provide.
3. Monitor consumer lag on any external trigger channel to detect processing issues early.
4. Always re-fetch entity state from the API server in external trigger consumers rather than deserialising resource state from the message payload.

Troubleshooting

Finalizers Not Attaching

• Check settings.AutoAttachFinalizers is true
• Verify finalizer is registered with AddFinalizer<TFinalizer, TEntity>()
• Check logs for finalizer attachment errors

Leader Election Issues

• Verify RBAC permissions for lease resources
• Check network connectivity between instances
• Review lease duration settings
• Monitor logs for leader election events
• Ensure cluster time and local time are synchronized (time drift can cause lease issues)

Queue Problems

• Verify the external queue connection when using QueueStrategy.Custom
• Check queue permissions and quotas when using an external message broker as a reconciliation trigger
• Monitor message processing errors in external trigger consumers
• Ensure entities still exist before enqueuing — the Kubernetes API is the authoritative source of truth
• If externally triggered reconciliations are not running, confirm the background service is registered and started correctly