Finalizers

Finalizers are a crucial mechanism in Kubernetes for ensuring proper cleanup of resources. They provide a way to guarantee that cleanup operations are completed before a resource is actually deleted from the cluster.

What are Finalizers?

Finalizers are markers on resources that prevent their deletion until certain conditions are met. They are used to:

Ensure proper cleanup of dependent resources
Prevent accidental deletion of critical resources
Guarantee that cleanup operations are completed successfully

How Kubernetes Handles Finalizers

When a resource is marked for deletion:

Kubernetes adds a deletionTimestamp to the resource
The resource remains in the cluster until all finalizers are removed
Each finalizer must explicitly remove itself after completing its cleanup
Only when all finalizers are removed is the resource actually deleted

This mechanism guarantees that cleanup operations are:

Guaranteed to run
Executed in a controlled sequence
Completed before resource deletion

Implementing Finalizers

To implement a finalizer, create a class that implements IEntityFinalizer<TEntity>:

public class DemoEntityFinalizer(
    ILogger<DemoEntityFinalizer> logger,
    IKubernetesClient client) : IEntityFinalizer<V1DemoEntity>
{
    public async Task<ReconciliationResult<V1DemoEntity>> FinalizeAsync(
        V1DemoEntity entity,
        CancellationToken cancellationToken)
    {
        logger.LogInformation("Finalizing entity {Entity}", entity);

        try
        {
            // Clean up resources
            await CleanupResources(entity, cancellationToken);
            return ReconciliationResult<V1DemoEntity>.Success(entity);
        }
        catch (Exception ex)
        {
            logger.LogError(ex, "Error finalizing entity {Entity}", entity);
            // Return failure to prevent finalizer removal
            return ReconciliationResult<V1DemoEntity>.Failure(
                entity,
                $"Finalization failed: {ex.Message}",
                ex);
        }
    }
}

Using Finalizers

By default, KubeOps automatically attaches finalizers to entities during reconciliation. This behavior can be configured through OperatorSettings. See Advanced Configuration for details on controlling automatic finalizer attachment.

If you need manual control, you can use the EntityFinalizerAttacher delegate, which is injected into your controller:

[EntityRbac(typeof(V1DemoEntity), Verbs = RbacVerb.All)]
public class V1DemoEntityController(
    ILogger<V1DemoEntityController> logger,
    EntityFinalizerAttacher<DemoEntityFinalizer, V1DemoEntity> finalizer)
    : IEntityController<V1DemoEntity>
{
    public async Task<ReconciliationResult<V1DemoEntity>> ReconcileAsync(
        V1DemoEntity entity,
        CancellationToken cancellationToken)
    {
        // Attach the finalizer to the entity
        entity = await finalizer(entity, cancellationToken);

        // Continue with reconciliation logic
        logger.LogInformation("Reconciling entity {Entity}", entity);

        return ReconciliationResult<V1DemoEntity>.Success(entity);
    }

    public async Task<ReconciliationResult<V1DemoEntity>> DeletedAsync(
        V1DemoEntity entity,
        CancellationToken cancellationToken)
    {
        logger.LogInformation("Entity {Entity} was deleted", entity);
        return ReconciliationResult<V1DemoEntity>.Success(entity);
    }
}

Best Practices

1. Idempotency

Make finalizer logic idempotent
Handle cases where resources are already deleted
Check resource existence before attempting cleanup

public async Task<ReconciliationResult<V1DemoEntity>> FinalizeAsync(
    V1DemoEntity entity,
    CancellationToken cancellationToken)
{
    // Check if resources still exist before cleanup
    var resources = await GetResources(entity, cancellationToken);
    if (!resources.Any())
    {
        // Resources already cleaned up
        return ReconciliationResult<V1DemoEntity>.Success(entity);
    }

    // Perform cleanup
    await CleanupResources(resources, cancellationToken);
    return ReconciliationResult<V1DemoEntity>.Success(entity);
}

2. Error Handling

Use ReconciliationResult.Failure() for errors
Return failure results to prevent finalizer removal
Log errors with appropriate context
Include retry delays when appropriate

public async Task<ReconciliationResult<V1DemoEntity>> FinalizeAsync(
    V1DemoEntity entity,
    CancellationToken cancellationToken)
{
    try
    {
        await FinalizeInternal(entity, cancellationToken);
        return ReconciliationResult<V1DemoEntity>.Success(entity);
    }
    catch (Exception ex)
    {
        logger.LogError(ex, "Error finalizing entity {Entity}", entity);
        // Return failure to prevent finalizer removal
        return ReconciliationResult<V1DemoEntity>.Failure(
            entity,
            $"Finalization failed: {ex.Message}",
            ex,
            requeueAfter: TimeSpan.FromMinutes(1));
    }
}

3. Resource Management

Clean up all resources created by the entity
Handle dependencies between resources
Consider cleanup order (e.g., delete pods before services)

4. Finalizer Results

The finalizer must return a ReconciliationResult<TEntity>:

Success: The finalizer is removed, and the entity is deleted
Failure: The finalizer remains, and the entity is requeued for retry

public async Task<ReconciliationResult<V1DemoEntity>> FinalizeAsync(
    V1DemoEntity entity,
    CancellationToken cancellationToken)
{
    // Check if external resources exist
    var externalResource = await GetExternalResource(entity.Spec.ResourceId);

    if (externalResource == null)
    {
        // Already cleaned up
        return ReconciliationResult<V1DemoEntity>.Success(entity);
    }

    try
    {
        await DeleteExternalResource(externalResource, cancellationToken);
        return ReconciliationResult<V1DemoEntity>.Success(entity);
    }
    catch (Exception ex) when (IsRetryable(ex))
    {
        // Transient error - retry after delay
        return ReconciliationResult<V1DemoEntity>.Failure(
            entity,
            "Failed to delete external resource",
            ex,
            requeueAfter: TimeSpan.FromSeconds(30));
    }
    catch (Exception ex)
    {
        // Permanent error - log and succeed to prevent stuck resource
        logger.LogError(ex, "Permanent error during finalization, allowing deletion");
        return ReconciliationResult<V1DemoEntity>.Success(entity);
    }
}

Common Pitfalls

1. Stuck Resources

If a finalizer fails to complete:

The resource will remain in the cluster
It will be marked for deletion but never actually deleted
The finalizer will be retried based on the requeueAfter value
Manual intervention may be required for permanent failures

To fix stuck resources:

Identify the failing finalizer
Fix the underlying issue
Check if the finalizer is being retried

Manually remove the finalizer only if necessary:

kubectl patch <resource> <name> -p '{"metadata":{"finalizers":[]}}' --type=merge

Manual Finalizer Removal

Only remove finalizers manually as a last resort. This can lead to orphaned resources and inconsistent cluster state.

2. Assuming Finalizers Run in Parallel

KubeOps executes finalizers sequentially, one at a time, in the order of the entity's finalizer list: only the first finalizer is invoked; once it succeeds and is removed, the resulting MODIFIED event triggers the next one. Don't build finalizers that wait on each other — the later one simply won't run until the earlier one has completed. DeletedAsync on the controller fires only after all finalizers are removed.

3. Blocking Deletion on an Unavailable Dependency

A finalizer that requires an external system to respond will keep the resource in Terminating for as long as that system is down. Decide per dependency what "cleanup impossible" means: fail and retry (default), or log and succeed when the external resource is already gone / the cleanup is not critical. Treat "not found" responses from external systems as success — that is the idempotency case from Best Practice 1.

When to Use Finalizers

Use finalizers when:

Resources need guaranteed cleanup
External systems need to be notified of deletion
Dependencies need to be cleaned up in a specific order
Resource deletion needs to be atomic

Don't use finalizers for:

Simple logging or monitoring
Non-critical cleanup tasks
Operations that can be handled by the DeletedAsync method in controllers

What are Finalizers?​

How Kubernetes Handles Finalizers​

Implementing Finalizers​

Using Finalizers​

Best Practices​

1. Idempotency​

2. Error Handling​

3. Resource Management​

4. Finalizer Results​

Common Pitfalls​

1. Stuck Resources​

2. Assuming Finalizers Run in Parallel​

3. Blocking Deletion on an Unavailable Dependency​

When to Use Finalizers​