Finalizers

Finalizers are a crucial mechanism in Kubernetes for ensuring proper cleanup of resources. They provide a way to guarantee that cleanup operations are completed before a resource is actually deleted from the cluster.

What are Finalizers?

Finalizers are markers on resources that prevent their deletion until certain conditions are met. They are used to:

• Ensure proper cleanup of dependent resources
• Prevent accidental deletion of critical resources
• Guarantee that cleanup operations are completed successfully

How Kubernetes Handles Finalizers

When a resource is marked for deletion:

1. Kubernetes adds a deletionTimestamp to the resource
2. The resource remains in the cluster until all finalizers are removed
3. Each finalizer must explicitly remove itself after completing its cleanup
4. Only when all finalizers are removed is the resource actually deleted

This mechanism guarantees that cleanup operations are:

• Guaranteed to run
• Executed in a controlled sequence
• Completed before resource deletion

Implementing Finalizers

To implement a finalizer, create a class that implements IEntityFinalizer<TEntity>:

public class DemoEntityFinalizer(
    ILogger<DemoEntityFinalizer> logger,
    IKubernetesClient client) : IEntityFinalizer<V1DemoEntity>
{
    public async Task<ReconciliationResult<V1DemoEntity>> FinalizeAsync(
        V1DemoEntity entity,
        CancellationToken cancellationToken)
    {
        logger.LogInformation("Finalizing entity {Entity}", entity);

        try
        {
            // Clean up resources
            await CleanupResources(entity, cancellationToken);
            return ReconciliationResult<V1DemoEntity>.Success(entity);
        }
        catch (Exception ex)
        {
            logger.LogError(ex, "Error finalizing entity {Entity}", entity);
            // Return failure to prevent finalizer removal
            return ReconciliationResult<V1DemoEntity>.Failure(
                entity,
                $"Finalization failed: {ex.Message}",
                ex);
        }
    }
}

Using Finalizers

By default, KubeOps automatically attaches finalizers to entities during reconciliation. This behavior can be configured through OperatorSettings. See Advanced Configuration for details on controlling automatic finalizer attachment.

If you need manual control, you can use the EntityFinalizerAttacher delegate, which is injected into your controller:

[EntityRbac(typeof(V1DemoEntity), Verbs = RbacVerb.All)]
public class V1DemoEntityController(
    ILogger<V1DemoEntityController> logger,
    EntityFinalizerAttacher<DemoEntityFinalizer, V1DemoEntity> finalizer)
    : IEntityController<V1DemoEntity>
{
    public async Task<ReconciliationResult<V1DemoEntity>> ReconcileAsync(
        V1DemoEntity entity,
        CancellationToken cancellationToken)
    {
        // Attach the finalizer to the entity
        entity = await finalizer(entity, cancellationToken);

        // Continue with reconciliation logic
        logger.LogInformation("Reconciling entity {Entity}", entity);

        return ReconciliationResult<V1DemoEntity>.Success(entity);
    }

    public async Task<ReconciliationResult<V1DemoEntity>> DeletedAsync(
        V1DemoEntity entity,
        CancellationToken cancellationToken)
    {
        logger.LogInformation("Entity {Entity} was deleted", entity);
        return ReconciliationResult<V1DemoEntity>.Success(entity);
    }
}

Best Practices

1. Idempotency

• Make finalizer logic idempotent
• Handle cases where resources are already deleted
• Check resource existence before attempting cleanup

public async Task<ReconciliationResult<V1DemoEntity>> FinalizeAsync(
    V1DemoEntity entity,
    CancellationToken cancellationToken)
{
    // Check if resources still exist before cleanup
    var resources = await GetResources(entity, cancellationToken);
    if (!resources.Any())
    {
        // Resources already cleaned up
        return ReconciliationResult<V1DemoEntity>.Success(entity);
    }

    // Perform cleanup
    await CleanupResources(resources, cancellationToken);
    return ReconciliationResult<V1DemoEntity>.Success(entity);
}

2. Error Handling

• Use ReconciliationResult.Failure() for errors
• Return failure results to prevent finalizer removal
• Log errors with appropriate context
• Include retry delays when appropriate

public async Task<ReconciliationResult<V1DemoEntity>> FinalizeAsync(
    V1DemoEntity entity,
    CancellationToken cancellationToken)
{
    try
    {
        await FinalizeInternal(entity, cancellationToken);
        return ReconciliationResult<V1DemoEntity>.Success(entity);
    }
    catch (Exception ex)
    {
        logger.LogError(ex, "Error finalizing entity {Entity}", entity);
        // Return failure to prevent finalizer removal
        return ReconciliationResult<V1DemoEntity>.Failure(
            entity,
            $"Finalization failed: {ex.Message}",
            ex,
            requeueAfter: TimeSpan.FromMinutes(1));
    }
}

3. Resource Management

• Clean up all resources created by the entity
• Handle dependencies between resources
• Consider cleanup order (e.g., delete pods before services)

4. Finalizer Results

The finalizer must return a ReconciliationResult<TEntity>:

• Success: The finalizer is removed, and the entity is deleted
• Failure: The finalizer remains, and the entity is requeued for retry

public async Task<ReconciliationResult<V1DemoEntity>> FinalizeAsync(
    V1DemoEntity entity,
    CancellationToken cancellationToken)
{
    // Check if external resources exist
    var externalResource = await GetExternalResource(entity.Spec.ResourceId);

    if (externalResource == null)
    {
        // Already cleaned up
        return ReconciliationResult<V1DemoEntity>.Success(entity);
    }

    try
    {
        await DeleteExternalResource(externalResource, cancellationToken);
        return ReconciliationResult<V1DemoEntity>.Success(entity);
    }
    catch (Exception ex) when (IsRetryable(ex))
    {
        // Transient error - retry after delay
        return ReconciliationResult<V1DemoEntity>.Failure(
            entity,
            "Failed to delete external resource",
            ex,
            requeueAfter: TimeSpan.FromSeconds(30));
    }
    catch (Exception ex)
    {
        // Permanent error - log and succeed to prevent stuck resource
        logger.LogError(ex, "Permanent error during finalization, allowing deletion");
        return ReconciliationResult<V1DemoEntity>.Success(entity);
    }
}

Common Pitfalls

1. Stuck Resources

If a finalizer fails to complete:

• The resource will remain in the cluster
• It will be marked for deletion but never actually deleted
• The finalizer will be retried based on the requeueAfter value
• Manual intervention may be required for permanent failures

To fix stuck resources:

1. Identify the failing finalizer
2. Fix the underlying issue
3. Check if the finalizer is being retried
4. Manually remove the finalizer only if necessary:

   kubectl patch <resource> <name> -p '{"metadata":{"finalizers":[]}}' --type=merge

Manual Finalizer Removal

Only remove finalizers manually as a last resort. This can lead to orphaned resources and inconsistent cluster state.

2. Race Conditions

• Multiple finalizers running concurrently
• Resources being deleted by other controllers
• Network issues during cleanup

Solution: Implement proper error handling and retry logic.

3. Infinite Loops

• Finalizer logic that never completes
• Resources that can't be deleted
• External dependencies that are unavailable

Solution: Implement timeouts and fallback mechanisms.

When to Use Finalizers

Use finalizers when:

• Resources need guaranteed cleanup
• External systems need to be notified of deletion
• Dependencies need to be cleaned up in a specific order
• Resource deletion needs to be atomic

Don't use finalizers for:

• Simple logging or monitoring
• Non-critical cleanup tasks
• Operations that can be handled by the DeletedAsync method in controllers