Shut down Orleans silos

Orleans is an IHostedService inside the .NET Generic Host. When the host stops, it calls Orleans’ StopAsync method. Orleans then leaves the cluster, closes gateways and networking, deactivates grains, and stops providers in reverse lifecycle order.

Graceful silo shutdown

Build the host with CreateApplicationBuilder and run it with RunAsync or Run. Both Host.CreateApplicationBuilder and the older Host.CreateDefaultBuilder register IHostLifetime with the console lifetime by default, so no additional call is needed to observe termination requests:

public static async Task RunSilo(string[] args)
{
    var builder = Host.CreateApplicationBuilder(args);

    builder.UseOrleans(siloBuilder =>
    {
        // Configure Orleans.
    });

    await builder.Build().RunAsync();
}

The configured .NET host lifetime handles the relevant termination events and initiates the Generic Host shutdown sequence, so applications shouldn’t add a separate process-exit handler. For details, see .NET Generic Host shutdown.

In tests or embedded hosts, call StopAsync and dispose the host:

public static async Task StopHost(
    IHost host,
    CancellationToken cancellationToken)
{
    await host.StopAsync(cancellationToken);
    host.Dispose();
}

Don’t call Exit, kill the process from application code, or dispose Orleans services independently of the host.

Configure a shutdown budget

The host passes a cancellation token to every hosted service during shutdown, including Orleans. ShutdownTimeout cancels that token after the configured budget. Configure it for the expected workload:

public static void ConfigureShutdownTimeout(
    IHostApplicationBuilder builder)
{
    builder.Services.Configure<HostOptions>(options =>
    {
        options.ShutdownTimeout = TimeSpan.FromSeconds(45);
    });
}

Set the orchestrator’s termination grace period longer than the host shutdown timeout. Include time for:

Load balancers and readiness probes to stop sending new traffic.
Gateway and membership changes to propagate.
Grain deactivation callbacks and state writes.
Admitted client-observer invocations and their response handling.
Stream, reminder, storage, and telemetry providers to flush and stop.

If the external grace period expires first, the process is killed and graceful shutdown can’t complete.

Make application code shutdown-safe

Honor cancellation tokens in hosted services, startup tasks, lifecycle participants, and provider calls.
Keep OnDeactivateAsync bounded; persist important state during normal operation rather than relying only on shutdown.
Stop accepting new application work before the termination deadline.
Make recovery safe after abrupt termination, because crashes and node loss remain possible.
Avoid synchronous blocking and unbounded retries in shutdown callbacks.

Grains can move or reactivate elsewhere after a silo leaves. Don’t use graceful shutdown as an application-wide drain barrier unless the application separately coordinates that behavior.

Queued requests during shutdown

When a silo shuts down gracefully, an activation which still has queued requests after its running requests finish migrates to another silo, carrying those requests in its migration package. The destination admits them through the activation’s ordinary FIFO queue and overload checks. A new activation receives them before it registers in the grain directory in place of the old one; requests already queued at the destination retain their place. The calling silos then record the request’s new location, so their calls keep waiting for a response after the original silo leaves the cluster. Calls retain their original identities and response timeouts. Recording the new location is best-effort: a call whose caller doesn’t record it fails if the original silo leaves the cluster first. Every migration carries the activation’s queued requests in the same way, including migrations requested with MigrateOnIdle. The destination drops requests whose message time-to-live has expired. Expiry tracking is best-effort across transport and retries, so a delayed request can execute after its caller has timed out.

When the migration fails or its outcome is unknown, the original silo rejects those requests, so their callers fail with a rejection which they can retry. A rejected request may still run on the destination silo if the destination took it but its response was lost.

Requests which can’t migrate with their activation are forwarded instead. This includes requests for stateless workers and ImmovableAttribute grains, requests whose size bound together with the activation’s migration state exceeds MaxMessageBodySize, requests which arrive after the activation’s migration has begun, requests for an activation which was already deactivating for another reason when shutdown began, and requests migrating to a silo which runs an earlier version. Arbitrary request-context values use MaxMessageHeaderSize as their conservative header bound. Callers of forwarded requests fail with SiloUnavailableException if the original silo leaves the cluster before the response arrives, as they do when a silo crashes.

A migration counts as a forward. Requests which have already been forwarded MaxForwardCount times stay with the original activation and are rejected when it deactivates, so their callers fail with a rejection without the request executing.

Migration adds activation time on the destination silo to shutdown, in two phases. First, a migrating activation waits up to SystemResponseTimeout for the acceptance invocation, including asynchronous outgoing call filters and the destination’s response. If that invocation times out, the source rejects its requests. Second, after the destination takes the requests, the source retargets their callers’ pending calls. When DeactivationTimeout elapses, a queued migration is canceled and its requests are forwarded, and an accepted migration completes without waiting for the retargeting to finish. Allow the host shutdown budget enough time for both phases.

Containers and orchestrators

Remove the instance from application traffic before requesting host shutdown. Then send the platform’s normal termination request, allow the host shutdown budget to elapse, and reserve forceful termination for hung processes. A forced termination, power loss, or process crash can bypass the host entirely, so correctness must not depend on graceful shutdown.