Graceful shutdown and upgrades

Orleans tolerates abrupt process loss, but a graceful shutdown reduces failed calls and avoids unnecessary recovery work. Correctness must never depend on graceful shutdown because a process, host, or network can fail without warning.

Graceful shutdown and scale-in

For each instance being removed:

1. Stop admitting new application traffic and report not ready.
2. Stop application-specific background work from accepting new items.
3. Ask the .NET host to stop and await IHost.StopAsync or normal host termination.
4. Allow Orleans to leave cluster membership and deactivate or transfer runtime responsibilities.
5. Terminate the process only after the shutdown deadline expires.

The orchestrator termination grace period must exceed the .NET host shutdown timeout. Measure the actual duration under load and leave margin for provider latency.

Scale in gradually. Remove one failure domain at a time and wait until membership stabilizes, remaining silos absorb activations, and latency returns to normal. Don’t reduce the cluster below its tested redundancy or capacity floor.

See Shut down Orleans for host APIs.

Rolling upgrades

Use a rolling upgrade when old and new versions can safely share:

• Grain interfaces and serialized payloads.
• Persisted grain state.
• Reminder, stream, and provider schemas.
• External side effects and deduplication records.
• Cluster configuration and transport settings.

Before rollout:

1. Upgrade clients and silos in an order compatible with both versions.
2. Use additive contract changes first; don’t reuse or renumber serializer field IDs.
3. Make storage migrations backward-compatible and independently reversible.
4. Verify the minimum ready-silo count and surge capacity.
5. Test rollback while both versions and both state formats exist.

Replace a bounded number of silos at a time. Pause when error rate, latency, membership churn, dependency load, or activation failures exceed the rollout threshold.

Orleans grain versioning can route calls among compatible grain implementations, but it doesn’t make arbitrary application or storage changes compatible. See Deploy new versions of grains and Backward compatibility guidelines.

Uniform silo hash API compatibility

GetUniformHashCodes returns an ImmutableArray<T> which shares the silo’s cached hash storage. Rebuild callers compiled against the earlier uint[] return type. Use ToArray() when a mutable buffer is needed, or Sort() to obtain a sorted immutable result while preserving the cached values.

Distributed-directory upgrade compatibility

Rolling upgrades using the experimental distributed grain directory preserve both the partition count and the mapping from hash boundaries to partition numbers. Directory requests and ownership-transfer snapshots address those numbered partitions, so all silos must agree on their ranges.

Orleans 10.1 used 30 partitions per silo and assigned partition numbers after sorting each silo’s uniform hashes by unsigned value. Orleans 10.2 changed the default count to one and assigned multiple partitions in hash-generation order.

When upgrading a distributed-directory cluster running Orleans 10.1, configure newer silos with PartitionsPerSilo set to 30 and provide an application-defined GetPartitionBoundaries delegate which returns each silo’s uniform hashes in sorted order:

siloBuilder.Configure<GrainDirectoryOptions>(options =>
{
    options.PartitionsPerSilo = 30;
    options.GetPartitionBoundaries = static (silo, partitionCount) =>
        silo.GetUniformHashCodes(partitionCount).Sort();
});

Sorting the immutable hash array preserves the silo’s cached hash order and produces the existing partition identities. Retain this configuration after the upgrade and throughout rollback.

Clusters already using the Orleans 10.2-and-later generated-order mapping retain the default boundary function and their existing partition count. Qualify upgrades and rollback under sustained traffic, checking directory registration preservation and activation uniqueness. Changing an established cluster’s partition count or boundary mapping requires a coordinated full-cluster restart or a blue-green cutover with controlled grain-state ownership.

Blue-green upgrades

Use blue-green deployment when versions can’t safely coexist in one Orleans cluster.

• Give blue and green distinct ClusterId values.
• Decide whether they can share grain storage. Sharing is safe only when both versions can read and write the same records without concurrent ownership or incompatible side effects.
• Route external traffic at the application ingress, not by placing a load balancer in front of silo endpoints.
• Keep the previous environment available until state compatibility and rollback constraints permit removal.

For stateful cutovers, use one of these explicit strategies:

• Quiesce writes, migrate state, validate, then switch traffic.
• Dual-write through an application-owned migration protocol with reconciliation.
• Use separate state stores and perform an offline transfer.

Never point two incompatible active clusters at the same mutable grain state and assume Orleans membership will coordinate them. Membership is scoped by cluster ID; it doesn’t provide cross-cluster ownership.

Rollback

A rollback plan must cover code, configuration, schema, credentials, and persisted state. If the new version has written data the old version can’t read, reverting only the image isn’t a rollback.

Stop a rollout when safety thresholds are exceeded. Preserve logs and traces from failed instances, restore capacity with a known-compatible version, and avoid repeated automated retries that churn membership.