Failure handling

Every grain call can fail because the caller, target silo, network, runtime, or a dependency failed. A timeout or connection failure tells the caller that no successful response arrived. It doesn’t prove that the grain method didn’t run.

Treat failed calls as unknown outcomes

Consider this sequence:

1. A grain commits state or invokes an external service.
2. The response is lost because the target silo or network fails.
3. The caller observes a timeout and retries.

The original operation might have completed. Retrying it as a new operation can duplicate a charge, reservation, message, or state transition. Orleans can reactivate the grain and reroute a later call, but it can’t infer the business outcome of the earlier call.

Classify operations:

• Read-only: Usually safe to retry when stale reads are acceptable.
• Naturally idempotent: Repeating the same desired-state assignment has the same effect.
• Deduplicated: The operation carries a stable ID and stores the result for replay.
• Non-idempotent: Repetition creates another effect and must not be retried automatically without a business reconciliation protocol.

Design idempotent operations

Prefer commands that describe a desired state or include an operation ID generated by the original caller. The grain can:

1. Check whether the operation ID was processed.
2. Return the stored result if it was.
3. Apply the state transition and record the operation ID and result atomically in grain state.
4. Persist before acknowledging success.

Bound the deduplication history by time or count only when the maximum retry and replay window is known. If an external system supports idempotency keys, pass the same operation ID through every retry.

Deduplication in grain state doesn’t make a separate external side effect atomic with that state. Use an outbox, inbox, process manager, transactional provider, or reconciliation workflow when one business operation spans stores.

Bound retries

Retry only when:

• The failure is plausibly transient.
• The operation is safe to repeat.
• The caller’s end-to-end deadline has time remaining.
• The retry budget and concurrency limit allow another attempt.

Use a small attempt limit, exponential backoff, and jitter. Honor cancellation and deadlines. Avoid retrying at every layer; multiplied retries can overload the cluster and its dependencies.

Don’t retry validation errors, authorization failures, incompatible payloads, or deterministic application exceptions. For sustained dependency failure, stop retries with a circuit breaker and return an explicit degraded or unavailable result.

Coordinate multi-step work

Calls across grains and external services aren’t automatically one transaction. A coordinator that crashes can leave some steps complete and others incomplete.

Choose an explicit consistency model:

• Orleans transactions where the selected storage providers and workload support them.
• A process manager or saga with durable progress and compensating actions.
• An outbox/inbox protocol for reliable message publication and consumption.
• Periodic reconciliation from authoritative business records.

Compensation is a business operation, not a memory rollback. It can also fail and must be idempotent.

Preserve evidence

Log and trace the stable operation ID, attempt number, deadline, grain type, and outcome category. Don’t use grain keys, tenant identifiers, or payload contents as unbounded metric dimensions.

When an outcome remains unknown, surface that state instead of reporting success or definite failure. Give operators or users a status lookup and reconciliation path.

Test failure behavior

Test failures before and after state persistence, lost responses, duplicate delivery, silo termination, provider timeout, and recovery after the retry window. Verify business invariants rather than only checking that the retry eventually returns.