Skip to content

Replay-safety rules for durable tasks

A durable task resumes after a deactivation or a failure by running its body again from the start, while each recorded operation returns its recorded outcome. The rules on this page keep that replay equivalent to the run which recorded the operations. The Microsoft.Orleans.Durable.Abstractions package carries analyzers which check them at build time and report diagnostics ORLEANS0200 to ORLEANS0223; each diagnostic’s page shows the code which triggers it and common fixes.

For the model behind the rules, see Replay and determinism and Idempotency in durable tasks.

Replayed code, recorded operations, and the tail

Section titled “Replayed code, recorded operations, and the tail”

The analyzers divide each durable task body into regions:

  • Recorded operations run once per execution, and a replay returns their outcomes: awaits of durable calls, local steps, DurableTask.Run delegates, delays, signal waits, combinators, and handles; DurableTask.RunInline effects; and Post. An await of a Task-returning helper in the same project which performs durable operations counts as one.
  • Replayed code is code from which a recorded operation is still reachable. A replay runs it again, after later operations committed their results and state changes.
  • The synchronous tail is code from which no recorded operation is reachable. It runs only in the run which completes the body, and its state changes commit with the task’s result.
  • Recorded bodies, the delegates which DurableTask.RunInline and DurableTask.Run receive and the bodies of local steps, run once per recorded result. A step’s body is itself a durable task body, with its own regions.

A DurableTask.RunInline effect runs immediately, in the current work item, before it returns; a DurableTask.Run step runs when it’s awaited, scheduled, or posted, possibly in a later turn, as a child execution. Consecutive effects in one synchronous block therefore commit in the same journal write with the state which they change, and no other turn of the grain runs between them, while each step has its own DurableTask.Current, cancellation scope, and journal write. An effect and a step with the same name share an identifier, so changing one into the other diverges for executions which recorded it; branch the change on DurableTask.GetVersion.

A replay which diverges from its record, by scheduling another operation under a recorded ID or by finding a recorded outcome which no longer decodes, doesn’t fail into the body: the runtime stalls the execution with DurableTaskAttemptCancellationReason.Diverged and keeps the task pending until code which matches the record replays it. See Diagnose replay divergence and Stalled executions.

The following grain shows each region:

[GenerateSerializer, Immutable]
public sealed record Ticket(int Number, Guid Reference, DateTimeOffset IssuedAt);
public interface ITicketGrain : IGrainWithStringKey
{
DurableTask<Ticket> IssueAsync(decimal amount, string cardLast4);
}
public sealed class TicketGrain : DurableGrain, ITicketGrain
{
private readonly ILogger _logger;
private readonly IDurableValue<int> _lastNumber;
private readonly IDurableDictionary<int, bool> _paid;
public TicketGrain(ILogger<TicketGrain> logger)
{
// Skips events while a task replays, so each is logged once.
_logger = logger.ReplaySafe();
_lastNumber = StateManager.GetOrAddValue<int>("last-number");
_paid = StateManager.GetOrAddDictionary<int, bool>("paid");
}
public async DurableTask<Ticket> IssueAsync(decimal amount, string cardLast4)
{
// Re-executed code: it runs in every replay, and logs once through the replay-safe logger.
_logger.LogInformation("Issuing a ticket for {Amount}.", amount);
// A recorded effect advances the counter, draws a reference, and reads the clock once. Its change to the counter
// commits with its recorded ticket, and a replay returns that ticket without running the effect again.
var ticket = DurableTask.RunInline(
"allocate",
() => new Ticket(++_lastNumber.Value, Guid.NewGuid(), DateTimeOffset.UtcNow));
var payment = await GrainFactory.GetGrain<IPaymentGrain>(ticket.Reference)
.AuthorizeAsync(amount, cardLast4)
.WithId("authorize");
// The synchronous tail: no durable operation follows this change, so it commits with the task's result.
_paid[ticket.Number] = payment.Succeeded;
return ticket;
}
}

A durable task body is an async method, local function, or lambda which returns DurableTask or DurableTask<TResult>, or a delegate which DurableTask.Run receives. Ordinary grain methods aren’t durable task bodies, and the rules don’t apply to them. The analysis is intraprocedural: it doesn’t inspect helpers which a body calls without awaiting, or helpers in other assemblies.

RuleFailure which it preventsDiagnostic
Read and change grain state in recorded effects, steps, or the tail.A replay applies a change again, or decides from state which later operations wrote.ORLEANS0200
Produce clock, random, and environment values in recorded effects, steps, or the tail, and read time from DurableTask.Current.UtcNow. Produce a value which the tail sends to an external service, such as an idempotency key, in a recorded effect or a step, because a tail whose result didn’t commit runs again.A replay computes other arguments, IDs, keys, or deadlines, and diverges or repeats an external effect.ORLEANS0201
Await external operations in steps or the tail.A replay repeats the operation, observes another result, or fails while the service is unavailable.ORLEANS0202
Change state after the last external await of a step or the tail.The change commits without the result, and the next run applies it again.ORLEANS0203
Return the outputs of a DurableTask.RunInline effect.A replay skips the effect, and the body’s variables keep their initial values.ORLEANS0204
Await, schedule, or post every durable task definition.The task never runs.ORLEANS0205
Run each async DurableTask definition once.The second run throws InvalidOperationException.ORLEANS0206
Name operations without $.The runtime rejects the name with ArgumentException.ORLEANS0207
Give each call site its own name.Removing or reordering a call site shifts the recorded IDs of the others.ORLEANS0208
Keep bodies on the activation’s scheduler.Work races with the grain, and RunInline and Post throw off the scheduler.ORLEANS0209
Run concurrent durable work through durable combinators.Children receive IDs in thread order, which a replay can change, and losing inputs keep running.ORLEANS0210
Wait with DurableTask.Delay and DelayUntil.A deactivation loses the wait, and a replay waits the whole duration again.ORLEANS0211
Filter cancellation handlers by DurableTask.Current.IsTaskCancellationRequested.The cancellation of an attempt, such as a deactivation, runs compensation for a task which nobody canceled.ORLEANS0212
Return a task from asynchronous code in bodies.Work continues after the body completed, and its failures escape.ORLEANS0213
Derive a signal name for each repeated wait.Every wait returns the first value which the signal received.ORLEANS0214
Log replayed code through logger.ReplaySafe().Each replay logs the events of the completed operations again.ORLEANS0215
Post from grains and durable task bodies, and schedule from clients.Post throws, because a client has no durable outbox.ORLEANS0216
Pass AttemptCancellationToken to the calls which task-cancellation cleanup makes.The cleanup’s calls end at once, because CancellationToken is already canceled.ORLEANS0217
Pass constant names and ranges to version checks.One change records several decisions, or a deployment records versions which no branch handles.ORLEANS0218
Await, combine, or return the handle of each scheduled child.The child’s failure goes unobserved, and deleting the execution’s tree cancels it.ORLEANS0219
Name operations in loops and conditions, and after other unnamed operations.A loop, a condition, or a deployment shifts generated identifiers, and a replay diverges.ORLEANS0220
Run synchronous work with DurableTask.RunInline.A step adds a child execution and a journal write without running asynchronous work.ORLEANS0221
Pass delegates which return a task to DurableTask.Run and DurableTask.RunInline.The task records an async void delegate as complete at its first await, before its work runs.ORLEANS0222
Assign new immutable values to grain state.A change in place bypasses the journal, so the recovered state differs from the state in memory.ORLEANS0223

The first four rules keep each side effect in exactly one run. Grain state holds every committed change, so replayed code reads newer state and applies its writes again. A DurableTask.RunInline effect, a step’s final code, and the tail each commit their changes with a record, so no replay repeats them. External calls run in steps, whose results the replay returns, and they send an idempotency key derived from the task, because a step whose result didn’t commit runs again; see Integrate durable tasks with other systems.

A durable task definition is cold and, for async DurableTask methods, single-use. Each recorded operation is identified by its name, or by its position among unnamed operations, so names stay unique per call site, without $, and stable across deployments. Task identity and references lists every identifier which the runtime generates and which changes affect it.

The activation’s scheduler orders a body’s operations, so the body stays on it and runs concurrent durable work through DurableTask.WhenAll, WhenAny, Race, ScheduleAll, and WhenEach, which identify inputs by index. Durable delays and timeouts record their due times in logical time. Structured concurrency describes how combinators settle their inputs.

Cancellation, signals, logging, and starts

Section titled “Cancellation, signals, logging, and starts”

Durable cancellation is recorded, and a host stop isn’t, so handlers tell them apart. Signals latch one value per name. Logs repeat in replayed code unless the logger skips replays. Post stages a start in the calling grain’s outbox, which clients don’t have.

These rules depend on data or deployment history which the analyzers can’t observe:

  • Schedule from ordered inputs. Build combinator inputs and loops which schedule durable work from arrays, lists, or sorted sequences. A HashSet<T> or a dictionary after removals can enumerate in another order in another process, which reassigns input indexes and occurrence numbers.
  • Decide from replay-stable configuration. Configuration which a replay reads, such as an option which sets a loop’s bound, stays the same while executions are pending. A recorded delay keeps its recorded due time when its configured duration changes.
  • Don’t branch on IsReplaying. DurableTask.Current.IsReplaying serves logging and diagnostics. A run which ended before its work committed is replayed, so code which runs only while IsReplaying is false can run twice or never.
  • Version by addition or by branch. Keep existing operations, names, and arguments while executions are pending: a replayed call matches its record by method and target, not arguments, and returns the outcome recorded for its original arguments. Add new operations with new names after existing ones, branch changes to operations which pending executions passed on DurableTask.GetVersion, and add new workflow shapes as new methods. See Version durable tasks.
  • Bound the history. Every recorded operation stays in the task’s tree until the tree is deleted. A workflow which loops indefinitely ends each generation and starts its successor as a detached task, such as subscription-{id}-{generation}.

The rules apply to every project which references the abstractions package. Set a rule’s severity in .editorconfig, for example to fail the build on replayed logs:

[*.cs]
dotnet_diagnostic.ORLEANS0215.severity = error

Each diagnostic’s page describes when suppressing it is appropriate.