Skip to content

Durable tasks

Orleans Durable Tasks runs grain methods which return DurableTask or DurableTask<T> as durable executions. A workflow is ordinary sequential C#: it awaits calls to other grains, local state transitions, delays, and external signals. Each step’s result is recorded in the grain’s journal, so the workflow survives activation loss and resumes where it stopped. Durable tasks run over Messaging, which carries invocations, results, cancellations, and signals between grains through each grain’s durable inbox and outbox, and over Durable Jobs, which wakes grains when delays are due and resumes executions after their grain deactivates.

This page describes the programming model and its guarantees. Continue with:

  • Task identity and references for how task IDs are constructed, which changes keep them stable, and how to address a task from elsewhere.
  • Structured concurrency for task trees, ownership, lifetimes, and how cancellation flows through them.
  • Idempotency for the effects which commit exactly once and the external effects which need idempotency keys.
  • Replay-safety rules for the rules which the durable task analyzers enforce.
  • Versioning for changing workflow code while executions are pending, and retiring old versions.
  • Integrate with other systems for payment providers, databases, HTTP services, webhooks, and where to retry.
  • Practical recipes for fan-out, races, compensation, signals, detached work, cancellation, and scheduling workflows from other grains.
  • Operations and troubleshooting for configuration, retention, diagnostics, replay divergence, workflow versioning, and capacity.

Both models commit business state, received messages, and outgoing messages in the grain’s journal, and they share one inbox and outbox per grain.

Use durable tasks forUse message handlers for
Multi-step workflows written as sequential code, with results flowing between steps.Reacting to individual commands and events, one handler call per message.
Fan-out, races, and completion-order processing over durable calls.Application-defined message identities, subjects, headers, and raw payloads.
Durable timers, deadlines, and waits for external events inside a workflow.Scheduled messages and state machines which the application models explicitly.
Cooperative cancellation with inline compensation and recorded reasons.Business replies and rejections exchanged as typed messages.

A grain interface declares a durable method by returning DurableTask or DurableTask<TResult>. Arguments and results cross grains and persist in task state, so their types need Orleans serializers:

public enum OrderStatus
{
None,
Submitted,
AwaitingValidation,
Paid,
Shipped,
Cancelled,
}
[GenerateSerializer, Immutable]
public sealed record OrderLine(int ProductId, decimal UnitPrice, int Units);
[GenerateSerializer, Immutable]
public sealed record SubmitOrder(string BuyerId, OrderLine[] Lines, string CardLast4);
[GenerateSerializer, Immutable]
public sealed record PaymentResult(bool Succeeded, string? Reason);
[GenerateSerializer, Immutable]
public sealed record OrderResult(OrderStatus Status, string? Description);
[GenerateSerializer, Immutable]
public sealed record Shipment(string Carrier, string TrackingNumber);
public interface ICatalogItemGrain : IGrainWithIntegerKey
{
Task RestockAsync(int units);
DurableTask<bool> ReserveAsync(Guid orderId, int units);
// Returns whether the order held a reservation, so releasing is idempotent.
DurableTask<bool> ReleaseAsync(Guid orderId);
}
public interface IPaymentGrain : IGrainWithGuidKey
{
DurableTask<PaymentResult> AuthorizeAsync(decimal amount, string cardLast4);
}
public interface IOrderGrain : IGrainWithGuidKey
{
DurableTask<OrderResult> PlaceOrderAsync(SubmitOrder order);
Task<bool> CancelAsync();
Task AbortAsync(string reason);
Task<bool> ShipAsync(Shipment shipment);
}

The implementing grain derives from DurableGrain. The grain’s durable state change and the task’s result commit in one journal write, and the caller receives the result once that write completes:

public sealed class CatalogItemGrain : DurableGrain, ICatalogItemGrain
{
private readonly IDurableValue<int> _available;
private readonly IDurableDictionary<Guid, int> _reservations;
public CatalogItemGrain()
{
_available = StateManager.GetOrAddValue<int>("available");
_reservations = StateManager.GetOrAddDictionary<Guid, int>("reservations");
}
public async Task RestockAsync(int units)
{
_available.Value += units;
await WriteStateAsync();
}
// The state change and the task's result commit in one journal write.
public async DurableTask<bool> ReserveAsync(Guid orderId, int units)
{
if (units <= 0 || _available.Value < units)
{
return false;
}
_available.Value -= units;
_reservations[orderId] = units;
return true;
}
public async DurableTask<bool> ReleaseAsync(Guid orderId)
{
if (!_reservations.Remove(orderId, out var units))
{
return false;
}
_available.Value += units;
return true;
}
}

An async DurableTask method body is a deferred definition: calling the method creates the definition, and the durable task runtime runs it under the task’s identity. A body which completes without awaiting is a normal step.

Inside a durable task, awaiting a durable call runs it as a child task of the running execution. From a client or an ordinary grain method, awaiting it or ScheduleAsync starts it as a root task. Starting and observing tasks describes every way to start a task.

The following grain runs an order as one durable task. It records each status change as a recorded effect, waits out a grace period, reserves every line concurrently, authorizes payment, and waits for a shipment signal. Ordinary grain methods cancel the workflow with a reason, abort it, and deliver the shipment. When the workflow is canceled, a line is out of stock, or payment is declined, it releases the reserved stock inline:

public sealed class OrderGrain : DurableGrain, IOrderGrain
{
private static readonly DurableSignal<Shipment> Shipped = new("shipped");
private readonly IDurableTaskManager _tasks;
private readonly IDurableValue<OrderStatus> _status;
public OrderGrain(IDurableTaskManager tasks)
{
_tasks = tasks;
_status = StateManager.GetOrAddValue<OrderStatus>("status");
}
private Guid OrderId => this.GetPrimaryKey();
public static string PlaceOrderTaskId(Guid orderId) => $"place-order-{orderId:N}";
public async DurableTask<OrderResult> PlaceOrderAsync(SubmitOrder order)
{
DurableTask.RunInline("submit", () => SetStatus(OrderStatus.Submitted));
bool[] reserved;
try
{
await DurableTask.Delay(TimeSpan.FromMinutes(1)).WithId("grace-period");
DurableTask.RunInline("validate", () => SetStatus(OrderStatus.AwaitingValidation));
// Reserves every line concurrently, as the children reserve/0, reserve/1, and so on.
reserved = await order.Lines.Select(line => Catalog(line.ProductId).ReserveAsync(OrderId, line.Units))
.WhenAll()
.WithId("reserve");
}
catch (OperationCanceledException) when (DurableTask.Current.IsTaskCancellationRequested)
{
// WhenAll has waited for every reservation, so none is still running.
return await CancelOrderAsync(order.Lines, DurableTask.Current.TaskCancellationReason);
}
if (Array.IndexOf(reserved, false) >= 0)
{
return await CancelOrderAsync(order.Lines, "A line is out of stock.");
}
var payment = await GrainFactory.GetGrain<IPaymentGrain>(OrderId)
.AuthorizeAsync(order.Lines.Sum(line => line.UnitPrice * line.Units), order.CardLast4)
.WithId("authorize-payment");
if (!payment.Succeeded)
{
return await CancelOrderAsync(order.Lines, payment.Reason);
}
DurableTask.RunInline("pay", () => SetStatus(OrderStatus.Paid));
// Returns at once when the shipment arrived before the wait.
var shipment = await Shipped.WaitAsync();
DurableTask.RunInline("ship", () => SetStatus(OrderStatus.Shipped));
return new OrderResult(OrderStatus.Shipped, shipment.TrackingNumber);
}
public async Task<bool> CancelAsync()
{
if (_status.Value is not (OrderStatus.Submitted or OrderStatus.AwaitingValidation))
{
return false;
}
await _tasks.CancelAsync(PlaceOrderTaskId(OrderId), "Cancelled by the buyer.");
return true;
}
public async Task AbortAsync(string reason) => await _tasks.AbortAsync(PlaceOrderTaskId(OrderId), reason);
public async Task<bool> ShipAsync(Shipment shipment)
{
if (_status.Value is not OrderStatus.Paid)
{
return false;
}
var result = await _tasks.TrySignalAsync(PlaceOrderTaskId(OrderId), Shipped, shipment);
return result is DurableSignalResult.Accepted or DurableSignalResult.Duplicate;
}
private async Task<OrderResult> CancelOrderAsync(OrderLine[] lines, string? reason)
{
// The release is shielded, so it runs to completion after the cancellation which ended the workflow.
await lines.Select(line => Catalog(line.ProductId).ReleaseAsync(OrderId))
.WhenAll()
.WithId("release")
.Shielded();
DurableTask.RunInline("cancel", () => SetStatus(OrderStatus.Cancelled));
return new OrderResult(OrderStatus.Cancelled, reason);
}
// The workflow runs each status change as a recorded effect, such as DurableTask.RunInline("pay", ...): the state change
// commits with its recorded result, and a replay returns the result without changing the state again.
private OrderStatus SetStatus(OrderStatus status)
{
_status.Value = status;
return status;
}
private ICatalogItemGrain Catalog(int productId) => GrainFactory.GetGrain<ICatalogItemGrain>(productId);
}

A basket grain starts the workflow under the root ID place-order-{orderId}. After a deactivation, the next activation replays PlaceOrderAsync from the start: submit, grace-period, validate, and the reserve inputs return their recorded results, and the workflow continues with the first operation which hasn’t completed. The playground/DurableOrderProcessing project in the Orleans repository runs a complete version of this workflow with a browser UI which places orders through durable HTTP operations, detached projection updates, and diagnostics endpoints.

A durable task is identified by its grain and its TaskId. Task IDs are hierarchical: a root ID, which the starting caller chooses, followed by one segment per level of nesting. Task identity and references lists every kind of ID and its stability. The order workflow above produces this tree:

place-order-{orderId}
place-order-{orderId}/submit
place-order-{orderId}/grace-period
place-order-{orderId}/validate
place-order-{orderId}/reserve/0
place-order-{orderId}/reserve/1
place-order-{orderId}/authorize-payment
place-order-{orderId}/pay
place-order-{orderId}/$signal:shipped
place-order-{orderId}/ship
  • WithId names a child with a stable segment. A name which one run repeats, for example in a loop, identifies its later occurrences by their order: line, line$2, line$3. A child awaited without a name receives a generated segment, which begins with $ and follows the order in which the execution creates unnamed children. Names don’t contain $.
  • A named combinator gives its inputs IDs beneath its own: DurableTask.WhenAll(...).WithId("reserve") identifies its inputs as reserve/0, reserve/1, and so on, in input order.
  • A call to another grain runs on that grain under the caller’s scope, $caller/{callerGrainId}/{incarnation}/{taskId}, where {incarnation} identifies the caller’s run: the order’s record of the call place-order-{orderId}/reserve/0 is the catalog grain’s task $caller/order/{orderId}/{incarnation}/place-order-{orderId}/reserve/0. Each caller has its own scope, so callers whose workflows share a root ID never share a callee’s task, and each run of a root ID has its own incarnation, so a run after the earlier one was deleted calls new tasks. A root task, such as a detached start, keeps its root ID on its grain, and the root ID $caller is reserved.
  • Scheduling a task ID again with the same method, from the same caller, reattaches to the existing task and its recorded result, whatever the arguments, so a retry whose arguments serialize differently reaches the task. Scheduling it with a different interface or method fails, so a task ID deduplicates schedules while the grain retains the task.

A durable task calls its own grain’s durable methods directly, as local steps, such as await RefreshPricesAsync().WithId("refresh"). A durable call through a reference to the task’s own grain is rejected.

Recorded effects, local steps, and calls to other grains

Section titled “Recorded effects, local steps, and calls to other grains”

A recorded effect, DurableTask.RunInline, runs synchronous code in the caller’s current work item before it returns, with no suspension and no child execution, and records its result, or the exception it threw, in the task’s tree under its name. The record commits in the same journal write as the journaled state which the effect changed, and a replay returns the recorded result without running the effect again. The order workflow changes its status this way:

// The workflow runs each status change as a recorded effect, such as DurableTask.RunInline("pay", ...): the state change
// commits with its recorded result, and a replay returns the result without changing the state again.
private OrderStatus SetStatus(OrderStatus status)
{
_status.Value = status;
return status;
}
  • An effect changes grain state exactly once, reads state which later operations change, and produces identifiers, random values, and wall-clock times which every replay observes.
  • The effect passes data to the workflow through its result: assignments to captured variables don’t happen in later runs.
  • A failure is recorded too, because journaled state has no rollback: changes made before the throw commit with the recorded exception. A result which doesn’t serialize is recorded as that failure. A run which ends before a journal write commits the record also loses the effect’s state changes, so the next run runs the effect again.
  • Consecutive effects in one synchronous block, and the journaled state which they change, commit in the same journal write, and no other turn of the grain runs between them. Returning from an effect doesn’t mean that its record has committed: it commits with the task’s next journal write. The effect runs after task cancellation too, so compensation records its effects.
  • A name identifies an effect as WithId identifies a step: pay, then pay$2 for a later occurrence in the same run.
  • DurableTask.RunInline(effect) omits the name, such as var reference = DurableTask.RunInline(Guid.NewGuid);. The effect takes the execution’s next generated child ID, such as $child-3, which unnamed effects share with unnamed children, so it finds its record when every run creates its unnamed operations in the same order. Name effects whose position can change between runs or deployments.
  • Effects are synchronous: asynchronous and durable effects don’t compile, and the error shows the replacement, await DurableTask.Run(async () => ...). An effect runs on the activation’s scheduler, so RunInline after ConfigureAwait(false) throws InvalidOperationException without running it.
  • The record applies only within a durable task. Outside one, such as in a helper which an ordinary grain method shares, the effect runs every time.

A local step is an async DurableTask method of the same grain, awaited as a child, for work which awaits. It records its result in the grain’s task state, and the record commits with its first durable write: its result, or a child, delay, or message which it schedules. Its result is staged in the work item which runs its final code, so the state changes which that code makes commit with the result. On replay, a step with a recorded result returns that result without running again.

A call to another grain is a durable invocation. The calling grain records the call in its task state and stages the invocation in its outbox; both commit with the execution’s next journal write. The callee’s inbox accepts the invocation directly into its task state, runs the method, and returns the result through the caller’s inbox, which records it in the caller’s task state. Messages use identities derived from the sender, receiver, task, and subject, so redeliveries and replayed sends are deduplicated.

DurableTask.Run creates an unstarted step, for work which isn’t a grain call, such as a call to an external service. The step runs its delegate when it’s awaited, scheduled, or posted, possibly in a later turn, as a child execution with its own DurableTask.Current and cancellation scope. Creating it starts nothing, and nothing runs on the thread pool. A replay returns the delegate’s recorded result, and runs a delegate whose result wasn’t recorded again, so the delegate’s external effect needs its own idempotency.

DurableTask.RunInline("hold-id", Guid.NewGuid) and await DurableTask.Run(async () => ...).WithId("hold-id") use the same identifier, {taskId}/hold-id, and both record their outcome once. They differ in when the code runs, and the other differences follow from that. “Inline” means what it means elsewhere in .NET, as in TaskScheduler.TryExecuteTaskInline: the code runs synchronously in the current work item instead of being scheduled.

  • An effect runs immediately, in the current work item, before RunInline returns. So consecutive effects commit together with the state they change, the effect observes the caller’s DurableTask.Current, and it runs after task cancellation.
  • A step runs when it’s awaited, scheduled, or posted, possibly in a later turn, as a child execution. So it has its own DurableTask.Current, a step which isn’t shielded completes as canceled when its parent’s cancellation was requested before it starts, and its record commits in a journal write of its own.
public interface IHoldGrain : IGrainWithStringKey
{
DurableTask<Guid> HoldAsync(int quantity);
DurableTask<Guid> ConfirmAsync(Guid holdId);
}
public sealed class HoldGrain : DurableGrain, IHoldGrain
{
private readonly IDurableValue<int> _held;
private readonly IDurableValue<Guid> _holdId;
public HoldGrain()
{
_held = StateManager.GetOrAddValue<int>("held");
_holdId = StateManager.GetOrAddValue<Guid>("hold-id");
}
public async DurableTask<Guid> HoldAsync(int quantity)
{
// Each effect runs in this work item before it returns, and records under {taskId}/hold-id and {taskId}/hold.
// Their records and their state changes commit in the same journal write, and no other turn runs between them.
var holdId = DurableTask.RunInline("hold-id", Guid.NewGuid);
DurableTask.RunInline("hold", () =>
{
_held.Value += quantity;
_holdId.Value = holdId;
});
await DurableTask.Delay(TimeSpan.FromMinutes(15)).WithId("expiry");
return holdId;
}
public async DurableTask<Guid> ConfirmAsync(Guid holdId)
{
// The step runs when it's awaited, as a child execution with its own context, and records under
// {taskId}/confirmation-id, the identifier which DurableTask.RunInline("confirmation-id", ...) would use.
return await DurableTask.Run(async () => await RequestConfirmationAsync(holdId)).WithId("confirmation-id");
}
private static Task<Guid> RequestConfirmationAsync(Guid holdId) => Task.FromResult(holdId);
}

A replay which finds one kind recorded where the body now runs the other diverges instead of running the work again: its execution stalls, and ExecutionStalled names both kinds. A deployment which changes an effect into a step, or a step into an effect, branches on DurableTask.GetVersion so that executions which recorded the old kind keep it.

A workflow publishes progress to observers through a journaled IDurableLog<T> of its grain. Observers read the log with ReadAsync, typically through an IAsyncEnumerable<T> grain method. Observe journaled state describes reading, resuming, and trimming logs.

To publish an entry, append it in a recorded effect, DurableTask.RunInline(() => log.Append(item)). The effect appends once for its position in the task, and the entry commits together with the effect’s record in the task’s next durable write. A replay returns the recorded sequence number without appending again, so a reader in the committed mode receives each entry once, in the order in which the task appended it. Compute each entry from the task’s arguments, recorded results, and logical time, so each replay makes the same appends at the same positions. Entries suit progress which callers and user interfaces watch, such as each status of an order or each completed partition of an export:

[GenerateSerializer, Immutable]
public sealed record ExportProgress(int Completed, int Total);
public interface IPartitionGrain : IGrainWithStringKey
{
DurableTask ExportAsync();
}
public interface IExportGrain : IGrainWithStringKey
{
DurableTask<int> ExportAsync(string[] partitions);
IAsyncEnumerable<ExportProgress> WatchAsync(CancellationToken cancellationToken);
}
public sealed class ExportGrain : DurableGrain, IExportGrain
{
private readonly IDurableLog<ExportProgress> _progress;
public ExportGrain() => _progress = StateManager.GetOrAddLog<ExportProgress>("progress");
public async DurableTask<int> ExportAsync(string[] partitions)
{
// Each append runs once, and a replay returns its recorded sequence, so it doesn't publish older progress again.
// The loop's appends share a name, which identifies them as progress, progress$2, and so on.
DurableTask.RunInline(() => _progress.Append(new ExportProgress(0, partitions.Length)));
for (var i = 0; i < partitions.Length; i++)
{
await GrainFactory.GetGrain<IPartitionGrain>(partitions[i]).ExportAsync().WithId("partition");
DurableTask.RunInline("progress", () => _progress.Append(new ExportProgress(i + 1, partitions.Length)));
}
return partitions.Length;
}
// Observers read committed entries in order, from the first.
public async IAsyncEnumerable<ExportProgress> WatchAsync([EnumeratorCancellation] CancellationToken cancellationToken)
{
await foreach (var change in _progress.ReadAsync(cancellationToken: cancellationToken))
{
if (change is { Kind: LogEventKind.Entry, Item: { } progress })
{
yield return progress;
if (progress.Completed == progress.Total)
{
yield break;
}
}
}
}
}

The playground/DurableOrderProcessing project keeps each order’s state as a log of snapshots which its workflow appends this way, and streams them to the browser as server-sent events.

To stream content as it’s produced, such as a model’s response, run the stream in a local step. The step opens a draft with BeginDraft, writes each item to it, and commits the final content in its final synchronous code, after its last await:

public interface IChatModel
{
IAsyncEnumerable<string> StreamAsync(string prompt, CancellationToken cancellationToken);
}
public interface IConversationGrain : IGrainWithStringKey
{
DurableTask<long> AskAsync(string prompt);
IAsyncEnumerable<LogEvent<string>> WatchAsync(long fromSequence, long epoch, CancellationToken cancellationToken);
}
public sealed class ConversationGrain : DurableGrain, IConversationGrain
{
private readonly IChatModel _model;
private readonly IDurableLog<string> _transcript;
public ConversationGrain(IChatModel model)
{
_model = model;
_transcript = StateManager.GetOrAddLog<string>("transcript");
}
// Returns the sequence number of the committed response.
public async DurableTask<long> AskAsync(string prompt) => await DurableTask.Run(async cancellationToken =>
{
using var draft = _transcript.BeginDraft();
var response = new StringBuilder();
await foreach (var chunk in _model.StreamAsync(prompt, cancellationToken))
{
// Volatile: watchers receive each chunk as it arrives, and no journal write includes it.
draft.Write(chunk);
response.Append(chunk);
}
// The step's final synchronous code: the entry commits in the same journal write as the step's result.
return draft.Commit(response.ToString());
}).WithId("respond");
// Watchers receive the draft's chunks live, then the committed entry which replaces them.
public IAsyncEnumerable<LogEvent<string>> WatchAsync(long fromSequence, long epoch, CancellationToken cancellationToken) =>
_transcript.ReadAsync(fromSequence, epoch, includeDrafts: true, cancellationToken: cancellationToken);
}
  • A draft’s items are volatile. Readers which pass includeDrafts: true to ReadAsync receive each item as the step writes it, and the content becomes durable when Commit appends the entries which replace the draft.
  • The step’s result is staged in the work item which runs its final synchronous code, so the entries which Commit appends and the step’s recorded result commit in the same journal write. Readers in the committed mode receive the entries once that write is durable.
  • The content therefore commits exactly once: every committed journal holds both the entries and the step’s record, or neither. A replay after that write returns the recorded result without running the step. A run which ends before that write loses both, so the next run runs the step, which streams and commits the content.
  • A step which throws, including after cancellation, disposes the draft, which abandons it, and the step’s failure is recorded, so a replay rethrows it without streaming again.
  • Journaled changes inside a step belong in its final synchronous code. A change which the step makes before an await, such as an Append, can commit with a journal write which other grain work requests before the step completes, and a replay which runs the step again makes the change again.

After a deactivation or a failure, the next activation replays each pending task: the method body runs again from the start, and every recorded operation returns its recorded outcome. Replay starts once the grain’s OnActivateAsync has completed, and a deactivation cancels the attempts of running tasks before the grain’s OnDeactivateAsync runs, so their later durable operations throw; see Task cancellation and attempt cancellation. A workflow resumes correctly when each replay schedules the same operations, with the same IDs and arguments, in the same order.

Replay doesn’t rebuild the grain’s journaled state, which holds the latest committed changes, including changes which later operations made. A durable body therefore has three kinds of code:

  • Recorded operations run once per execution, and a replay returns their recorded outcomes: durable calls to other grains, local steps, DurableTask.Run delegates, delays, signal waits and sends, detached and posted starts, and DurableTask.RunInline effects.
  • Re-executed code precedes the body’s last recorded operation and runs again in every replay. It computes values from the method’s arguments, recorded results, the execution’s logical time and cancellation state, and configuration which stays the same while executions are pending.
  • The synchronous tail follows the body’s last recorded operation, with no other await before the body returns. It runs only in a run which completes the body, and its state changes commit in the same journal write as the task’s result, whichever operation requests that write, so no replay runs it again.

The following grain allocates a ticket number in an effect, authorizes a payment, and records the outcome in its synchronous tail:

[GenerateSerializer, Immutable]
public sealed record Ticket(int Number, Guid Reference, DateTimeOffset IssuedAt);
public interface ITicketGrain : IGrainWithStringKey
{
DurableTask<Ticket> IssueAsync(decimal amount, string cardLast4);
}
public sealed class TicketGrain : DurableGrain, ITicketGrain
{
private readonly ILogger _logger;
private readonly IDurableValue<int> _lastNumber;
private readonly IDurableDictionary<int, bool> _paid;
public TicketGrain(ILogger<TicketGrain> logger)
{
// Skips events while a task replays, so each is logged once.
_logger = logger.ReplaySafe();
_lastNumber = StateManager.GetOrAddValue<int>("last-number");
_paid = StateManager.GetOrAddDictionary<int, bool>("paid");
}
public async DurableTask<Ticket> IssueAsync(decimal amount, string cardLast4)
{
// Re-executed code: it runs in every replay, and logs once through the replay-safe logger.
_logger.LogInformation("Issuing a ticket for {Amount}.", amount);
// A recorded effect advances the counter, draws a reference, and reads the clock once. Its change to the counter
// commits with its recorded ticket, and a replay returns that ticket without running the effect again.
var ticket = DurableTask.RunInline(
"allocate",
() => new Ticket(++_lastNumber.Value, Guid.NewGuid(), DateTimeOffset.UtcNow));
var payment = await GrainFactory.GetGrain<IPaymentGrain>(ticket.Reference)
.AuthorizeAsync(amount, cardLast4)
.WithId("authorize");
// The synchronous tail: no durable operation follows this change, so it commits with the task's result.
_paid[ticket.Number] = payment.Succeeded;
return ticket;
}
}

Follow these rules in re-executed code:

  • Put each side effect in a recorded operation or in the synchronous tail. Code between recorded operations computes values from the method’s arguments, earlier results, and other values which are the same on every replay.
  • Read and change grain state in recorded effects, steps, or the synchronous tail. Grain state holds every committed change, so code which replays reads the state which later operations wrote.
  • Read time from DurableTask.Current.UtcNow, the execution’s logical time, which DurableTask.Current exposes. The runtime persists it before the execution’s effects commit and advances it to the recorded completion time of each child result the execution observes, so a replay observes the same times and a delay starts when the work before it completed.
  • Produce random values and identifiers in recorded effects or children, and call external services in steps or DurableTask.Run delegates, whose external effects need their own idempotency.
  • Name children with WithId when they’re scheduled conditionally, in loops, or concurrently outside a combinator, and name combinators. Generated segments are stable only when unnamed children are created in the same order. Detached tasks and signal sends are recorded under IDs derived from their target and root ID or signal, so adding one doesn’t move others.
  • Log through a logger which logger.ReplaySafe() returns, so each event is logged once.

Replay-safety rules lists every rule, the failure which it prevents, and the analyzer diagnostic which reports it.

DurableTaskContext.IsReplaying reports whether a run is re-running operations which earlier runs recorded: it is true from the start of a run which finds recorded operations in the task’s tree, other than signal values, until the run has reached each of them or starts an operation which no earlier run recorded, and then stays false. Use it for logging and diagnostics only. A run which ended before a journal write committed its work is replayed, so code which runs only while IsReplaying is false can run twice or never, and code which branches on it schedules different operations in different runs.

A replay which schedules a different operation under a recorded ID, or which can’t decode a recorded outcome, diverges: the runtime stalls its execution, which stays pending until code which matches the record replays it, and raises ExecutionStalled, which names the recorded and the requested operations. See Diagnose replay divergence. To change a workflow while executions are pending, branch on DurableTask.GetVersion, which gives each execution the version of the change which it recorded, and version 0 to executions which earlier code ran past the change; see Version durable tasks.

Combinators schedule their inputs concurrently, so the runtime commits the batch of invocations together. Awaited, a combinator runs as part of the awaiting execution and records no result of its own; each input runs as a child identified by its input index under the combinator’s operation ID. Name the operation with WithId, so its inputs keep their IDs when a later deployment adds or removes earlier combinators.

Each combinator is also an extension method on a sequence of tasks, so a list, an array, or a query composes in place, as in lines.Select(line => Catalog(line.ProductId).ReserveAsync(OrderId, line.Units)).WhenAll().WithId("reserve"). Lists of handles compose the same way, for example handles.WhenAny(cancellationToken). A collection expression has no type of its own, so [a, b] goes to the static form: DurableTask.WhenAll([a, b]).

  • DurableTask.WhenAll follows Task.WhenAll. It completes after every input is terminal, including inputs which durable cancellation reaches. It then throws the first failure in input order; otherwise the first cancellation in input order; otherwise WhenAll<TResult> returns the results in input order. Input order selects the exception, so a replay observes the same one.
  • DurableTask.WhenAny returns the index of the input which completed first, and WhenAny<TResult> also returns its result. A winner which failed or was canceled fails the combinator. The other inputs keep running.
  • DurableTask.Race selects its winner like WhenAny and returns the same values, then requests cancellation of every other input with the reason Another task won the race. and waits until each is terminal, so no input outlives the race. A winner which failed or was canceled fails it; the other inputs’ outcomes don’t.
  • WaitAsync and WaitUntilAsync race a task, input 0, with a durable delay, input 1. When the task completes first, the combinator cancels the delay and returns the task’s outcome. When the delay elapses first, the task receives a cancellation request whose reason names the timeout, and the combinator waits for the task’s outcome: a task which ends canceled fails it with a TimeoutException, and any other outcome, such as a result which the task returns anyway, is its outcome.
  • DurableTask.ScheduleAll schedules every input and returns their ScheduledTask handles in input order, without waiting for results.
  • DurableTask.WhenAllCompleted waits until every input is terminal and returns the handles. An input’s failure or cancellation leaves the combinator successful, so the workflow can act on each outcome, for example by compensating exactly the inputs which succeeded.
  • DurableTask.WhenEach schedules its inputs and yields each one’s handle as an IAsyncEnumerable, in completion order. An input’s failure or cancellation is yielded like a success.

The grain stamps each child’s terminal result with a durable completion sequence, in the journal write which records the result. WhenAny, Race, and WhenEach select the remaining input with the lowest sequence, so every replay selects the same winner and yields the same order, including a replay in which more inputs have already completed. ScheduleAll, WhenAllCompleted, and WhenEach run only when awaited inside a durable execution, because their handles belong to that execution.

A ScheduledTask handle observes one task. Once a wait, a poll, or a combinator observes its terminal response, IsCompleted, Status, and Exception report it synchronously. GetResult returns the result or throws the recorded failure or cancellation, and TryGetResult returns true with the result of a task which completed successfully. Before the handle observes a terminal response, Status is Pending. Inside a durable task, a handle’s Id names the task’s operation, which combinators and replays address, and TargetTaskId names the task on the grain which stores it, which other parties address: a detached task’s root ID, the task which GetDurableTask observes, or a call’s task on its target. For existing handles, ScheduledTask.WhenAll, ScheduledTask.WhenAllCompleted, and ScheduledTask.WhenEach provide the same semantics, and ScheduledTask.Race returns the winning handle once every handle has observed its outcome. A handle’s WaitAsync and WaitUntilAsync with a timeout follow the combinators’ timeout rules; outside a durable task, such as in a client, they measure the timeout on the system clock. WhenEach accepts no WithId, so the named form schedules the inputs with ScheduleAll(...).WithId(name) and enumerates ScheduledTask.WhenEach(handles).

DurableTask.Delay and DurableTask.DelayUntil are durable timers measured in logical time. The runtime records the delay’s due time and sends the grain a scheduled resume message, which its inbox holds until the Durable Jobs clock reaches the due time. Logical time reads the Durable Jobs clock, so due times and the inbox share one time domain. A recorded due time is a commitment: a replay which requests another due time, for example after a configured duration changed, keeps the recorded one and logs a warning. Timeouts use these delays, so a timeout counts from the run which first awaited it.

Cancellation is cooperative, durable, and idempotent. A grain requests it for its own tasks with IDurableTaskManager.CancelAsync(rootId, reason), and clients and other callers request it through ScheduledTask.CancelAsync. A request for a task which the grain doesn’t retain throws DurableTaskNotFoundException and changes nothing. The request commits before it propagates through the task’s execution tree:

  • Delays, children whose bodies haven’t started, and new children which the canceled task schedules terminate as canceled without running.
  • Running children observe the request through DurableTaskContext.CancellationToken, and pending calls to other grains receive a cancellation which their callee applies to its own task.
  • The task’s body decides its outcome. Awaiting a canceled child throws OperationCanceledException. The task is canceled when that exception escapes the body; a body which catches it completes with its own result or failure.
  • Completed children keep their recorded results, and a replay of a canceled task observes the request at the same point as the run which it replays, so the body takes the same path. That point is the start of one of the body’s durable operations, such as a child’s start, a DurableTask.RunInline effect, or a wait for a task or a signal: a running body observes the request at its current operation, which for a body waiting durably is that wait, and the runtime records the point with the task. A replay keeps the durable token uncanceled until it reaches the recorded point. A replay of a run whose point didn’t commit observes the request where it reaches the last operation which the earlier run recorded, and records that point. A replay which ends before its recorded point stalls as diverged.

The first request’s reason is durable: DurableTaskContext.TaskCancellationReason returns it in the canceled execution, its descendants, and every replay, and the cancellation carries it to pending calls to other grains.

A body compensates inline by starting its compensation with Shielded. A shielded child, such as a local step, a delay, or a call to another grain, runs to completion although the task’s cancellation was requested, and a later request skips it, while the body still owns it and awaits its result. A shielded combinator shields each input which it starts. Every awaited operation honors the shield the same way, so a shielded signal wait, delay, GetDurableTask subscription, Race, WhenAny, or WaitAsync completes after the task’s cancellation. The cancellation of the attempt, by a deactivation, an abort, or the deletion of the task’s execution tree, still ends shielded operations.

An administrative abort, through IDurableTaskManager.AbortAsync(rootId, reason) or ScheduledTask.AbortAsync, commits the task and its execution tree as failed with DurableTaskAbortedException in one journal write, stops cancels their running attempts, and sends an abort to each pending call to another grain. A canceled attempt’s durable operations throw DurableTaskAttemptCanceledException, and IsTaskCancellationRequested stays false, so an abort ends a task without compensation. Use it for a task whose body ignores cancellation or whose cleanup is stuck.

Task cancellation and attempt cancellation differ in what they record; see Task cancellation and attempt cancellation.

Configuration on a durable call sets what the task is, and the call which starts it sets when the caller continues:

await callawait call.ScheduleAsync()call.Post()
Attached (default)Runs the child and returns its result once the result is durable.Returns the child’s handle once its grain has accepted and stored it.Stages the child with the execution’s next journal write and returns its handle.
.Shielded()As attached; the caller’s cancellation doesn’t reach the child.As attached; the caller’s cancellation doesn’t reach the child.As attached; the caller’s cancellation doesn’t reach the child.
.Detached()Starts an independent root task and returns its result.Returns the root task’s handle once its grain has accepted and stored it.Stages the root task with the caller’s next journal write and returns its handle.
  • An attached child belongs to the running execution’s tree: the execution owns it, its cancellation reaches the child, and deleting the tree deletes the child.
  • Detached starts an independent root task on its target, for follow-up work, cleanup which outlives the starter, and calls whose results the starter doesn’t need. The starter can complete while it runs, and the task keeps running through the starter’s cancellation, abort, and deletion. Its root ID is the WithId name, verbatim, so other parties address it by that name, or {grainId}/{taskId}:{segment}, where {grainId} is the starting grain and {segment} is the generated ID of the execution’s next unnamed operation, so workflows with the same task ID on different grains start different tasks. Starters which share a task name it. The start names the execution as a subscriber, so awaiting the task, or its handle, receives its result; canceling that wait ends only the subscription.
  • ScheduleAsync returns once the target grain has accepted and durably stored the task, so the task’s ID, the handle’s TargetTaskId, can be handed to other parties, which address it with GetDurableTask. The target acknowledges the invocation, and the caller’s outbox retires it, once the target’s journal write stores the task. A start which the target rejects, because the task ID already identifies another method, throws DurableTaskConflictException. Other failures to accept the start, such as a request which the target can’t decode, are retried, and a start which isn’t accepted before Messaging’s OutboxRetryTimeout throws DurableTaskDeliveryException. Awaiting the task throws the same failure. See Dead letters.
  • Post is synchronous. It stages the invocation in the caller’s outbox, which commits with the grain’s next journal write, together with the state changes which the calling code made, and delivers it once. The target holds the task once the invocation is delivered, so a task whose ID other parties look up starts with ScheduleAsync. Inside a durable task, the start is recorded like a DurableTask.RunInline effect, so a replay returns the same handle, and awaiting the handle later observes the result. A client has no outbox, so Post throws InvalidOperationException there, and ORLEANS0216 reports it.
  • Inside a durable task, the record of a detached task, $detach:{target}:{rootId}, commits with the start, so a replay which reaches the record starts nothing. The starter holds the invocation, which carries the request, until its target accepts it, so the start survives the deletion of the starter’s tree. Starting the same method under the same root ID again leaves the task unchanged.
  • Outside a durable task, every task is a root task, so .Detached() leaves the call unchanged. WithId supplies the root ID and makes a start safe to retry: a caller which retries a start after an ambiguous failure, such as a timeout, reaches the same task. A start without WithId receives a new random root ID, which the handle’s Id reports, and behaves like an ordinary call.

A checkout clears the basket and starts the order’s workflow in one journal write:

_lines.Clear();
// Posts the invocation to the outbox, and the journal write commits it with the cleared basket.
_ = GrainFactory.GetGrain<IOrderGrain>(orderId)
.PlaceOrderAsync(order)
.WithId(OrderGrain.PlaceOrderTaskId(orderId))
.Post();
await WriteStateAsync();

A DurableSignal<T> carries an external event, such as a shipment notice or an approval, into a running task. Senders and waiters share one descriptor, which names the signal:

  • WaitAsync waits for the signal’s value. TryWaitAsync races the wait with a deadline and returns (true, value) or (false, default), and the same outcome on every replay.
  • A signal addresses an invoked task: a task which a client, another grain, or a durable task which detached it started on the grain. Waits anywhere in that task’s execution tree observe its signals.
  • The task records the first value for each signal name and returns it to every wait for that name, including waits during replay. A signal which arrives before its wait is recorded, and the wait returns it at once. A repeated signal with an equal value leaves the task unchanged, and a different value is rejected. For derives a distinct name, such as approved/round-2, for each occurrence of a repeated event.
  • A grain signals its own tasks with IDurableTaskManager.SignalAsync or TrySignalAsync, which commit with the grain’s pending state changes. Clients and ordinary methods of other grains use the SignalDurableTaskAsync and TrySignalDurableTaskAsync grain-reference extensions, and durable tasks on other grains use SignalDurableTaskAsync, which records the send for replay under $send:{target}:{taskId}:{name}. Repeated sends of one signal in an execution, such as a retry after TaskNotFound, are numbered by occurrence, $send:...$2 and so on, so each sends again. The Try forms return a DurableSignalResult: Accepted, Duplicate, Conflict, TaskNotFound, or TaskCompleted.
  • A grain can commit an invocation on another grain and a signal for it in one journal write. The signal message can then arrive first; the receiving grain records its value, and the invocation observes it when it arrives.

The GetDurableTask<TResult>(rootId) grain-reference extension, or its TaskId overload, returns a DurableTaskReference<T> to an existing task, such as a client-scheduled operation or a detached task. A durable task awaits the reference to subscribe to the task’s result, and other code awaits it to wait for the result. The reference is awaitable through an extension method, so an ASP.NET Core Minimal API handler returns it as a value, and IDurableTaskReference reads its Target and Id without its result type. A webhook relay, for example, waits for an operation and then delivers its result:

[GenerateSerializer, Immutable]
public sealed record OperationResult(string Summary);
public interface IOperationGrain : IGrainWithGuidKey
{
DurableTask<OperationResult> RunAsync(string input);
}
public interface IWebhookRelayGrain : IGrainWithGuidKey
{
DurableTask DeliverWebhookAsync(Guid operationKey, string operationId, Uri callback);
}
public sealed class WebhookRelayGrain(HttpClient http) : DurableGrain, IWebhookRelayGrain
{
public async DurableTask DeliverWebhookAsync(Guid operationKey, string operationId, Uri callback)
{
var operation = GrainFactory.GetGrain<IOperationGrain>(operationKey);
try
{
// Subscribes to the operation's result; the subscription and its deadline are recorded for replay.
var result = await operation.GetDurableTask<OperationResult>(operationId).WaitAsync(TimeSpan.FromDays(1));
await DurableTask.Run(async ct =>
{
using var response = await http.PostAsJsonAsync(callback, result, ct);
response.EnsureSuccessStatusCode();
}).WithId("deliver");
}
catch (TimeoutException)
{
// The operation is still running after a day; it continues without this relay.
}
}
}
  • In a durable task, awaiting the reference, scheduling it, or passing it to WhenAny, Race, WaitAsync, or WaitUntilAsync subscribes a child of the execution, recorded under the child ID which that operation assigns, like a call to another grain. The child and its subscription commit in one journal write, and a replay which reaches the child doesn’t subscribe again.
  • The subscription travels through the outbox to the task’s grain as an invocation without a request, which neither runs nor claims the task. The grain adds the child to the task’s subscribers, beside the task’s invoker, sends the task’s result through its outbox, as it does to the invoker, and retains the task until that completion retires. The child completes with the task’s outcome: its result, its failure, or its cancellation.
  • A subscription to a terminal task completes promptly, and a subscription to a task which the grain doesn’t retain as an invoked task, because its retention ended or it was never invoked, fails promptly with DurableTaskNotFoundException; replays observe the same failure. A task ID which reaches the subscriber refers to a task which its grain holds, because ScheduleAsync returns once the grain has accepted the task. Bound the wait with WaitAsync or WaitUntilAsync.
  • Canceling the child, for example when a WaitAsync deadline passes or a Race input loses, ends it as canceled and ends its subscription. The task continues.
  • A subscription to a task of the same grain completes in the journal write which stores the task’s result.
  • Clients and ordinary grain methods await the same reference, or poll, cancel, or abort the task through it, with calls to the task’s grain.

A DurableTaskAddress pairs a task’s grain with its ID, so code stores a task in journaled state, sends it in grain calls, or returns it in HTTP payloads, and reaches the task later. Orleans serializes it, and System.Text.Json writes it as {"Target":"order/42","Id":"place-order"} in JSON journals and HTTP payloads, with no registration. A TaskId serializes as its escaped path, such as place-order/reserve.

  • GrainFactory.GetDurableTask<TResult>(address), or the same method on an IClusterClient, returns the task’s reference, which awaits, subscribes, polls, cancels, or aborts it, as above.
  • DurableTaskReference<TResult>.Address reads a reference’s address.
  • handle.TryGetAddress(out var address) reads the address of a ScheduledTask handle’s task: a root task which a client or an ordinary grain method started, or, inside a durable task, a detached task or a task which GetDurableTask observes. An attached child belongs to its caller’s tree, so its handle has no address.

A grain records the detached task which it hands off, and later awaits, polls, or cancels it, after any number of reactivations:

public interface IPublisherGrain : IGrainWithStringKey
{
DurableTask HandOffAsync(string reportId);
DurableTask<bool> AwaitArchiveAsync();
Task<DurableTaskStatus?> GetArchiveStatusAsync();
Task CancelArchiveAsync(string reason);
}
public sealed class PublisherGrain : DurableGrain, IPublisherGrain
{
private readonly IDurableValue<DurableTaskAddress?> _archive;
public PublisherGrain() => _archive = StateManager.GetOrAddValue<DurableTaskAddress?>("archive");
public async DurableTask HandOffAsync(string reportId)
{
// A detached task is a root on the archive grain, so its handle has an address.
var archive = await GrainFactory.GetGrain<IArchiveGrain>("archive")
.ArchiveAsync(reportId, version: 1)
.Detached()
.ScheduleAsync();
archive.TryGetAddress(out var address);
// The synchronous tail: the address commits with this task's result.
_archive.Value = address;
}
// A durable task subscribes to the stored task, and the subscription is recorded for replay.
public async DurableTask<bool> AwaitArchiveAsync()
{
var archive = DurableTask.RunInline("archive", () => _archive.Value);
return archive is { } address && await GrainFactory.GetDurableTask<bool>(address).WithId("archived");
}
// An ordinary method polls or cancels the stored task through the same address.
public async Task<DurableTaskStatus?> GetArchiveStatusAsync() =>
_archive.Value is { } address
? await GrainFactory.GetDurableTask<bool>(address).GetStatusAsync(new PollingOptions { PollTimeout = TimeSpan.Zero })
: null;
public async Task CancelArchiveAsync(string reason)
{
if (_archive.Value is { } address)
{
await GrainFactory.GetDurableTask<bool>(address).CancelAsync(reason);
}
}
}

Task identity and references describes which tasks have addresses and how long they resolve.

  • (grain, TaskId) identifies one execution. A task ID’s reuse is the same request when it invokes the same interface method from the same caller; arguments aren’t compared, and reuse with a different interface or method conflicts.
  • Request identity, task state, caller subscription, and inbox completion commit before execution begins.
  • Task state and outgoing messages share the grain’s journal commit boundary with the grain’s own journaled state. A failed journal write deactivates the grain, and the next activation recovers from committed state.
  • Each segment of an execution, the code between two durable waits, runs as a request on the grain, so it follows the grain’s reentrancy rules as a grain method does.
  • While an execution’s segment runs or waits to run, the grain keeps a Durable Jobs wake-up, so a grain which crashes is reactivated within about 30 seconds and resumes the execution. An execution parked on a call’s result, a signal, or a delay resumes when the message it waits for reactivates its grain.
  • Posting a durable call from a durable grain, including from an ordinary method, stages the invocation in the outbox, so it commits atomically with the grain’s state changes. Inside a durable task, awaiting and scheduling a call stage it the same way.
  • Each request and result payload is journaled once per grain, in task state. A dead-lettered completion’s dead letter also holds its result, for inspection and replay.
  • Recovery precedes new execution. Child IDs, completion order, and outgoing messages are replay-stable.
  • Terminal results and the completion notifications to callers commit together. The first durable terminal result wins a race between completion and cancellation.
  • A body’s result is staged in the work item which runs the body’s final code on the activation, so it commits in the same journal write as the state which that code changed, whichever operation requests the write. A DurableTask.RunInline record is staged in the same work item as the effect’s state changes, and commits with them.

Durability covers state and messages which participate in Orleans Journaling and Messaging commits. External side effects, such as calls to a payment provider, need their own idempotency, for example a request key derived from the task ID; see Idempotency in durable tasks.

An invoked task, which a client or another grain scheduled, owns an execution tree: the task and its descendants which aren’t invoked tasks themselves, such as local steps, delays, DurableTask.RunInline records, records of detached tasks, signal-send markers, subscriptions, and the records of calls to other grains. Once an invoked task is terminal, its grain deletes the tree:

  • When a grain called the task, in the journal write which retires the completion after the calling grain has durably accepted the result.
  • When the task has no calling grain, such as a task which a client scheduled or a detached task, DurableTaskOptions.ResultRetentionPeriod after completion. Polls during that period return the retained result, and later polls throw DurableTaskNotFoundException.
  • In either case, not before the completion to each subscribing durable task retires.

Deleting a tree cancels its unfinished tasks. A task ID therefore deduplicates schedules until its task is deleted, and scheduling it after the deletion runs the request again. Completions and delay resumes which arrive for deleted tasks are ignored. See Retention and deletion for operational details.

Durable tasks need Journaling storage and Durable Jobs storage. AddDurableTasks on the silo builder adds Messaging and the durable task runtime, stores task state in the grain’s journal, and registers the JSON metadata which the JSON journal format needs for task state. Its optional callback configures DurableTaskOptions. For development, configure in-memory storage:

siloBuilder.UseInMemoryDurableJobs();
siloBuilder.AddVolatileJournalStorage();
siloBuilder.AddDurableTasks(options => options.ResultRetentionPeriod = TimeSpan.FromDays(7));

External clients which schedule, observe, cancel, or signal durable tasks register the client-side adapter:

clientBuilder.AddDurableTasks();

Grains which derive from DurableGrain host durable tasks, and AddDurableTasks gives every such grain an inbox and an outbox. During activation setup, before journal recovery, the runtime binds to the grain’s journal, inbox, and outbox, and the grain can inject IDurableTaskManager to observe, cancel, abort, and signal its own tasks. Task state can use any journal format; requests and results are stored with the Orleans serializer. Use shared, production-grade Journaling and Durable Jobs providers for multi-silo deployments; see Journaling configuration.