Integrate durable tasks with other systems
These recipes connect durable tasks with systems outside Orleans: a payment provider, a database, HTTP services with long-running operations, HTTP clients which start durable tasks, and webhooks which report back. Each external call runs in a step, which records its result, and sends an idempotency key derived from the task, because a step whose result didn’t commit runs again. The external clients are interfaces, so the examples compile without a provider’s SDK. Idempotency in durable tasks explains the guarantees which these recipes rely on.
Choose where to retry
Section titled “Choose where to retry”A failure is either transient, such as a timeout, a rate limit, or an outage, which a later attempt can overcome, or permanent, such as a declined card, a validation error, or a conflict, which every attempt reproduces. Retry transient failures at the level which matches how long they last, and handle permanent failures as outcomes:
| Failure | Where it’s retried | How |
|---|---|---|
| A grain call or a durable task start can’t be delivered or processed | Messaging | Automatically, until OutboxRetryTimeout (6 hours by default), then DurableTaskDeliveryException. The workflow doesn’t retry. |
| A brief external fault, seconds long | Inside the step | A loop or a resilience pipeline, such as Polly’s or an IDurableHttpClient’s resilience handler, around the call, with the same idempotency key on every attempt. |
| An external outage, minutes to hours long | Across steps | A loop of steps with DurableTask.Delay backoff, which survives deactivation, bounded by a deadline in logical time. |
| A rejection, such as a declined payment, a conflict, or a validation error | Nowhere | Return it from the step as a value, or catch its exception, and compensate or report it. |
Messaging treats only explicit rejections as permanent: a start whose task ID already
names another method fails with DurableTaskConflictException, and a target which
rejects a request with its own code fails it with DurableTaskRejectedException.
Call a payment API with an idempotency key
Section titled “Call a payment API with an idempotency key”Task: charge a customer through a provider which deduplicates requests by an
Idempotency-Key, such as Stripe.
The provider’s client reports declines and transient unavailability as exceptions:
[GenerateSerializer, Immutable]public sealed record ChargeResult(string ChargeId, decimal Amount);
// Stands in for a payment provider's client, such as Stripe's, which deduplicates requests by an idempotency key.public interface IPaymentProvider{ Task<ChargeResult> ChargeAsync(string customerId, decimal amount, string idempotencyKey, CancellationToken cancellationToken);}
// The provider rejected the charge, for example because the card was declined. Retrying returns the same rejection.public sealed class PaymentDeclinedException(string reason) : Exception(reason);
// The provider couldn't process the request now, for example because of a rate limit or an outage.public sealed class PaymentUnavailableException(TimeSpan? retryAfter) : Exception("The payment provider is unavailable."){ public TimeSpan? RetryAfter { get; } = retryAfter;}The step converts the outcomes which the workflow handles into a value, so the step’s recorded result tells a replay which outcome occurred:
// A step returns the outcomes which the workflow handles as values, so a replay observes the same outcome.[GenerateSerializer, Immutable]public sealed record ChargeOutcome(ChargeResult? Charge, string? DeclineReason, TimeSpan? RetryAfter){ public static ChargeOutcome Charged(ChargeResult charge) => new(charge, null, null);
public static ChargeOutcome Declined(string reason) => new(null, reason, null);
public static ChargeOutcome Unavailable(TimeSpan? retryAfter) => new(null, null, retryAfter ?? TimeSpan.FromMinutes(1));}The key combines the grain’s identity, the task’s ID, and the operation’s name, which are the same in every replay and every attempt:
// The grain, the task's ID, and its incarnation identify this run in every replay and every retry. The operation's// name distinguishes the run's external requests from each other.private string IdempotencyKey(string operation) => $"{this.GetGrainId()}/{DurableTask.Current.Id}/{DurableTask.Current.Incarnation:N}/{operation}";The workflow charges in a DurableTask.Run step, which retries a brief unavailability
twice inside the step:
public async DurableTask<ChargeResult?> ChargeAsync(string customerId, decimal amount){ var key = IdempotencyKey("charge"); var outcome = await DurableTask.Run(async cancellationToken => { // Retries brief, transient failures inside the step. Each attempt sends the same key, so the provider // charges once however many attempts reach it, including attempts which a replay of the step repeats. for (var attempt = 1; ; attempt++) { try { return ChargeOutcome.Charged(await payments.ChargeAsync(customerId, amount, key, cancellationToken)); } catch (PaymentDeclinedException declined) { return ChargeOutcome.Declined(declined.Message); } catch (PaymentUnavailableException unavailable) when (attempt < 3) { await Task.Delay(unavailable.RetryAfter ?? TimeSpan.FromSeconds(attempt), cancellationToken); } catch (PaymentUnavailableException unavailable) { return ChargeOutcome.Unavailable(unavailable.RetryAfter); } } }).WithId("charge");
return outcome.Charge;}- The step receives
DurableTask.Current.CancellationToken, which the workflow’s task cancellation and the cancellation of the step’s attempt both cancel. Pass it to the client and to the delay between attempts. - Every attempt sends the same key, so the provider charges once. When the silo fails after the provider charged and before the step’s result committed, the next run sends the same key again, and the provider returns the original charge.
- A declined card returns a value. The step completes, and the workflow continues with the decline, in every replay.
- A resilience pipeline, such as a Polly
ResiliencePipeline, replaces the loop inside the delegate in the same way: it retries within the step’s run, with the same key and the step’s cancellation token.
Pitfalls:
- Compute the key outside the step’s retry loop, from values which are stable across
replays. A key from
Guid.NewGuid()in the body changes in every replay; see ORLEANS0201. - Keep in-step retries short. The step holds an execution running while it retries, and a deactivation cancels the step’s attempt, which the next run starts again from the first retry.
Verify: the grain lists charge as Completed with the charge’s ID, and the
provider’s dashboard shows one charge for the key, with any repeated requests marked as
idempotent replays.
Retry across steps with durable backoff
Section titled “Retry across steps with durable backoff”Task: keep trying to charge through a provider outage which lasts for hours, without holding the grain active.
public async DurableTask<ChargeResult?> ChargeWithBackoffAsync(string customerId, decimal amount){ var key = IdempotencyKey("charge"); var deadline = DurableTask.Current.UtcNow.AddHours(6); while (true) { // Each attempt is a step, "charge", "charge$2", and so on, and each sends the same key. var outcome = await DurableTask.Run(async cancellationToken => { try { return ChargeOutcome.Charged(await payments.ChargeAsync(customerId, amount, key, cancellationToken)); } catch (PaymentDeclinedException declined) { return ChargeOutcome.Declined(declined.Message); } catch (PaymentUnavailableException unavailable) { return ChargeOutcome.Unavailable(unavailable.RetryAfter); } }).WithId("charge");
if (outcome.RetryAfter is not { } retryAfter) { // Charged or declined: a declined card stays declined, so the workflow doesn't retry it. return outcome.Charge; }
if (DurableTask.Current.UtcNow + retryAfter > deadline) { throw new TimeoutException("The payment provider stayed unavailable for six hours."); }
// A durable delay: the grain can deactivate while it waits, and the workflow resumes when the delay is due. await DurableTask.Delay(retryAfter).WithId("charge-backoff"); }}Each attempt is a step: the loop names them charge, charge$2, and so on, and the
delays charge-backoff, charge-backoff$2. A step which reports unavailability completes
with that outcome, the workflow waits for the provider’s Retry-After with a durable
delay, and the grain can deactivate while it waits. A replay returns each recorded
attempt and resumes the delay which was pending. The deadline is computed from logical
time, so every replay computes the same one.
Pitfalls:
- Bound the retries with a deadline or an attempt count, and decide what the workflow does when it passes, such as compensating or reporting the failure.
- Each attempt and delay adds a record to the task’s tree until the tree is deleted. A workflow which retries for days at short intervals uses longer intervals or a detached successor task.
Verify: while the provider is unavailable, the grain lists the attempts as Completed
with the unavailable outcome and the latest charge-backoff as Delayed until its due
time.
Write to a database
Section titled “Write to a database”Task: record a shipment in a relational database, once, and decide from the database’s current contents.
[GenerateSerializer, Immutable]public sealed record ShipmentRecord(Guid OrderId, string Carrier, string TrackingNumber);
// Stands in for a data access layer over a relational database.public interface IShipmentStore{ // INSERT ... ON CONFLICT (operation_id) DO NOTHING: a repeated write with the same operation ID changes nothing. Task UpsertAsync(string operationId, ShipmentRecord shipment, CancellationToken cancellationToken);
Task<ShipmentRecord?> FindAsync(Guid orderId, CancellationToken cancellationToken);}
public interface IShipmentGrain : IGrainWithGuidKey{ DurableTask<ShipmentRecord> RecordShipmentAsync(string carrier, string trackingNumber);}
public sealed class ShipmentGrain(IShipmentStore store) : DurableGrain, IShipmentGrain{ public async DurableTask<ShipmentRecord> RecordShipmentAsync(string carrier, string trackingNumber) { var orderId = this.GetPrimaryKey();
// A read in a step: the workflow decides from the recorded row, so a replay takes the same path. var existing = await DurableTask.Run(ct => store.FindAsync(orderId, ct)).WithId("find"); if (existing is not null) { return existing; }
// A write keyed by an ID which the task derives, so a write which a replay repeats changes nothing. var shipment = new ShipmentRecord(orderId, carrier, trackingNumber); var operationId = $"{this.GetGrainId()}/{DurableTask.Current.Id}/{DurableTask.Current.Incarnation:N}/insert"; await DurableTask.Run(ct => store.UpsertAsync(operationId, shipment, ct)).WithId("insert"); return shipment; }}- Reads run in steps. The workflow decides from the row which the
findstep recorded, so a replay takes the same branch even when the table changed since. - Writes are keyed by an operation ID. An upsert with a unique constraint on the ID,
such as
INSERT ... ON CONFLICT (operation_id) DO NOTHING, applies a repeated write once. A store without upserts records the operation ID in the same transaction as the change, and checks it before applying the change. - One step per transaction. A transaction which commits several changes runs in one step, so its changes and their operation ID commit together, and the step’s record tells the workflow that they committed.
Deliver database changes to grains
Section titled “Deliver database changes to grains”A service which commits a change to its database and must start a workflow for it writes a row to an outbox table in the same transaction. A relay reads the table and starts a durable task for each row, named after the row:
[GenerateSerializer, Immutable]public sealed record ShipmentRequested(Guid OrderId, string Carrier);
public interface IShipmentRequestGrain : IGrainWithGuidKey{ DurableTask<bool> HandleAsync(ShipmentRequested request);}
// A row which an application committed to its database's outbox table in the same transaction as its own change.public sealed record OutboxRow(long Id, ShipmentRequested Request);
public interface IOutboxTable{ Task<IReadOnlyList<OutboxRow>> ReadPendingAsync(int count, CancellationToken cancellationToken);
Task MarkSentAsync(long id, CancellationToken cancellationToken);}
public sealed class OutboxRelay(IClusterClient client, IOutboxTable outbox){ public async Task RelayAsync(CancellationToken cancellationToken) { foreach (var row in await outbox.ReadPendingAsync(100, cancellationToken)) { // The row's ID names the task, so a relay which crashes before MarkSentAsync and sends the row again // reaches the same task, which runs once. await client.GetGrain<IShipmentRequestGrain>(row.Request.OrderId) .HandleAsync(row.Request) .WithId($"outbox-{row.Id}") .ScheduleAsync(cancellationToken); await outbox.MarkSentAsync(row.Id, cancellationToken); } }}The row’s ID makes the start idempotent: a relay which crashes before it marks a row as
sent starts the row’s task again, which reaches the existing task. The receiving grain’s
task state is the inbox. Size ResultRetentionPeriod to exceed the longest delay between
a row’s first and last relay.
Verify: the database holds one row per operation ID, and relaying the same outbox row
twice leaves the grain with one task named outbox-{id}.
Call HTTP APIs from durable tasks
Section titled “Call HTTP APIs from durable tasks”Task: submit a label to a service which runs it as a long-running operation, wait for the label, and cancel the job when the workflow is canceled.
The Microsoft.Orleans.Durable.Http package’s IDurableHttpClient runs HTTP
requests and operations as steps of the calling task. Register a named client in the
silo, configure its HttpClient through the builder which AddDurableHttpClient
returns, and inject the client as a keyed service:
[GenerateSerializer, Immutable]public sealed record LabelResult(string Url);
public interface ILabelPrinterGrain : IGrainWithGuidKey{ DurableTask<string> PrintLabelAsync(string address);}
public sealed class LabelPrinterGrain([FromKeyedServices("labels")] IDurableHttpClient labels) : DurableGrain, ILabelPrinterGrain{ public async DurableTask<string> PrintLabelAsync(string address) { // A step: its request carries an Idempotency-Key derived from this step's identity, so a retry or a replay of // the step reaches the job which its first attempt started. var job = await labels.StartOperationAsync(() => new HttpRequestMessage(HttpMethod.Post, "labels") { Content = JsonContent.Create(new { address }), });
try { // Reads the job's status events within a step, which records the result, and gives up after a day of // logical time. var label = await labels.WaitForOperationAsync<LabelResult>(job, new() { Mode = DurableHttpWaitMode.Events }) .WaitAsync(TimeSpan.FromDays(1)); return label.Url; } catch (Exception exception) when (exception is TimeoutException || DurableTask.Current.IsTaskCancellationRequested && exception is OperationCanceledException) { // Cancels the remote job, then lets this task end timed out or canceled. if (job.Location is { } location) { await labels.CancelOperationAsync(location, DurableTask.Current.TaskCancellationReason ?? "The label wasn't printed within a day.").Shielded(); }
throw; } }}
internal static class LabelPrinterConfiguration{ internal static void ConfigureSilo(ISiloBuilder siloBuilder, Uri labelService) { // The handlers which the builder adds, such as resilience handlers, follow the key handler, so their retries // carry the key. siloBuilder.Services.AddDurableHttpClient("labels") .ConfigureHttpClient(client => client.BaseAddress = labelService); }}- Idempotency keys come from step identities. Each step’s unsafe requests carry an
Idempotency-Keyderived from the service, the grain, the step’s task ID, and the task’s run,DurableTaskContext.Incarnation, so a step which runs again after a failure or a deactivation reaches the job which its first attempt started, and a later run of the same task ID sends new keys. The key handler precedes the handlers which the builder adds, such as a resilience handler, so their retries carry the same key. - Responses are recorded. A replay reads each step’s recorded response without sending its request again.
- Operations follow long-running operation conventions. A
202 Acceptedresponse locates the operation throughOperation-Location,Azure-AsyncOperation, orLocation.DurableHttpWaitMode.Eventsreads the status monitor’s server-sent events within one step, andDurableHttpWaitMode.Pollpolls after the interval whichRetry-Afteradvertises, within the task’s poll intervals, with durable delays between polls, so a long wait doesn’t hold the grain active.WaitAsyncbounds the wait with a deadline in logical time. A failed or canceled operation throwsDurableHttpOperationException. - Cancellation reaches the remote operation.
CancelOperationAsyncsendsPOST {status monitor}:cancelwith the task’s cancellation reason, or the deadline’s, shielded so it runs after the workflow’s cancellation.SignalOperationAsyncdelivers a signal the same way.
A step which sends its own requests with DurableTask.Run gets the same keys from an
HttpClient which AddDurableIdempotencyKey configures. For every method and option,
see the package README.
Verify: the label service receives one job per step identity, however often the
step’s request is retried, and the grain lists the start and wait steps, each
Completed with its recorded response.
Serve durable tasks as HTTP operations
Section titled “Serve durable tasks as HTTP operations”Task: let HTTP clients start a return as a long-running operation, retry the request safely, follow its progress, cancel it, and deliver an inspection result.
WithDurableOperations makes each endpoint of a route group whose handler returns a
durable grain call start an operation which performs it:
public interface IReturnGrain : IGrainWithStringKey{ DurableTask<string> ProcessReturnAsync(string reason);}
internal static class ReturnEndpoints{ public static readonly DurableSignal<bool> Inspected = new("inspected");
internal static void Map(WebApplicationBuilder builder) { builder.Services.AddDurableHttp(); var app = builder.Build();
// Each POST starts the grain call as an operation which the request's Idempotency-Key names, within the caller's // scope, and the group serves its status monitor, events, cancellation, and signals under // /returns/{returnId}/operations/{id}. The scope "returns" names the operations, so renaming the routes keeps them. var returns = app.MapGroup("/returns/{returnId}") .WithDurableOperations("returns") .RequireAuthorization("returns");
returns.MapPost("/process", (string returnId, string reason, IGrainFactory grains) => grains.GetGrain<IReturnGrain>(returnId).ProcessReturnAsync(reason)); returns.MapDurableSignal(Inspected); }}- The
Idempotency-Keynames the operation. Each retry of a request reaches the operation which its first attempt started, and a start without a key receives400 Bad Request. The operation’s key combines the group’s scope, its route values, the caller’s authorization scope, and the identifier, so a caller reaches only its own operations, and the call runs as a root task named by that key. A key which another request used to start other work, a call of another method or on another grain, receives422 Unprocessable Content; a retry whose arguments differ reaches the operation. - The status monitor reports progress. A start answers
200 OKwith the result when the operation completes, including within the wait whichPrefer: wait=Nrequests, and otherwise202 Acceptedwith anOperation-Locationheader, the URL ofGET {group}/operations/{id}. That status monitor reportsRunning,Succeeded,Failed, orCanceled, with the result or an error, and also honorsPrefer: wait=N. - Events stream the status.
GET {group}/operations/{id}/eventsstreams the status monitor as server-sent events until the operation is terminal, so a browser’sEventSourcefollows it without polling. - Clients cancel and signal the task.
POST {group}/operations/{id}:cancelrequests durable cancellation with a reason, andPOST {group}/operations/{id}/signals/{name}delivers a signal whichMapDurableSignalmaps. The operation’s host records the task which performs its work in its journal, so cancellations and signals reach that task whichever activation serves them. - Authorization is the application’s. Policies and filters on the group cover its starts and the routes which serve its operations.
The operation’s host retains a completed operation, and so deduplicates its key, for
ResultRetentionPeriod. The package README describes every response, the
Nexus protocol, and callbacks which deliver an operation’s outcome to a webhook URL.
Verify: a repeated POST with the same Idempotency-Key returns the same
Operation-Location, the status monitor reports the operation’s outcome, and its
events stream ends with the terminal status.
Receive webhooks and callbacks as signals
Section titled “Receive webhooks and callbacks as signals”Task: wait for a payment provider’s webhook which confirms a charge.
[GenerateSerializer, Immutable]public sealed record PaymentConfirmation(string EventId, string ChargeId, bool Succeeded);
public interface IPaymentWorkflowGrain : IGrainWithGuidKey{ DurableTask<bool> AwaitPaymentAsync(string chargeId);
Task<DurableSignalResult> OnPaymentWebhookAsync(PaymentConfirmation confirmation);}
public sealed class PaymentWorkflowGrain(IDurableTaskManager tasks) : DurableGrain, IPaymentWorkflowGrain{ private static readonly DurableSignal<PaymentConfirmation> Confirmed = new("payment-confirmed");
public static string WorkflowId(Guid paymentId) => $"payment-{paymentId:N}";
public async DurableTask<bool> AwaitPaymentAsync(string chargeId) { // A webhook which arrives before the wait is recorded, and the wait returns it at once. var (received, confirmation) = await Confirmed.TryWaitAsync(DurableTask.Current.UtcNow.AddDays(1)).WithId("confirmation"); return received && confirmation.Succeeded; }
// Called by the web endpoint which verified the webhook's signature. A provider which delivers the same event again // receives Duplicate, which the endpoint acknowledges like Accepted. public async Task<DurableSignalResult> OnPaymentWebhookAsync(PaymentConfirmation confirmation) => await tasks.TrySignalAsync(WorkflowId(this.GetPrimaryKey()), Confirmed, confirmation);}The web endpoint verifies the webhook’s signature and calls an ordinary grain method,
which delivers the confirmation as a signal with TrySignalAsync. For a task which runs
as an HTTP operation, MapDurableSignal serves the same delivery as
POST {group}/operations/{id}/signals/{name}. The task records the
first value for the signal, so:
- A webhook which arrives before the workflow waits is recorded, and the wait returns it at once.
- A provider which delivers the same event again receives
Duplicate, which the endpoint acknowledges with a success status, so the provider stops retrying. - A different value for the same signal returns
Conflict, which the endpoint reports and logs. TryWaitAsyncbounds the wait with a durable deadline.
Webhooks which report several events for one task use one signal name per event, derived
with DurableSignal<T>.For, such as one per charge ID.
Verify: the first delivery returns Accepted, a repeated delivery returns
Duplicate, and the workflow completes with the confirmation’s outcome.
Handle rejections without retrying
Section titled “Handle rejections without retrying”Task: start a follow-up survey under a business ID, and handle a start which can’t succeed.
public interface IFollowUpGrain : IGrainWithGuidKey{ DurableTask<string> ScheduleFollowUpAsync(Guid orderId);}
public sealed class FollowUpGrain : DurableGrain, IFollowUpGrain{ public async DurableTask<string> ScheduleFollowUpAsync(Guid orderId) { try { // Messaging retries delivery of the start until the target accepts it, so the workflow doesn't retry it. var survey = await GrainFactory.GetGrain<IExpenseGrain>(orderId) .SubmitAsync(new ExpenseClaim("survey", 0)) .Detached($"survey-{orderId:N}") .ScheduleAsync(); return survey.TargetTaskId.ToString(); } catch (DurableTaskConflictException) { // The ID already names another request, and every retry reaches the same conflict. return "conflict"; } catch (DurableTaskDeliveryException) { // The start wasn't accepted within Messaging's OutboxRetryTimeout. return "undelivered"; } }}DurableTaskConflictExceptionreports that the ID already names another method. Every retry under the same ID reaches the same conflict, so the workflow resolves it, for example by reading the existing task, or by choosing another ID.DurableTaskRejectedExceptionreports a target which rejected the request with its own code, inCode.DurableTaskDeliveryExceptionreports a start which Messaging couldn’t deliver withinOutboxRetryTimeout. The workflow treats it like an outage which outlasted its deadline.- Application outcomes, such as a declined card or an out-of-stock line, are values which steps and grain methods return, so the workflow branches on them instead of retrying.
Verify: a second start under the same ID with the same method and different
arguments reaches the first task, and a start under that ID with another method throws
DurableTaskConflictException, and the target’s dead letters record the
durable-rpc/conflict code; see Dead letters.
