Skip to content

Integrate durable tasks with other systems

These recipes connect durable tasks with systems outside Orleans: a payment provider, a database, HTTP services with long-running operations, HTTP clients which start durable tasks, and webhooks which report back. Each external call runs in a step, which records its result, and sends an idempotency key derived from the task, because a step whose result didn’t commit runs again. The external clients are interfaces, so the examples compile without a provider’s SDK. Idempotency in durable tasks explains the guarantees which these recipes rely on.

A failure is either transient, such as a timeout, a rate limit, or an outage, which a later attempt can overcome, or permanent, such as a declined card, a validation error, or a conflict, which every attempt reproduces. Retry transient failures at the level which matches how long they last, and handle permanent failures as outcomes:

FailureWhere it’s retriedHow
A grain call or a durable task start can’t be delivered or processedMessagingAutomatically, until OutboxRetryTimeout (6 hours by default), then DurableTaskDeliveryException. The workflow doesn’t retry.
A brief external fault, seconds longInside the stepA loop or a resilience pipeline, such as Polly’s or an IDurableHttpClient’s resilience handler, around the call, with the same idempotency key on every attempt.
An external outage, minutes to hours longAcross stepsA loop of steps with DurableTask.Delay backoff, which survives deactivation, bounded by a deadline in logical time.
A rejection, such as a declined payment, a conflict, or a validation errorNowhereReturn it from the step as a value, or catch its exception, and compensate or report it.

Messaging treats only explicit rejections as permanent: a start whose task ID already names another method fails with DurableTaskConflictException, and a target which rejects a request with its own code fails it with DurableTaskRejectedException.

Call a payment API with an idempotency key

Section titled “Call a payment API with an idempotency key”

Task: charge a customer through a provider which deduplicates requests by an Idempotency-Key, such as Stripe.

The provider’s client reports declines and transient unavailability as exceptions:

[GenerateSerializer, Immutable]
public sealed record ChargeResult(string ChargeId, decimal Amount);
// Stands in for a payment provider's client, such as Stripe's, which deduplicates requests by an idempotency key.
public interface IPaymentProvider
{
Task<ChargeResult> ChargeAsync(string customerId, decimal amount, string idempotencyKey, CancellationToken cancellationToken);
}
// The provider rejected the charge, for example because the card was declined. Retrying returns the same rejection.
public sealed class PaymentDeclinedException(string reason) : Exception(reason);
// The provider couldn't process the request now, for example because of a rate limit or an outage.
public sealed class PaymentUnavailableException(TimeSpan? retryAfter) : Exception("The payment provider is unavailable.")
{
public TimeSpan? RetryAfter { get; } = retryAfter;
}

The step converts the outcomes which the workflow handles into a value, so the step’s recorded result tells a replay which outcome occurred:

// A step returns the outcomes which the workflow handles as values, so a replay observes the same outcome.
[GenerateSerializer, Immutable]
public sealed record ChargeOutcome(ChargeResult? Charge, string? DeclineReason, TimeSpan? RetryAfter)
{
public static ChargeOutcome Charged(ChargeResult charge) => new(charge, null, null);
public static ChargeOutcome Declined(string reason) => new(null, reason, null);
public static ChargeOutcome Unavailable(TimeSpan? retryAfter) => new(null, null, retryAfter ?? TimeSpan.FromMinutes(1));
}

The key combines the grain’s identity, the task’s ID, and the operation’s name, which are the same in every replay and every attempt:

// The grain, the task's ID, and its incarnation identify this run in every replay and every retry. The operation's
// name distinguishes the run's external requests from each other.
private string IdempotencyKey(string operation) =>
$"{this.GetGrainId()}/{DurableTask.Current.Id}/{DurableTask.Current.Incarnation:N}/{operation}";

The workflow charges in a DurableTask.Run step, which retries a brief unavailability twice inside the step:

public async DurableTask<ChargeResult?> ChargeAsync(string customerId, decimal amount)
{
var key = IdempotencyKey("charge");
var outcome = await DurableTask.Run(async cancellationToken =>
{
// Retries brief, transient failures inside the step. Each attempt sends the same key, so the provider
// charges once however many attempts reach it, including attempts which a replay of the step repeats.
for (var attempt = 1; ; attempt++)
{
try
{
return ChargeOutcome.Charged(await payments.ChargeAsync(customerId, amount, key, cancellationToken));
}
catch (PaymentDeclinedException declined)
{
return ChargeOutcome.Declined(declined.Message);
}
catch (PaymentUnavailableException unavailable) when (attempt < 3)
{
await Task.Delay(unavailable.RetryAfter ?? TimeSpan.FromSeconds(attempt), cancellationToken);
}
catch (PaymentUnavailableException unavailable)
{
return ChargeOutcome.Unavailable(unavailable.RetryAfter);
}
}
}).WithId("charge");
return outcome.Charge;
}
  • The step receives DurableTask.Current.CancellationToken, which the workflow’s task cancellation and the cancellation of the step’s attempt both cancel. Pass it to the client and to the delay between attempts.
  • Every attempt sends the same key, so the provider charges once. When the silo fails after the provider charged and before the step’s result committed, the next run sends the same key again, and the provider returns the original charge.
  • A declined card returns a value. The step completes, and the workflow continues with the decline, in every replay.
  • A resilience pipeline, such as a Polly ResiliencePipeline, replaces the loop inside the delegate in the same way: it retries within the step’s run, with the same key and the step’s cancellation token.

Pitfalls:

  • Compute the key outside the step’s retry loop, from values which are stable across replays. A key from Guid.NewGuid() in the body changes in every replay; see ORLEANS0201.
  • Keep in-step retries short. The step holds an execution running while it retries, and a deactivation cancels the step’s attempt, which the next run starts again from the first retry.

Verify: the grain lists charge as Completed with the charge’s ID, and the provider’s dashboard shows one charge for the key, with any repeated requests marked as idempotent replays.

Task: keep trying to charge through a provider outage which lasts for hours, without holding the grain active.

public async DurableTask<ChargeResult?> ChargeWithBackoffAsync(string customerId, decimal amount)
{
var key = IdempotencyKey("charge");
var deadline = DurableTask.Current.UtcNow.AddHours(6);
while (true)
{
// Each attempt is a step, "charge", "charge$2", and so on, and each sends the same key.
var outcome = await DurableTask.Run(async cancellationToken =>
{
try
{
return ChargeOutcome.Charged(await payments.ChargeAsync(customerId, amount, key, cancellationToken));
}
catch (PaymentDeclinedException declined)
{
return ChargeOutcome.Declined(declined.Message);
}
catch (PaymentUnavailableException unavailable)
{
return ChargeOutcome.Unavailable(unavailable.RetryAfter);
}
}).WithId("charge");
if (outcome.RetryAfter is not { } retryAfter)
{
// Charged or declined: a declined card stays declined, so the workflow doesn't retry it.
return outcome.Charge;
}
if (DurableTask.Current.UtcNow + retryAfter > deadline)
{
throw new TimeoutException("The payment provider stayed unavailable for six hours.");
}
// A durable delay: the grain can deactivate while it waits, and the workflow resumes when the delay is due.
await DurableTask.Delay(retryAfter).WithId("charge-backoff");
}
}

Each attempt is a step: the loop names them charge, charge$2, and so on, and the delays charge-backoff, charge-backoff$2. A step which reports unavailability completes with that outcome, the workflow waits for the provider’s Retry-After with a durable delay, and the grain can deactivate while it waits. A replay returns each recorded attempt and resumes the delay which was pending. The deadline is computed from logical time, so every replay computes the same one.

Pitfalls:

  • Bound the retries with a deadline or an attempt count, and decide what the workflow does when it passes, such as compensating or reporting the failure.
  • Each attempt and delay adds a record to the task’s tree until the tree is deleted. A workflow which retries for days at short intervals uses longer intervals or a detached successor task.

Verify: while the provider is unavailable, the grain lists the attempts as Completed with the unavailable outcome and the latest charge-backoff as Delayed until its due time.

Task: record a shipment in a relational database, once, and decide from the database’s current contents.

[GenerateSerializer, Immutable]
public sealed record ShipmentRecord(Guid OrderId, string Carrier, string TrackingNumber);
// Stands in for a data access layer over a relational database.
public interface IShipmentStore
{
// INSERT ... ON CONFLICT (operation_id) DO NOTHING: a repeated write with the same operation ID changes nothing.
Task UpsertAsync(string operationId, ShipmentRecord shipment, CancellationToken cancellationToken);
Task<ShipmentRecord?> FindAsync(Guid orderId, CancellationToken cancellationToken);
}
public interface IShipmentGrain : IGrainWithGuidKey
{
DurableTask<ShipmentRecord> RecordShipmentAsync(string carrier, string trackingNumber);
}
public sealed class ShipmentGrain(IShipmentStore store) : DurableGrain, IShipmentGrain
{
public async DurableTask<ShipmentRecord> RecordShipmentAsync(string carrier, string trackingNumber)
{
var orderId = this.GetPrimaryKey();
// A read in a step: the workflow decides from the recorded row, so a replay takes the same path.
var existing = await DurableTask.Run(ct => store.FindAsync(orderId, ct)).WithId("find");
if (existing is not null)
{
return existing;
}
// A write keyed by an ID which the task derives, so a write which a replay repeats changes nothing.
var shipment = new ShipmentRecord(orderId, carrier, trackingNumber);
var operationId = $"{this.GetGrainId()}/{DurableTask.Current.Id}/{DurableTask.Current.Incarnation:N}/insert";
await DurableTask.Run(ct => store.UpsertAsync(operationId, shipment, ct)).WithId("insert");
return shipment;
}
}
  • Reads run in steps. The workflow decides from the row which the find step recorded, so a replay takes the same branch even when the table changed since.
  • Writes are keyed by an operation ID. An upsert with a unique constraint on the ID, such as INSERT ... ON CONFLICT (operation_id) DO NOTHING, applies a repeated write once. A store without upserts records the operation ID in the same transaction as the change, and checks it before applying the change.
  • One step per transaction. A transaction which commits several changes runs in one step, so its changes and their operation ID commit together, and the step’s record tells the workflow that they committed.

A service which commits a change to its database and must start a workflow for it writes a row to an outbox table in the same transaction. A relay reads the table and starts a durable task for each row, named after the row:

[GenerateSerializer, Immutable]
public sealed record ShipmentRequested(Guid OrderId, string Carrier);
public interface IShipmentRequestGrain : IGrainWithGuidKey
{
DurableTask<bool> HandleAsync(ShipmentRequested request);
}
// A row which an application committed to its database's outbox table in the same transaction as its own change.
public sealed record OutboxRow(long Id, ShipmentRequested Request);
public interface IOutboxTable
{
Task<IReadOnlyList<OutboxRow>> ReadPendingAsync(int count, CancellationToken cancellationToken);
Task MarkSentAsync(long id, CancellationToken cancellationToken);
}
public sealed class OutboxRelay(IClusterClient client, IOutboxTable outbox)
{
public async Task RelayAsync(CancellationToken cancellationToken)
{
foreach (var row in await outbox.ReadPendingAsync(100, cancellationToken))
{
// The row's ID names the task, so a relay which crashes before MarkSentAsync and sends the row again
// reaches the same task, which runs once.
await client.GetGrain<IShipmentRequestGrain>(row.Request.OrderId)
.HandleAsync(row.Request)
.WithId($"outbox-{row.Id}")
.ScheduleAsync(cancellationToken);
await outbox.MarkSentAsync(row.Id, cancellationToken);
}
}
}

The row’s ID makes the start idempotent: a relay which crashes before it marks a row as sent starts the row’s task again, which reaches the existing task. The receiving grain’s task state is the inbox. Size ResultRetentionPeriod to exceed the longest delay between a row’s first and last relay.

Verify: the database holds one row per operation ID, and relaying the same outbox row twice leaves the grain with one task named outbox-{id}.

Task: submit a label to a service which runs it as a long-running operation, wait for the label, and cancel the job when the workflow is canceled.

The Microsoft.Orleans.Durable.Http package’s IDurableHttpClient runs HTTP requests and operations as steps of the calling task. Register a named client in the silo, configure its HttpClient through the builder which AddDurableHttpClient returns, and inject the client as a keyed service:

[GenerateSerializer, Immutable]
public sealed record LabelResult(string Url);
public interface ILabelPrinterGrain : IGrainWithGuidKey
{
DurableTask<string> PrintLabelAsync(string address);
}
public sealed class LabelPrinterGrain([FromKeyedServices("labels")] IDurableHttpClient labels) : DurableGrain, ILabelPrinterGrain
{
public async DurableTask<string> PrintLabelAsync(string address)
{
// A step: its request carries an Idempotency-Key derived from this step's identity, so a retry or a replay of
// the step reaches the job which its first attempt started.
var job = await labels.StartOperationAsync(() => new HttpRequestMessage(HttpMethod.Post, "labels")
{
Content = JsonContent.Create(new { address }),
});
try
{
// Reads the job's status events within a step, which records the result, and gives up after a day of
// logical time.
var label = await labels.WaitForOperationAsync<LabelResult>(job, new() { Mode = DurableHttpWaitMode.Events })
.WaitAsync(TimeSpan.FromDays(1));
return label.Url;
}
catch (Exception exception) when (exception is TimeoutException || DurableTask.Current.IsTaskCancellationRequested && exception is OperationCanceledException)
{
// Cancels the remote job, then lets this task end timed out or canceled.
if (job.Location is { } location)
{
await labels.CancelOperationAsync(location, DurableTask.Current.TaskCancellationReason ?? "The label wasn't printed within a day.").Shielded();
}
throw;
}
}
}
internal static class LabelPrinterConfiguration
{
internal static void ConfigureSilo(ISiloBuilder siloBuilder, Uri labelService)
{
// The handlers which the builder adds, such as resilience handlers, follow the key handler, so their retries
// carry the key.
siloBuilder.Services.AddDurableHttpClient("labels")
.ConfigureHttpClient(client => client.BaseAddress = labelService);
}
}
  • Idempotency keys come from step identities. Each step’s unsafe requests carry an Idempotency-Key derived from the service, the grain, the step’s task ID, and the task’s run, DurableTaskContext.Incarnation, so a step which runs again after a failure or a deactivation reaches the job which its first attempt started, and a later run of the same task ID sends new keys. The key handler precedes the handlers which the builder adds, such as a resilience handler, so their retries carry the same key.
  • Responses are recorded. A replay reads each step’s recorded response without sending its request again.
  • Operations follow long-running operation conventions. A 202 Accepted response locates the operation through Operation-Location, Azure-AsyncOperation, or Location. DurableHttpWaitMode.Events reads the status monitor’s server-sent events within one step, and DurableHttpWaitMode.Poll polls after the interval which Retry-After advertises, within the task’s poll intervals, with durable delays between polls, so a long wait doesn’t hold the grain active. WaitAsync bounds the wait with a deadline in logical time. A failed or canceled operation throws DurableHttpOperationException.
  • Cancellation reaches the remote operation. CancelOperationAsync sends POST {status monitor}:cancel with the task’s cancellation reason, or the deadline’s, shielded so it runs after the workflow’s cancellation. SignalOperationAsync delivers a signal the same way.

A step which sends its own requests with DurableTask.Run gets the same keys from an HttpClient which AddDurableIdempotencyKey configures. For every method and option, see the package README.

Verify: the label service receives one job per step identity, however often the step’s request is retried, and the grain lists the start and wait steps, each Completed with its recorded response.

Task: let HTTP clients start a return as a long-running operation, retry the request safely, follow its progress, cancel it, and deliver an inspection result.

WithDurableOperations makes each endpoint of a route group whose handler returns a durable grain call start an operation which performs it:

public interface IReturnGrain : IGrainWithStringKey
{
DurableTask<string> ProcessReturnAsync(string reason);
}
internal static class ReturnEndpoints
{
public static readonly DurableSignal<bool> Inspected = new("inspected");
internal static void Map(WebApplicationBuilder builder)
{
builder.Services.AddDurableHttp();
var app = builder.Build();
// Each POST starts the grain call as an operation which the request's Idempotency-Key names, within the caller's
// scope, and the group serves its status monitor, events, cancellation, and signals under
// /returns/{returnId}/operations/{id}. The scope "returns" names the operations, so renaming the routes keeps them.
var returns = app.MapGroup("/returns/{returnId}")
.WithDurableOperations("returns")
.RequireAuthorization("returns");
returns.MapPost("/process", (string returnId, string reason, IGrainFactory grains) =>
grains.GetGrain<IReturnGrain>(returnId).ProcessReturnAsync(reason));
returns.MapDurableSignal(Inspected);
}
}
  • The Idempotency-Key names the operation. Each retry of a request reaches the operation which its first attempt started, and a start without a key receives 400 Bad Request. The operation’s key combines the group’s scope, its route values, the caller’s authorization scope, and the identifier, so a caller reaches only its own operations, and the call runs as a root task named by that key. A key which another request used to start other work, a call of another method or on another grain, receives 422 Unprocessable Content; a retry whose arguments differ reaches the operation.
  • The status monitor reports progress. A start answers 200 OK with the result when the operation completes, including within the wait which Prefer: wait=N requests, and otherwise 202 Accepted with an Operation-Location header, the URL of GET {group}/operations/{id}. That status monitor reports Running, Succeeded, Failed, or Canceled, with the result or an error, and also honors Prefer: wait=N.
  • Events stream the status. GET {group}/operations/{id}/events streams the status monitor as server-sent events until the operation is terminal, so a browser’s EventSource follows it without polling.
  • Clients cancel and signal the task. POST {group}/operations/{id}:cancel requests durable cancellation with a reason, and POST {group}/operations/{id}/signals/{name} delivers a signal which MapDurableSignal maps. The operation’s host records the task which performs its work in its journal, so cancellations and signals reach that task whichever activation serves them.
  • Authorization is the application’s. Policies and filters on the group cover its starts and the routes which serve its operations.

The operation’s host retains a completed operation, and so deduplicates its key, for ResultRetentionPeriod. The package README describes every response, the Nexus protocol, and callbacks which deliver an operation’s outcome to a webhook URL.

Verify: a repeated POST with the same Idempotency-Key returns the same Operation-Location, the status monitor reports the operation’s outcome, and its events stream ends with the terminal status.

Task: wait for a payment provider’s webhook which confirms a charge.

[GenerateSerializer, Immutable]
public sealed record PaymentConfirmation(string EventId, string ChargeId, bool Succeeded);
public interface IPaymentWorkflowGrain : IGrainWithGuidKey
{
DurableTask<bool> AwaitPaymentAsync(string chargeId);
Task<DurableSignalResult> OnPaymentWebhookAsync(PaymentConfirmation confirmation);
}
public sealed class PaymentWorkflowGrain(IDurableTaskManager tasks) : DurableGrain, IPaymentWorkflowGrain
{
private static readonly DurableSignal<PaymentConfirmation> Confirmed = new("payment-confirmed");
public static string WorkflowId(Guid paymentId) => $"payment-{paymentId:N}";
public async DurableTask<bool> AwaitPaymentAsync(string chargeId)
{
// A webhook which arrives before the wait is recorded, and the wait returns it at once.
var (received, confirmation) = await Confirmed.TryWaitAsync(DurableTask.Current.UtcNow.AddDays(1)).WithId("confirmation");
return received && confirmation.Succeeded;
}
// Called by the web endpoint which verified the webhook's signature. A provider which delivers the same event again
// receives Duplicate, which the endpoint acknowledges like Accepted.
public async Task<DurableSignalResult> OnPaymentWebhookAsync(PaymentConfirmation confirmation) =>
await tasks.TrySignalAsync(WorkflowId(this.GetPrimaryKey()), Confirmed, confirmation);
}

The web endpoint verifies the webhook’s signature and calls an ordinary grain method, which delivers the confirmation as a signal with TrySignalAsync. For a task which runs as an HTTP operation, MapDurableSignal serves the same delivery as POST {group}/operations/{id}/signals/{name}. The task records the first value for the signal, so:

  • A webhook which arrives before the workflow waits is recorded, and the wait returns it at once.
  • A provider which delivers the same event again receives Duplicate, which the endpoint acknowledges with a success status, so the provider stops retrying.
  • A different value for the same signal returns Conflict, which the endpoint reports and logs.
  • TryWaitAsync bounds the wait with a durable deadline.

Webhooks which report several events for one task use one signal name per event, derived with DurableSignal<T>.For, such as one per charge ID.

Verify: the first delivery returns Accepted, a repeated delivery returns Duplicate, and the workflow completes with the confirmation’s outcome.

Task: start a follow-up survey under a business ID, and handle a start which can’t succeed.

public interface IFollowUpGrain : IGrainWithGuidKey
{
DurableTask<string> ScheduleFollowUpAsync(Guid orderId);
}
public sealed class FollowUpGrain : DurableGrain, IFollowUpGrain
{
public async DurableTask<string> ScheduleFollowUpAsync(Guid orderId)
{
try
{
// Messaging retries delivery of the start until the target accepts it, so the workflow doesn't retry it.
var survey = await GrainFactory.GetGrain<IExpenseGrain>(orderId)
.SubmitAsync(new ExpenseClaim("survey", 0))
.Detached($"survey-{orderId:N}")
.ScheduleAsync();
return survey.TargetTaskId.ToString();
}
catch (DurableTaskConflictException)
{
// The ID already names another request, and every retry reaches the same conflict.
return "conflict";
}
catch (DurableTaskDeliveryException)
{
// The start wasn't accepted within Messaging's OutboxRetryTimeout.
return "undelivered";
}
}
}
  • DurableTaskConflictException reports that the ID already names another method. Every retry under the same ID reaches the same conflict, so the workflow resolves it, for example by reading the existing task, or by choosing another ID.
  • DurableTaskRejectedException reports a target which rejected the request with its own code, in Code.
  • DurableTaskDeliveryException reports a start which Messaging couldn’t deliver within OutboxRetryTimeout. The workflow treats it like an outage which outlasted its deadline.
  • Application outcomes, such as a declined card or an out-of-stock line, are values which steps and grain methods return, so the workflow branches on them instead of retrying.

Verify: a second start under the same ID with the same method and different arguments reaches the first task, and a start under that ID with another method throws DurableTaskConflictException, and the target’s dead letters record the durable-rpc/conflict code; see Dead letters.