Version durable tasks
A durable task which is pending during a deployment resumes on the new code: the next
activation runs the body again from the start, and each recorded operation returns its
recorded outcome. DurableTask.GetVersion lets one body hold the code for every version
of a change, and gives each execution the version which it follows: executions which
started on earlier code keep their operations, and new executions take the new ones.
This page builds on Replay and determinism and Task identity and references. Changes which only add operations after existing ones, or which add a new method, need no version; see Version workflows safely.
Branch on a change’s version
Section titled “Branch on a change’s version”The workflow below shipped orders through a warehouse. A later deployment books a carrier instead, and a deployment after that also notifies the customer. Each deployment keeps the earlier branches, and the check states the versions which the code supports:
public async DurableTask<string> FulfillAsync(string orderId){ await Reserve(orderId).WithId("reserve");
// Version 0 is the code which shipped through the warehouse. Version 1 books a carrier, and version 2 also // notifies the customer. Pending executions keep the version which they recorded. var shipping = DurableTask.GetVersion("carrier-booking", minSupported: 0, maxSupported: 2); if (shipping >= 1) { var trackingId = await Carrier.BookAsync(orderId).WithId("book"); if (shipping >= 2) { await Carrier.NotifyAsync(orderId, trackingId).WithId("notify"); }
return trackingId; }
return await Warehouse.ShipAsync(orderId).WithId("ship");}The first check of a change in an execution decides the version, and records it beneath
the execution as $version:{name}. Every later check of the change in the execution,
and every replay, returns the recorded version:
| Execution | Version |
|---|---|
| Reaches the check for the first time: a new execution, or one whose earlier runs stopped before this point | maxSupported |
| Ran past this point on code without the check | 0, the code before the change |
| Recorded a version | The recorded version |
| A step whose enclosing execution on the same grain recorded a version | The enclosing execution’s version |
A replay identifies an execution which ran past the check from its history: the replay reaches the check while operations which the earlier run recorded remain ahead of it. Place the check before the first operation which the change adds, removes, or alters, so that every execution reaches it with the operations which its version expects.
- A check inside a loop, or in several helpers, returns the first decision every time.
- The record commits with the execution’s next durable write, as a
DurableTask.RunInlineeffect’s does, and a replay which ends before that write decides again from the same history. - Version decisions are scoped to an invoked task’s execution tree: a call to another grain decides its own versions.
- A decision applies to the execution which records it and to the steps beneath it. A step which checks a change before its parent does records its own decision, which doesn’t bind the parent: the parent’s later check decides from the parent’s history.
- A task whose cancellation was requested replays the request at the point where its
body first observed it, so its body takes its compensation path where the earlier run
took it. A check which only compensation reaches, and which no earlier run recorded,
finds the operations which the earlier run recorded ahead of it and decides
0. Check the change before the body branches on cancellation, such as at the start of the body, when compensation depends on the version. - The check runs on the activation’s scheduler, as
DurableTask.RunInlinedoes. Outside a durable task, it returnsmaxSupported. - Names follow the rules for operation names: they’re non-empty and don’t contain
$. ORLEANS0218 reports names and ranges which aren’t constants.
Evolve a change
Section titled “Evolve a change”Each deployment of a change raises maxSupported and adds a branch:
- The first deployment checks
GetVersion("carrier-booking", 0, 1): version 1 books a carrier, and version 0 keeps the warehouse. - The next deployment checks
GetVersion("carrier-booking", 0, 2): version 2 adds the notification, and executions which recorded version 1 keep booking without it.
An unrelated change elsewhere in the body gets its own name. Give each version’s new operations new names, so their records never meet another version’s records.
Declare a change once
Section titled “Declare a change once”A DurableVersion holds a change’s name and supported range in one declaration, as a
DurableSignal<T> holds a signal’s name. Every check of the change then states the same
range, and a deployment which adds or retires a version edits the declaration:
// One declaration states the change's name and the versions which this code supports.private static readonly DurableVersion CarrierBooking = new("carrier-booking", minSupported: 0, maxSupported: 2);
public async DurableTask<string> FulfillAsync(string orderId){ if (!CarrierBooking.IsAtLeast(1)) { return await Warehouse.ShipAsync(orderId).WithId("ship"); }
var trackingId = await Carrier.BookAsync(orderId).WithId("book"); if (CarrierBooking.IsAtLeast(2)) { await Carrier.NotifyAsync(orderId, trackingId).WithId("notify"); }
return trackingId;}Get() returns the execution’s version, and IsAtLeast(version) reports whether the
execution follows version or a newer one.
Retire a version
Section titled “Retire a version”Retire a version once no pending execution follows it:
-
Find the executions which follow it. Each execution holds its decision as the
$version:{name}record, whose response is the version, and theVersionRecordedevent reports each decision as it’s recorded; see Monitor versions. -
Raise
minSupportedabove the version, and remove its branch:public async DurableTask<string> FulfillAsync(string orderId){await Reserve(orderId).WithId("reserve");// No pending execution follows version 0, so the warehouse branch is gone. An execution which recorded version// 0 stalls here until code which supports it replays it, or until an operator aborts it.var shipping = DurableTask.GetVersion("carrier-booking", minSupported: 1, maxSupported: 2);var trackingId = await Carrier.BookAsync(orderId).WithId("book");if (shipping >= 2){await Carrier.NotifyAsync(orderId, trackingId).WithId("notify");}return trackingId;} -
Once
minSupportedequalsmaxSupportedand every pending execution follows that version, remove the check. The records which executions hold stay in their trees until the trees are deleted, and replays skip them.
Give a later change a new name rather than reusing a retired one: an execution which ran
past the retired check before it was added holds no record, and a new check under the
old name would decide version 0 for it.
Stalled executions
Section titled “Stalled executions”An execution stalls when the running code can’t replay it. The runtime cancels the attempt with one of two reasons, as it cancels the attempts of a stopping activation; see Task cancellation and attempt cancellation:
| Reason | Cause | Inner exception |
|---|---|---|
UnsupportedVersion | The execution recorded a version outside the range which the running code supports. | A DurableTaskVersionException, which names the change, the recorded version, and the supported range |
Diverged | The replay requested another operation under a recorded ID, a recorded outcome no longer decodes, the task’s stored request no longer decodes, or the body ended before the point where an earlier run observed the task’s cancellation. See Diagnose replay divergence. | An InvalidOperationException which names the recorded and the requested operations, the decoding failure, or the unreached cancellation point |
The runtime:
- Throws
DurableTaskAttemptCanceledExceptionfrom the operation which found the stall, with the inner exception above. The attempt’s later durable operations throw the same cancellation, andAttemptCancellationTokenis canceled, so the attempt unwinds through itsfinallyblocks. - Discards the attempt’s result, whatever the body does with the exception.
- Keeps the task pending. Callers which await it keep waiting.
- Logs a warning and raises the
ExecutionStalledevent, which carries theReasonand theException. - Keeps a wake-up for the grain every
DurableTaskOptions.StalledExecutionRetryInterval, one minute by default, so a later activation replays the task, for example on the silos of the next deployment.
A version stall happens when a deployment raises minSupported while executions follow
the retired version, or when code which supports fewer versions replays an execution,
such as during a rollback. Deploy code which supports the recorded version, or code which
matches the record of a diverged execution, and the execution resumes; or abort the task,
see Abort stuck tasks.
Rolling deployments
Section titled “Rolling deployments”During a rolling deployment, a grain can activate on a silo which runs the previous
code. That code stalls executions which recorded a version newer than its
maxSupported, and resumes none of them until a silo with newer code activates the
grain. Code from before a change’s first check has no check, so it continues every
execution on the code before the change, including executions which newer code started.
Drain old executions first. Let a deployment which adds a change’s first check, or raises
maxSupported, complete on every silo before work depends on the new version, and let
executions which earlier code started complete before a later deployment retires their
version.
Monitor versions
Section titled “Monitor versions”The DurableTaskEvents listener reports each recorded decision and each stall. A
monitor which counts decisions shows which versions executions still record, and logs
stalls:
public sealed class VersionMonitor : IObserver<DurableTaskEvents.DurableTaskEvent>, IDisposable{ private readonly IDisposable _subscription; private readonly Dictionary<(string Name, int Version), long> _recorded = [];
public VersionMonitor() => _subscription = DurableTaskEvents.AllEvents.Subscribe(this);
public void OnNext(DurableTaskEvents.DurableTaskEvent value) { switch (value) { case DurableTaskEvents.VersionRecorded recorded: // Executions which record an old version show that it is still in use. lock (_recorded) { _recorded[(recorded.Name, recorded.Version)] = _recorded.GetValueOrDefault((recorded.Name, recorded.Version)) + 1; }
break; case DurableTaskEvents.ExecutionStalled stalled: // Reason is UnsupportedVersion or Diverged; the task stays pending until code which can replay it runs. Console.WriteLine($"{stalled.GrainContext.GrainId} {stalled.TaskId} stalled ({stalled.Reason}): {stalled.Exception.Message}"); break; } }
public void OnCompleted() { }
public void OnError(Exception error) { }
public void Dispose() => _subscription.Dispose();}VersionRecorded reports the execution, the change’s name, and the version: 0 for an
execution which earlier code ran past the check. ExecutionStalled reports the stalled
execution, the Reason, UnsupportedVersion or Diverged, and the Exception which
describes the stall.
Choose versions or new methods
Section titled “Choose versions or new methods”- Version in place for a change within a body, such as a replaced step, a changed argument, or an added notification, while executions are pending.
- Add a method for a new workflow shape. Schedule new work with the new method, and keep the old method until its pending tasks complete.
- Start a generation for a workflow which runs indefinitely: each generation ends and
starts its successor as a detached task, such as
subscription-{id}-{generation}, and a new generation starts on the code which is current when it starts.
