Observe journaled state and stream drafts
Watch streams deliver the committed changes made to a grain’s journaled state. A grain exposes them to callers as IAsyncEnumerable<T> methods. IDurableLog<T> adds an append-only log whose readers resume from a sequence number, and drafts which stream volatile content, such as model output, before its durable entry exists.
Configure journaling as described in Use durable state.
Expose a watch from a grain
Section titled “Expose a watch from a grain”This grain records a conversation in a durable log and streams each response through a draft:
public interface IConversationGrain : IGrainWithStringKey{ ValueTask<long> Ask(string question, CancellationToken cancellationToken);
// Ends the conversation. Every watch ends after the final entry. ValueTask End(CancellationToken cancellationToken);
// Committed history from a position. An epoch of zero reads the current history. IAsyncEnumerable<LogEvent<string>> WatchAsync(long epoch, long nextSequence, CancellationToken cancellationToken = default);
// Committed history and streaming drafts. IAsyncEnumerable<LogEvent<string>> WatchLiveAsync( long epoch, long nextSequence, LogDraftId? draftId, CancellationToken cancellationToken = default);
IAsyncEnumerable<ValueChange<string>> WatchStatusAsync(CancellationToken cancellationToken = default);}public sealed class ConversationGrain( IDurableStateManager stateManager, IResponseGenerator generator) : Grain, IConversationGrain{ private readonly IDurableLog<string> _messages = stateManager.GetOrAddLog<string>("messages"); private readonly IDurableValue<string> _status = stateManager.GetOrAddValue<string>("status");
public async ValueTask<long> Ask(string question, CancellationToken cancellationToken) { _messages.Append(question); _status.Value = "responding"; await stateManager.WriteStateAsync(cancellationToken);
// Readers which include drafts receive each token while the response is generated. using var draft = _messages.BeginDraft(); var response = new StringBuilder(); try { await foreach (var token in generator.GenerateAsync(question, cancellationToken)) { draft.Write(token); response.Append(token); } } catch (Exception exception) { draft.Abandon($"The response failed: {exception.Message}"); _status.Value = "failed"; await stateManager.WriteStateAsync(CancellationToken.None); throw; }
// The committed entry replaces the draft's tokens. var sequence = draft.Commit(response.ToString()); _status.Value = "idle"; await stateManager.WriteStateAsync(cancellationToken); return sequence; }
public async ValueTask End(CancellationToken cancellationToken) { _messages.Seal(); _status.Value = "ended"; await stateManager.WriteStateAsync(cancellationToken); }
public IAsyncEnumerable<LogEvent<string>> WatchAsync(long epoch, long nextSequence, CancellationToken cancellationToken = default) => _messages.ReadAsync(nextSequence, epoch, cancellationToken: cancellationToken);
public IAsyncEnumerable<LogEvent<string>> WatchLiveAsync( long epoch, long nextSequence, LogDraftId? draftId, CancellationToken cancellationToken = default) => _messages.ReadAsync(nextSequence, epoch, includeDrafts: true, draftId, cancellationToken);
public IAsyncEnumerable<ValueChange<string>> WatchStatusAsync(CancellationToken cancellationToken = default) => _status.WatchAsync(cancellationToken: cancellationToken);}Orleans enumerates a grain’s IAsyncEnumerable<T> result through interleaving calls, so watches keep delivering while Ask streams a response. Enumerate a watch in the owning activation’s execution context, as grain methods do. End seals the log, which ends every watch of it.
Committed delivery
Section titled “Committed delivery”Watch streams deliver only durable state. Each change is tagged with an activation-local write version: the version of the journal write which captures it. Each write captures every state, persists what it captured, and then publishes one watermark for its version to every watcher, which delivers the changes that write made durable. A change staged by another turn after the write captured its entries waits for the following write, so readers never observe it early. The same rule governs other acknowledgements: a durable task completion source’s task completes after the write which captured its result.
Durable state takes ownership of the values, items, and entries passed to it, and the journal records each change when it is made. To change one, assign or append a new instance. Watch streams deliver those same instances, so an observed change matches what its write stored.
Watch a value
Section titled “Watch a value”WatchAsync watches an IDurableValue<T>. The stream begins with the current value once it is durable, so a watch which begins while the value is staged receives it after the next write, and then delivers the latest value after each write which changes it. Each ValueChange<T> carries the value and the write version which made it durable. A slow reader receives the newest value and skips intermediate values, so a value watch retains no backlog.
A value which nobody watches publishes nothing and allocates nothing for observation. The manager creates its observer state when the first stream begins, once it has recovered its state.
Read a durable log
Section titled “Read a durable log”IDurableLog<T>, created with GetOrAddLog, assigns each appended entry the next value of a durable, monotonically increasing sequence number:
- Append stages an entry and returns its sequence number.
- FirstSequence, NextSequence, and CommittedSequence describe the retained, appended, and durable entries. Serve paged history from entries before
CommittedSequence. - TrimBefore discards old entries to bound retention. The retained entries remain valid.
- Clear discards every entry to replace them, for example with a compacted summary. A clear consumes one sequence number, so sequence numbers are never reused within a history.
- Epoch identifies the history. The first change after the log is created, or after the grain’s journal is deleted, begins a history with a new random epoch, which is journaled with it. Deleting the journal restarts sequence numbers at zero in the new history.
- Seal durably ends the log. A sealed log rejects
Append,Clear, andBeginDraft, abandons open drafts, and can still be trimmed. Only deleting the journal unseals it.
ReadAsync reads retained entries from a position and then waits for new ones. Readers re-read entries from the log at their position instead of buffering them, so a slow reader retains no entries. A position is an epoch and a sequence number. Every event carries the epoch of its history in Epoch, so an epoch and a sequence number form a complete replay checkpoint. Pass zero as the epoch to read the current history. Besides entries, a stream delivers these LogEventKind events:
| Event | Meaning |
|---|---|
Trimmed | TrimBefore discarded the entries before Sequence. Every active reader receives it, including readers which have read every entry. A reader positioned before Sequence continues from it. |
Reset | The entries the reader observed were replaced: Clear discarded them, the journal was deleted, the reader resumed with another history’s epoch, or the reader’s position follows the end of the log. Discard the entries read before it. The stream continues from Sequence in the history identified by Epoch. |
CaughtUp | The stream has delivered every entry which was committed when the read began. It’s delivered once. |
Sealed | The log ended before Sequence. It’s the stream’s final event. |
A stream which begins before the first retained entry starts with Trimmed. A stream which begins with another history’s epoch, or at or before a cleared sequence number, starts with Reset, including after the grain reactivates, while a stream which begins after the clear doesn’t. Streams receive entries, Trimmed, Reset, and Sealed once the append, trim, clear, or seal is durable, so every entry a reader receives is a valid resume point. A reader positioned in entries which an uncommitted trim has discarded waits for the trim’s write. To resume, pass the epoch of the latest event and the position after the latest entry received.
Sealed distinguishes a log which ended from an activation which stopped. When the activation stops, streams end without it.
Stream drafts
Section titled “Stream drafts”BeginDraft opens an IDurableLogDraft<T>:
Writedelivers a volatile item immediately to streams which include drafts: passincludeDrafts: truetoReadAsync. The draft holds its items in memory only while it is open.Commitcloses the draft and appends zero or more entries which replace its items. The entries form one journal entry, so one write makes all of them durable, and an entry which can’t be encoded leaves the draft open without appending any entry. Readers receiveDraftCommitted, whoseSequenceandCountgive the range of committed entries, followed by the entries.Commitwith no entries closes the draft cleanly.Abandoncloses the draft without entries and reportsDraftAbandonedwith the owner’sMessage. Disposing an open draft abandons it, so a failed operation, such as a model stream which fails mid-response, releases its draft.
A stream reports DraftCommitted once the committed entries are durable. A stream which includes drafts and begins while a draft is open receives the draft’s items from position zero. When a draft closes, its items are released: a reader which hasn’t yet read every item receives DraftCommitted or DraftAbandoned next and never receives the remaining items. The committed entries carry the final content.
DraftAbandoned carries a LogDraftAbandonReason:
| Reason | Cause |
|---|---|
Abandoned | The owner called Abandon. |
Disposed | The owner disposed the open draft. |
JournalDeleted | The log’s journal was deleted. |
Sealed | The log was sealed while the draft was open. |
PreviousActivation | Another activation opened the draft and the log retains no commit of it, so its content was lost. |
Unknown | This activation neither retains a commit of the draft nor records why it closed. |
The log closes a draft when its journal is deleted or the log is sealed. Its owner then holds a closed draft, whose Write and Commit throw an InvalidOperationException that states why it closed. Check IsOpen before committing a draft which the log can close.
Each LogDraftId includes an identifier for the log instance, and therefore the activation, which opened the draft. A reconnecting reader passes its position and the draft it was viewing. If that draft is still open, the stream repeats its items from position zero. The log journals a draft’s identifier with its committed entries, so when the draft committed, the stream reports DraftCommitted with the range which replaced it, in any activation, while the log retains the first of those entries. If this activation abandoned the draft, the stream reports DraftAbandoned with its reason; each log records the reasons for its 16 most recently abandoned drafts and reports Unknown for older ones. A draft which a previous activation opened and never committed is reported as PreviousActivation; regenerate its content from the last committed entry.
This client renders committed history and in-progress drafts and tracks its resume position:
public static async Task RenderAsync(IConversationGrain conversation, CancellationToken cancellationToken){ // Every entry is durable, so the epoch and the position after the latest entry are a valid resume point. var epoch = 0L; var nextSequence = 0L; var history = new SortedList<long, string>(); var drafts = new Dictionary<LogDraftId, StringBuilder>(); await foreach (var logEvent in conversation.WatchLiveAsync(epoch, nextSequence, draftId: null, cancellationToken)) { epoch = logEvent.Epoch; switch (logEvent.Kind) { case LogEventKind.Entry: history[logEvent.Sequence] = logEvent.Item!; nextSequence = logEvent.Sequence + 1; Console.WriteLine($"[{logEvent.Sequence}] {logEvent.Item}"); break; case LogEventKind.CaughtUp: Console.WriteLine($"Loaded {history.Count} messages."); break; case LogEventKind.Trimmed: // Older entries were pruned. Later entries remain valid. while (history.Count > 0 && history.Keys[0] < logEvent.Sequence) { history.RemoveAt(0); }
nextSequence = Math.Max(nextSequence, logEvent.Sequence); break; case LogEventKind.Reset: // The entries were replaced, for example by a compacted summary or by deleting the journal. history.Clear(); nextSequence = logEvent.Sequence; break; case LogEventKind.DraftItem: var draft = logEvent.Sequence == 0 ? drafts[logEvent.DraftId!.Value] = new StringBuilder() : drafts[logEvent.DraftId!.Value]; draft.Append(logEvent.Item); break; case LogEventKind.DraftCommitted: // The draft's tokens are superseded by the entries in [Sequence, Sequence + Count). drafts.Remove(logEvent.DraftId!.Value); break; case LogEventKind.DraftAbandoned: drafts.Remove(logEvent.DraftId!.Value); Console.WriteLine($"Response abandoned ({logEvent.AbandonReason}): {logEvent.Message}"); break; case LogEventKind.Sealed: Console.WriteLine("The conversation ended."); break; } }}Failure and shutdown
Section titled “Failure and shutdown”A failed journal write fences the manager and requests grain deactivation. Watch streams then fault with an InvalidOperationException whose inner exception is the storage failure, and readers never receive the changes the failed write contained. Readers reconnect to the new activation, which recovers the durable state. When the manager stops normally, streams complete. Cancelling a stream’s token ends that stream.
Deleting the grain’s journal delivers the default value to value watches, or Reset for a log reader which observed entries, and abandons open drafts. A reader following a draft whose committed entries the deletion discarded receives DraftAbandoned with JournalDeleted.
Test code which uses a log
Section titled “Test code which uses a log”Test grains which use durable state against real journaled state managers. An InProcessTestCluster configured with volatile journal storage keeps journals in silo memory, and DeactivateAsync exercises recovery in a new activation. Configure the journal format as production silos do, or, as this test does, with reflection-based JSON metadata:
public static async Task AskCommitsTheResponseAndSurvivesReactivation(){ var builder = new InProcessTestClusterBuilder(initialSilosCount: 1); builder.ConfigureSilo((options, siloBuilder) => { // Real journaled state managers, with journals held in silo memory. Reflection-based JSON metadata // suits tests; production silos register source-generated metadata. siloBuilder.AddJournaling() .UseJsonJournalFormat(new DefaultJsonTypeInfoResolver()) .AddVolatileJournalStorage(); siloBuilder.Services.AddSingleton<IResponseGenerator>(new FixedResponseGenerator("Hello", ", world")); });
await using var cluster = builder.Build(); await cluster.DeployAsync(); var conversation = cluster.Client.GetGrain<IConversationGrain>("test"); var sequence = await conversation.Ask("Hi", CancellationToken.None);
// A new activation recovers the conversation from the journal. await cluster.DeactivateAsync(conversation); using var timeout = new CancellationTokenSource(TimeSpan.FromSeconds(30)); var history = new List<string>(); await foreach (var logEvent in conversation.WatchAsync(epoch: 0, nextSequence: 0, timeout.Token)) { if (logEvent.Kind is LogEventKind.Entry) { history.Add(logEvent.Item!); } else if (logEvent.Kind is LogEventKind.CaughtUp) { break; } }
if (sequence != 1 || !history.SequenceEqual(["Hi", "Hello, world"])) { throw new InvalidOperationException("The conversation did not record the committed response."); }}Host code which takes an IDurableLog<T> in a test grain to test it the same way. A hand-written IDurableLog<T> substitute doesn’t reproduce durability, drafts, or read streams.
