Grain request scheduling

Each grain activation executes one turn at a time. Orleans runs a request until it completes or reaches an incomplete await, then later schedules its continuation as another turn. Two turns never execute in parallel on the same activation.

By default, an activation doesn’t start a different request while the current request is incomplete. This non-reentrant model makes mutable grain state easier to reason about:

public sealed class CounterGrain : Grain, ICounterGrain
{
    private int _value;

    public async ValueTask<int> AddAfterDelay(int amount)
    {
        await Task.Delay(TimeSpan.FromMilliseconds(100));
        _value += amount;
        return _value;
    }
}

Although the method yields at await, another request doesn’t observe or change _value before this request completes.

Avoid blocking

Never synchronously block on incomplete tasks from grain code. .Result, .Wait(), WaitAll, and GetAwaiter().GetResult() can deadlock an activation and consume thread-pool threads. Use await.

Normal await resumes grain code on the activation scheduler. Don’t use ConfigureAwait(false) in grain methods because the continuation can escape the grain scheduler. General-purpose libraries can use it internally; grain code returns to the grain scheduler when it awaits the library normally.

Interleaving and reentrancy

Interleaving lets Orleans start or resume another request while an earlier request is incomplete. The activation is still single-threaded, but turns from different requests can alternate. Re-check assumptions about mutable state after every await.

Mechanism	Scope
ReentrantAttribute	All requests to the grain class can interleave.
AlwaysInterleaveAttribute	The marked interface method can interleave with any request.
ReadOnlyAttribute	Marked read-only methods can interleave with other read-only methods.
MayInterleaveAttribute	A predicate inspects each request.
AllowCallChainReentrancy	A scoped call chain can call back into an activation already in that chain.

Use the narrowest mechanism that solves the problem. Reentrancy can improve throughput for I/O-heavy grains and prevent cyclic call deadlocks, but it also allows state to change between turns.

Reentrant grains

[Reentrant]
public sealed class CatalogGrain : Grain, ICatalogGrain
{
    public async ValueTask<Product> GetProduct(string productId)
    {
        return await LoadProduct(productId);
    }
}

Code in different requests doesn’t run simultaneously, but multiple incomplete calls can make progress by alternating turns.

Method-level interleaving

public interface IStatusGrain : IGrainWithStringKey
{
    Task Update(Status status);

    [ReadOnly]
    ValueTask<Status> Get();

    [AlwaysInterleave]
    ValueTask Ping();
}

ReadOnly is a scheduling promise. Active read-only requests can interleave with one another and collectively exclude writable requests until every read-only request completes, unless another interleaving policy permits overlap. Keep grain state unchanged from a read-only method.

Predicate-based interleaving

MayInterleave names a predicate that accepts IInvokable. Use its accessor methods; there is no Arguments property:

[MayInterleave(nameof(CanInterleave))]
public sealed class WorkGrain : Grain, IWorkGrain
{
    public static bool CanInterleave(IInvokable request)
    {
        return request.GetArgumentCount() == 1
            && request.GetArgument(0) is WorkItem { IsReadOnly: true };
    }

    public Task Process(WorkItem item) => Task.CompletedTask;
}

Keep predicates deterministic, fast, and side-effect free.

Call-chain reentrancy

Use call-chain reentrancy when a known call path must call back into the initiating activation:

public async Task JoinRoom(string roomName)
{
    using var scope = RequestContext.AllowCallChainReentrancy();

    IChatRoomGrain room =
        GrainFactory.GetGrain<IChatRoomGrain>(roomName);

    await room.Join(this.AsReference<IUserGrain>());
}

The scope permits callbacks associated with that call chain until it is disposed. It is narrower than marking the entire grain reentrant.

Timers and external work

Grain timer callbacks participate in scheduling. Callbacks don’t interleave by default; configure GrainTimerCreationOptions.Interleave when intentional.

To run work from another thread under the activation’s scheduling, see Run background work as a grain request.

Use Run only to isolate unavoidable synchronous blocking or CPU work from the Orleans scheduler. Don’t access grain state from the thread-pool delegate. Await it and update grain state after execution resumes on the activation scheduler. See External tasks and grains.

Run background work as a grain request

RunRequestAsync runs a callback as a request on the activation. Call it from any thread, such as a stream, messaging, or library callback, to apply results to grain state under the same scheduling as grain calls:

public sealed class QuoteGrain(IPriceFeed feed) : Grain, IQuoteGrain
{
    private IDisposable? _subscription;
    private decimal _price;

    public override Task OnActivateAsync(CancellationToken cancellationToken)
    {
        // The feed invokes OnPrice on its own threads.
        _subscription = feed.Subscribe(this.GetPrimaryKeyString(), OnPrice);
        return Task.CompletedTask;
    }

    public override Task OnDeactivateAsync(DeactivationReason reason, CancellationToken cancellationToken)
    {
        _subscription?.Dispose();
        return Task.CompletedTask;
    }

    public ValueTask<decimal> GetPrice() => ValueTask.FromResult(_price);

    // Runs the update as a request on this activation, where it can safely access grain state.
    private Task OnPrice(decimal price, CancellationToken cancellationToken) =>
        this.RunRequestAsync(
            static (state, _) => state.Grain._price = state.Price,
            (Grain: this, Price: price),
            cancellationToken);
}

With the default LocalRequestOptions, the request:

• Follows the grain’s reentrancy rules and queues behind the current request on a non-reentrant grain.
• Passes through incoming grain call filters, which observe it as IGrainRequestInvoker.RunRequestAsync with no arguments.
• Starts a new call chain. RequestContext values from the calling thread don’t flow into the callback.
• Restarts the activation’s idle collection period.
• Bypasses MaxEnqueuedRequestsHardLimit.

The callback can be synchronous, as in this example, or return a Task or Task<TResult>. The returned task completes with the callback’s result or exception. The callback receives the cancellation token passed to RunRequestAsync, and cancelling it while the request is queued cancels the request. If the activation deactivates before the request runs, the task faults with OrleansMessageRejectionException.

Pass the grain and other values through the state argument and use a static lambda, so the call doesn’t allocate a closure. On a non-reentrant grain, awaiting the returned task from another request on the same activation deadlocks, because the queued request waits for the current one to complete.

Configure the request

Pass LocalRequestOptions to change how the request is scheduled. Set KeepAlive to false for frequent background work that shouldn’t keep an otherwise idle activation in memory. Set Interleave to true to run the request between the turns of other requests, as with an [AlwaysInterleave] method:

// Heartbeats don't extend the activation's lifetime, and they run between the turns of other requests.
private Task OnHeartbeat(DateTimeOffset timestamp, CancellationToken cancellationToken) =>
    this.RunRequestAsync(
        static (state, _) => state.Grain._lastSeen = state.Timestamp,
        (Grain: this, Timestamp: timestamp),
        new LocalRequestOptions { KeepAlive = false, Interleave = true },
        cancellationToken);

Deliver a custom request

ReceiveLocalRequest delivers an ILocalRequest to the activation with the same scheduling as RunRequestAsync. Call filters, logs, and traces observe the request’s own interface name, method name, and arguments, so a custom request can describe the work to them. The activation passes the request’s response to Complete, or an exception if the activation rejects the request. The request reports that outcome, including any failure, to its observers:

// A request which increments IngestGrain's counter. Call filters observe it as IIngestRequests.Increment.
internal sealed class IncrementRequest : Request, ILocalRequest
{
    private readonly TaskCompletionSource _completion = new(TaskCreationOptions.RunContinuationsAsynchronously);
    private IngestGrain? _grain;

    public Task Completion => _completion.Task;

    // Receives the outcome once the request and its call filters complete, or when the activation rejects the request.
    // The task keeps the first outcome.
    public void Complete(Response response)
    {
        if (response.Exception is { } exception)
        {
            _completion.TrySetException(exception);
        }
        else
        {
            _completion.TrySetResult();
        }
    }

    public override object GetTarget() => _grain!;

    public override void SetTarget(ITargetHolder holder) => _grain = (IngestGrain?)holder.GetTarget();

    protected override ValueTask InvokeInner()
    {
        _grain!.Increment();
        return default;
    }

    public override string GetInterfaceName() => "IIngestRequests";

    public override string GetMethodName() => "Increment";

    public override string GetActivityName() => "IIngestRequests/Increment";

    public override Type GetInterfaceType() => typeof(IIngestGrain);

    public override System.Reflection.MethodInfo GetMethod() => typeof(IIngestGrain).GetMethod(nameof(IIngestGrain.GetCount))!;

    public override void Dispose() { }
}

public static class IngestRequests
{
    // Delivers the request to the activation and returns a task which completes with the request's outcome.
    public static Task Increment(IGrainContext activation)
    {
        var request = new IncrementRequest();
        activation.ReceiveLocalRequest(request, default);
        return request.Completion;
    }
}