Operate and tune Orleans streams

Tune from measured lag, throughput, failures, and memory pressure. Provider defaults are starting points, not universal production settings.

Backpressure

Persistent providers pull events into silo-side caches and deliver them to subscriptions. A consumer applies backpressure by not completing OnNextAsync until it has accepted responsibility for the item. Slow consumers can therefore increase retained queue data, cache pressure, and end-to-end lag.

Keep consumer turns bounded:

• Persist required state and complete promptly.
• Move independent long-running work behind an explicitly modeled handoff.
• Avoid synchronous blocking and unbounded parallel work.
• Scale by choosing enough provider queues or partitions and by distributing stream keys. One hot stream targeting one stateful grain remains limited by that grain’s processing rate.

Backpressure on the consumer side doesn’t imply producer completion. OnNextAsync on the producer reports provider acceptance, not that downstream consumers caught up.

Protect a slow Event Hubs consumer

The Event Hubs provider maintains an independent cache cursor for each subscription, so a fast subscription can continue while another subscription falls behind. All cursors for an Event Hubs partition share the same silo-side cache, however. By default, the provider uses a weighted average of the pressure contributions from those cursors: contributions at or above the flow-control threshold receive three times the weight of lower-pressure contributions. Repeated contributions from faster subscriptions can still outweigh a small number of lagging subscriptions.

DataMinTimeInCache and DataMaxAgeInCache control time-based cache eviction; they don’t guarantee that every subscription remains within the cache. If eviction advances past a lagging cursor, delivery reports Item not found in cache.

The slow-consuming monitor lets a single observed lagging cursor apply cache pressure instead of averaging that pressure with faster cursors:

configurator.ConfigureCachePressuring(builder => builder.Configure(options =>
{
    options.AveragingCachePressureMonitorFlowControlThreshold = null;
    options.SlowConsumingMonitorFlowControlThreshold = 0.7;
    options.SlowConsumingMonitorPressureWindowSize =
        TimeSpan.FromSeconds(10);
}));

The monitor starts calculating cursor pressure after the partition cache spans at least 10,000 Event Hubs sequence numbers. In this example, when Orleans reads the next cached item for a subscription whose cursor is more than 70% of the cache span behind the newest cached position, the provider stops new Event Hubs reads for at least 10 seconds. Tune both values from measured lag and processing time. The slow-consuming policy intentionally limits partition ingestion to protect the slowest observed subscription: cache misses are less likely, but end-to-end lag for every subscription can increase and backlog can move into Event Hubs. Ensure that Event Hubs retention can absorb that backlog and that sustained ingress doesn’t exceed the slowest required subscription’s capacity.

Pressure is sampled as Orleans advances subscriptions through cached items, so keep consumer turns bounded to keep detection current. First reduce CPU saturation and hot-grain bottlenecks. If workloads need independent throughput or retention policies, isolate them using separate Orleans stream providers and Event Hubs consumer groups instead of coupling them through one partition cache.

Tune the pulling pipeline

Persistent providers expose common configuration through their stream configurators:

• GetQueueMsgsTimerPeriod trades polling frequency against latency and service calls.
• BatchContainerBatchSize controls how many queue batches are grouped for delivery.
• MaxEventDeliveryTime bounds delivery attempts before the configured failure handler is involved.
• CacheSize controls item capacity for providers using the simple queue cache.

Provider-specific controls matter as much as common controls: QueueNames, Event Hubs partitions and cache-pressure settings, Redis ReadCount and retention, NATS BatchSize and PartitionCount, and ADO.NET visibility, expiry, and dead-letter settings.

Change one bottleneck at a time. More queues can increase parallelism but also broker cost, polling load, cache memory, and rebalance work. Reducing polling delay can lower latency while increasing empty reads.

Observe health

Export Orleans meters and correlate them with broker metrics and application event IDs. Useful Orleans instruments include:

Signal	Instruments
Active topology	orleans-streams-pubsub-producers, orleans-streams-pubsub-consumers, orleans-streams-persistent-stream-pulling-agents
Throughput	orleans-streams-persistent-stream-messages-read, orleans-streams-persistent-stream-messages-sent, orleans-streams-queue-messages-received
Queue health	orleans-streams-queue-read-failures, orleans-streams-queue-read-exceptions, orleans-streams-queue-oldest-message-enqueue-age
Cache health	orleans-streams-queue-cache-size, orleans-streams-queue-cache-length, orleans-streams-queue-cache-pressure, orleans-streams-queue-cache-under-pressure
Memory	orleans-streams-block-pool-total-memory, orleans-streams-block-pool-available-memory

Alert on sustained oldest-message age, cache pressure, read failures, repeated consumer exceptions, dead-letter growth, and a mismatch between expected and active subscription counts. Broker-side backlog and retention alarms remain necessary because Orleans can only report what its adapters observe.

For the components behind these signals, see Orleans streams implementation.