Operate and tune Orleans streams
Tune from measured lag, throughput, failures, and memory pressure. Provider defaults are starting points, not universal production settings.
Backpressure
Section titled “Backpressure”Persistent providers pull events into silo-side caches and deliver them to subscriptions. A consumer applies backpressure by not completing OnNextAsync until it has accepted responsibility for the item. Slow consumers can therefore increase retained queue data, cache pressure, and end-to-end lag.
Keep consumer turns bounded:
- Persist required state and complete promptly.
- Move independent long-running work behind an explicitly modeled handoff.
- Avoid synchronous blocking and unbounded parallel work.
- Scale by choosing enough provider queues or partitions and by distributing stream keys. One hot stream targeting one stateful grain remains limited by that grain’s processing rate.
Backpressure on the consumer side doesn’t imply producer completion. OnNextAsync on the producer reports provider acceptance, not that downstream consumers caught up.
Tune the pulling pipeline
Section titled “Tune the pulling pipeline”Persistent providers expose common configuration through their stream configurators:
- GetQueueMsgsTimerPeriod trades polling frequency against latency and service calls.
- BatchContainerBatchSize controls how many queue batches are grouped for delivery.
- MaxEventDeliveryTime bounds delivery attempts before the configured failure handler is involved.
- CacheSize controls item capacity for providers using the simple queue cache.
Provider-specific controls matter as much as common controls: QueueNames, Event Hubs partitions and cache-pressure settings, Redis ReadCount and retention, NATS BatchSize and PartitionCount, and ADO.NET visibility, expiry, and dead-letter settings.
Change one bottleneck at a time. More queues can increase parallelism but also broker cost, polling load, cache memory, and rebalance work. Reducing polling delay can lower latency while increasing empty reads.
Observe health
Section titled “Observe health”Export Orleans meters and correlate them with broker metrics and application event IDs. Useful Orleans instruments include:
| Signal | Instruments |
|---|---|
| Active topology | orleans-streams-pubsub-producers, orleans-streams-pubsub-consumers, orleans-streams-persistent-stream-pulling-agents |
| Throughput | orleans-streams-persistent-stream-messages-read, orleans-streams-persistent-stream-messages-sent, orleans-streams-queue-messages-received |
| Queue health | orleans-streams-queue-read-failures, orleans-streams-queue-read-exceptions, orleans-streams-queue-oldest-message-enqueue-age |
| Cache health | orleans-streams-queue-cache-size, orleans-streams-queue-cache-length, orleans-streams-queue-cache-pressure, orleans-streams-queue-cache-under-pressure |
| Memory | orleans-streams-block-pool-total-memory, orleans-streams-block-pool-available-memory |
Alert on sustained oldest-message age, cache pressure, read failures, repeated consumer exceptions, dead-letter growth, and a mismatch between expected and active subscription counts. Broker-side backlog and retention alarms remain necessary because Orleans can only report what its adapters observe.
For the components behind these signals, see Orleans streams implementation.