Azure Queue stream implementation

The Microsoft.Orleans.Streaming.AzureStorage package implements a persistent stream adapter over Azure Queue Storage. It uses the common persistent stream pulling architecture and supplies Azure-specific queue mapping, encoding, receive, delete, and configuration behavior.

Registration APIs

Silo and client builders expose SiloBuilderExtensions.AddAzureQueueStreams and ClientBuilderExtensions.AddAzureQueueStreams, respectively. The concise overload configures named AzureQueueOptions. Both configurator types can replace the queue data adapter; only the silo configurator exposes the cache and pulling-agent components which run on silos.

Applications can instead supply a keyed QueueServiceClient through configuration-driven provider registration. Configuration supports a service key, connection name, connection string, or queue-service URI. The ConfigureQueueServiceClient overloads and AzureQueueOptions.ClientOptions property are obsolete and should not be used in new code.

API: SiloBuilderExtensions.AddAzureQueueStreams, ClientBuilderExtensions.AddAzureQueueStreams, AzureQueueStreamProviderBuilder, and AzureQueueOptions. Implementation: silo registration, configuration-driven registration, and queue options.

Adapter behavior

AzureQueueAdapter is read-write and not rewindable. A producer encodes the stream identity, payload, and request context using an IQueueDataAdapter<T, U>, then sends an Azure Queue message to the mapped queue. The receiver assigns the local sequence token used for delivery and acknowledgement.

Because Azure Queue Storage does not expose a durable arbitrary stream offset, a non-null rewind token is rejected. AzureQueueDataAdapterV2 is the default encoding. Version 1 remains for compatibility with existing messages, not as the preferred format for new providers.

Source: AzureQueueAdapter and IAzureQueueDataAdapter.

Receive and acknowledgement

AzureQueueAdapterReceiver asks Azure Queue Storage for up to 32 visible messages at a time. Azure Queue Storage’s Get Messages operation makes a received message temporarily invisible; it does not delete it. The pulling agent decodes and delivers the batch through its cache and cursors. Only messages reported as delivered are removed using the Delete Message operation.

Rendering diagram.

If the receiver, agent, or silo fails before delete, the visibility timeout eventually expires and Azure can return the message again. Consumers must tolerate redelivery. If visibility expires while a message is still being processed, delete can fail because the pop receipt is no longer current.

Source: AzureQueueAdapterReceiver.

Queue mapping and defaults

When AzureQueueOptions.QueueNames is not supplied, provider configuration generates names from the Orleans service ID and provider name. The hash-ring queue mapper defaults to eight queues. Each owned queue has one pulling agent and one queue cache on its current silo.

Common persistent-stream defaults used by this provider are:

Setting	Default
HashRingStreamQueueMapperOptions.TotalQueueCount	8
StreamPullingAgentOptions.GetQueueMsgsTimerPeriod	100 ms
SimpleQueueCacheOptions.CacheSize	4,096 batch containers
StreamPullingAgentOptions.MaxEventDeliveryTime	1 minute
StreamPullingAgentOptions.StreamInactivityPeriod	30 minutes

The number of queues bounds pulling parallelism and ownership granularity. Changing queue names changes the physical partition set and must be treated as a data migration, not routine tuning.

Visibility and cache progress

Azure visibility and Orleans cache retention are different clocks:

• visibility controls when an undeleted Azure message can be received again;
• the queue cache controls how long the pulling agent retains a batch for active subscription cursors.

A AzureQueueOptions.MessageVisibilityTimeout value which is shorter than worst-case delivery increases duplicate receive and stale pop-receipt risk. An excessively long timeout delays recovery after agent failure. Choose values from measured delivery latency and failure objectives; operational tuning belongs with the stream provider guidance.

Extension and compatibility points

A custom IQueueDataAdapter<T, U> changes payload encoding while retaining the Azure transport. It preserves stream identity, the request-context values defined by the wire contract, and the sequence information used by consumers. Follow Customize persistent-stream data formats for a compiling implementation and versioned rollout guidance.

Configuration binding behavior is tested by AzureQueueStreamProviderBuilderTests. Adapter acknowledgement and cursor behavior are covered by AzureQueueAdapterTests.