Build and recover an Orleans streaming application

This walkthrough uses the maintained Simple Streaming sample to follow an event from a producer grain to an implicit consumer. You first run with an in-memory provider, then switch to Azure Event Hubs and verify recovery.

Run the local path

From an empty directory:

git clone https://github.com/dotnet/orleans.git
cd orleans\samples\Streaming\Simple
dotnet build .\Streaming.sln

Start the silo, then the client in another terminal:

dotnet run --project .\SiloHost
dotnet run --project .\Client

The default local configuration registers the named in-memory stream provider on both hosts. The client asks a producer grain to publish integer events. The stream identity determines which implicitly subscribed consumer grain receives each event.

Observe these transitions in the silo log:

1. ConsumerGrain activates when the first matching event arrives.
2. Orleans supplies an IStreamSubscriptionHandleFactory.
3. The grain resumes the subscription with its observer.
4. OnNextAsync logs the event and its sequence token.

ImplicitStreamSubscriptionAttribute maps the stream namespace and identity to the consumer grain identity and activates that consumer when matching events arrive.

Trace the code

Read these files in order:

1. SiloHost/Program.cs registers PubSubStore and the stream provider.
2. Grains/ProducerGrain.cs obtains a typed stream and publishes events.
3. Grains/ConsumerGrain.cs declares the implicit subscription and attaches an observer.
4. Client/Program.cs drives the scenario.

The PubSubStore name is significant. Persistent stream providers use it for subscription metadata; register durable storage under that name in a production cluster.

Switch to a persistent provider

Create an Azure Event Hubs namespace, an event hub named my-path, a consumer group named my-group, and an Azure Storage account in a nonproduction subscription. Create Secrets.json in the sample directory with this content, replacing both values with connection strings. Keep Secrets.json outside source control.

{
  "DataConnectionString": "<Azure Storage connection string>",
  "EventHubConnectionString": "<Event Hubs namespace connection string>"
}

Restart the silo and client. The startup log should report the provider switch from in-memory streaming to Azure Event Hubs. The silo configures:

• Azure Table grain storage for PubSubStore;
• the Event Hubs stream provider;
• an Azure Table checkpointer; and
• the configured Event Hubs consumer group.

For deployed applications, replace local secrets with managed identity or your platform’s secret store and grant only the required data-plane permissions.

Verify recovery

Before this exercise, replace Guid.NewGuid() in Client/Program.cs with a fixed GUID so that each client run publishes to the same stream.

1. Run the silo normally.
2. Run the client with dotnet run --project .\Client -- --pause-after-checkpoint.
3. The client allows two 10-second checkpoint intervals, then stops the stream pulling agents while the producer grain continues publishing.
4. Record the frozen consumer sequence token and the next ten event numbers reported by the producer in the silo log. These events form a backlog beyond the stored checkpoint.
5. Terminate the silo process while the pulling agents remain paused, then stop the client. Leave Event Hubs and Azure Storage running.
6. Restart the silo with the same service ID, cluster ID, provider name, hub, and consumer group. The pulling agents load the stored checkpoint during startup.
7. Verify that the consumer receives the recorded pre-restart event numbers from the backlog. Their sequence tokens advance beyond the frozen token.
8. Restart the client to resume production after the backlog has drained.

Delivery guarantees are provider-specific. In this Event Hubs scenario, a consumer can observe duplicates around failures, so production handlers should be idempotent or deduplicate using an application-owned event identity. Sequence tokens represent provider delivery position; application-owned event identities support business deduplication.

Test failure boundaries

Repeat the recovery check while terminating a silo abruptly. Monitor queue lag, delivery retries, duplicate processing, poison-event handling, and checkpoint age. Then use streaming operations to define alerts and recovery procedures for the selected provider.

For provider choices and guarantees, continue with stream providers, delivery semantics, and pub-sub storage.