Custom grain storage

In the tutorial on declarative actor storage, you learned how to allow grains to store their state in an Azure table using one of the built-in storage providers. While Azure is a great place to store your data, many alternatives exist. There are so many that supporting them all isn’t feasible. Instead, Orleans is designed to let you easily add support for your preferred storage by writing a custom grain storage provider.

In this tutorial, you’ll build a simple file-based grain storage provider for a local, single-silo application. It stores binary state records on the local filesystem and uses persisted ETags for basic optimistic concurrency checks.

Get started

An Orleans grain storage provider is a class that implements IGrainStorage, included in the Microsoft.Orleans.Core NuGet package. This sample also implements ILifecycleParticipant<T>, where the lifecycle type is ISiloLifecycle, so that it can initialize during the silo lifecycle. Start by creating a class named FileGrainStorage; the complete, compiling implementation appears at the end of the tutorial.

Each method implements the corresponding method in the IGrainStorage interface, accepting a generic type parameter for the underlying state type. The methods are:

• IGrainStorage.ReadStateAsync: Reads the state of a grain.
• IGrainStorage.WriteStateAsync: Writes the state of a grain.
• IGrainStorage.ClearStateAsync: Clears the state of a grain.

All three methods receive the same arguments:

Argument	Meaning
stateName	The logical name of this state record. For IPersistentState<T>, this is the state name configured by PersistentStateAttribute; legacy Grain<T> state uses state. A grain can have multiple named state records, so include this value in the storage key. Older versions of the interface called this argument grainType, but it doesn’t describe the state object’s .NET type.
grainId	The complete Orleans grain identity, including its grain type and primary key. Combine it with stateName to identify a record. Don’t key records using only the primary key, because different grain types can use the same key.
grainState	The state container Orleans passes to the provider. Its State property contains the application state, ETag carries the provider’s optimistic-concurrency token, and RecordExists indicates whether the record exists. A read populates these properties; a write or clear updates them to reflect the completed operation.
T	The declared .NET type of the state payload. Use T (or typeof(T)) for serialization and type metadata. It isn’t a record identifier and can be shared by many grains and named state records.

Therefore, stateName and grainState.State?.GetType().Name aren’t interchangeable. The first identifies one of a grain’s logical state records and is stable independently of the payload implementation. The second is only the runtime type name of the current payload; it can be null, can differ from typeof(T) when polymorphism is involved, and can change during a refactoring. The sample uses stateName and grainId for identity and delegates payload type handling to the configured serializer.

The ILifecycleParticipant<T>.Participate method subscribes to the silo’s lifecycle.

Before starting the implementation, create an options class containing the root directory where grain state files are persisted. Create an options file named FileGrainStorageOptions containing the following:

using Orleans.Runtime;
using Orleans.Storage;

namespace GrainStorage;

public sealed class FileGrainStorageOptions : IStorageProviderSerializerOptions
{
    public required string RootDirectory { get; set; }

    public required IGrainStorageSerializer GrainStorageSerializer { get; set; }
}

internal sealed class FileGrainStorageOptionsValidator(
    FileGrainStorageOptions options,
    string name) : IConfigurationValidator
{
    public void ValidateConfiguration()
    {
        if (string.IsNullOrWhiteSpace(options.RootDirectory))
        {
            throw new OrleansConfigurationException(
                $"Invalid configuration for {nameof(FileGrainStorage)} with name {name}. " +
                $"{nameof(FileGrainStorageOptions)}.{nameof(FileGrainStorageOptions.RootDirectory)} is required.");
        }

        if (File.Exists(options.RootDirectory))
        {
            throw new OrleansConfigurationException(
                $"Invalid configuration for {nameof(FileGrainStorage)} with name {name}. " +
                $"{nameof(FileGrainStorageOptions)}.{nameof(FileGrainStorageOptions.RootDirectory)} must identify a directory.");
        }

    }
}

With the options class created, explore the constructor parameters of the FileGrainStorage class:

• storageName: Specifies which grains should use this storage provider through StorageProviderAttribute, for example, [StorageProvider(ProviderName = "File")].
• options: The options class just created.
• clusterOptions: The cluster options used for retrieving the ServiceId.
• activatorProvider: Creates missing or cleared state instances using the same activation rules as Orleans serialization.

Initialize the storage

To initialize the storage, subscribe to the ServiceLifecycleStage.ApplicationServices stage with an onStart function. Consider the following ILifecycleParticipant<T>.Participate implementation:

public void Participate(ISiloLifecycle lifecycle) =>
    lifecycle.Subscribe(
        observerName: OptionFormattingUtilities.Name<FileGrainStorage>(storageName),
        stage: ServiceLifecycleStage.ApplicationServices,
        onStart: _ =>
        {
            Directory.CreateDirectory(_rootDirectory);
            return Task.CompletedTask;
        });

The onStart function creates the root directory before application services use the provider.

Also, derive a fixed-length filename from length-delimited service ID, grain type, grain key, and state name components:

private string GetRecordPath(string stateName, GrainId grainId)
{
    using var identity = new MemoryStream();
    WriteIdentityComponent(identity, Encoding.UTF8.GetBytes(_serviceId));
    WriteIdentityComponent(identity, grainId.Type.AsSpan());
    WriteIdentityComponent(identity, grainId.Key.AsSpan());
    WriteIdentityComponent(identity, Encoding.UTF8.GetBytes(stateName));

    var hash = SHA256.HashData(identity.GetBuffer().AsSpan(0, checked((int)identity.Length)));
    return Path.Combine(_rootDirectory, $"{Convert.ToHexString(hash)}.grain");
}

private static void WriteIdentityComponent(Stream destination, ReadOnlySpan<byte> value)
{
    Span<byte> length = stackalloc byte[sizeof(int)];
    System.Buffers.Binary.BinaryPrimitives.WriteInt32BigEndian(length, value.Length);
    destination.Write(length);
    destination.Write(value);
}

Read state

To read a grain state, derive its record path and read the file if it exists.

public async Task ReadStateAsync<T>(
    string stateName,
    GrainId grainId,
    IGrainState<T> grainState)
{
    var record = await TryReadRecordAsync(GetRecordPath(stateName, grainId)).ConfigureAwait(false);
    if (record is null)
    {
        ResetState(grainState);
        return;
    }

    grainState.State = options.GrainStorageSerializer.Deserialize<T>(new BinaryData(record.Value.Payload));
    grainState.ETag = record.Value.ETag;
    grainState.RecordExists = true;
}

The record header contains a persisted opaque ETag. Set RecordExists to indicate whether the read found a record, and reset all three state-container properties when the record is absent.

Read the payload as bytes and deserialize it using IStorageProviderSerializerOptions.GrainStorageSerializer, preserving arbitrary serializer output without text conversion.

Write state

Writing the state is similar to reading the state.

public async Task WriteStateAsync<T>(
    string stateName,
    GrainId grainId,
    IGrainState<T> grainState)
{
    var path = GetRecordPath(stateName, grainId);
    var existingRecord = await TryReadRecordAsync(path).ConfigureAwait(false);
    if (existingRecord is not null)
    {
        ValidateETag<T>("WriteState", grainId, grainState.ETag, existingRecord.Value.ETag);
    }
    else if (grainState.ETag is not null)
    {
        throw CreateInconsistentStateException<T>("WriteState", grainId);
    }

    var etag = Guid.NewGuid().ToString("N");
    var payload = options.GrainStorageSerializer.Serialize(grainState.State).ToArray();
    await File.WriteAllBytesAsync(path, CreateRecord(etag, payload)).ConfigureAwait(false);

    grainState.ETag = etag;
    grainState.RecordExists = true;
}

Use IStorageProviderSerializerOptions.GrainStorageSerializer to produce the binary payload. Compare the caller’s ETag with the persisted token and throw an InconsistentStateException when they differ. A successful write creates a new opaque ETag and writes the record file.

Clear state

Clearing the state involves deleting the file if it exists.

public async Task ClearStateAsync<T>(
    string stateName,
    GrainId grainId,
    IGrainState<T> grainState)
{
    var path = GetRecordPath(stateName, grainId);
    var record = await TryReadRecordAsync(path).ConfigureAwait(false);
    if (record is not null)
    {
        ValidateETag<T>("ClearState", grainId, grainState.ETag, record.Value.ETag);
        File.Delete(path);
    }

    ResetState(grainState);
}

Before deleting an existing record, verify that the caller’s ETag matches the persisted token. A successful clear resets the state instance, ETag, and RecordExists.

Put it all together

Next, create a factory that allows scoping the options to the provider name while creating an instance of FileGrainStorage to ease registration with the service collection.

using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Options;
using Orleans.Configuration.Overrides;

namespace GrainStorage;

internal static class FileGrainStorageFactory
{
    internal static FileGrainStorage Create(
        IServiceProvider services, string name)
    {
        var optionsMonitor =
            services.GetRequiredService<IOptionsMonitor<FileGrainStorageOptions>>();

        return ActivatorUtilities.CreateInstance<FileGrainStorage>(
            services,
            name,
            optionsMonitor.Get(name),
            services.GetProviderClusterOptions(name));
    }
}

Lastly, create extensions on ISiloBuilder and IServiceCollection. They configure named options, register configuration validation and serializer defaults, and add the provider using Orleans storage registration.

using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Options;
using Orleans.Hosting;
using Orleans.Providers;
using Orleans.Runtime;
using Orleans.Runtime.Hosting;
using Orleans.Storage;

namespace GrainStorage;

public static class FileSiloBuilderExtensions
{
    public static ISiloBuilder AddFileGrainStorage(
        this ISiloBuilder builder,
        Action<FileGrainStorageOptions> options) =>
        builder.AddFileGrainStorage(ProviderConstants.DEFAULT_STORAGE_PROVIDER_NAME, options);

    public static ISiloBuilder AddFileGrainStorage(
        this ISiloBuilder builder,
        string providerName,
        Action<FileGrainStorageOptions> options) =>
        builder.ConfigureServices(
            services => services.AddFileGrainStorage(providerName, options));

    public static IServiceCollection AddFileGrainStorage(
        this IServiceCollection services,
        Action<FileGrainStorageOptions> options) =>
        services.AddFileGrainStorage(ProviderConstants.DEFAULT_STORAGE_PROVIDER_NAME, options);

    public static IServiceCollection AddFileGrainStorage(
        this IServiceCollection services,
        string providerName,
        Action<FileGrainStorageOptions> options)
    {
        services.AddOptions<FileGrainStorageOptions>(providerName)
            .Configure(options);

        services.AddTransient<IConfigurationValidator>(
            serviceProvider => new FileGrainStorageOptionsValidator(
                serviceProvider.GetRequiredService<IOptionsMonitor<FileGrainStorageOptions>>().Get(providerName),
                providerName));
        services.AddTransient<
            IPostConfigureOptions<FileGrainStorageOptions>,
            DefaultStorageProviderSerializerOptionsConfigurator<FileGrainStorageOptions>>();

        return services.AddGrainStorage(providerName, FileGrainStorageFactory.Create);
    }
}

The Orleans storage registration detects that FileGrainStorage implements ILifecycleParticipant<T> for ISiloLifecycle and registers its lifecycle participation.

services.AddTransient<IConfigurationValidator>(
    serviceProvider => new FileGrainStorageOptionsValidator(
        serviceProvider.GetRequiredService<IOptionsMonitor<FileGrainStorageOptions>>().Get(providerName),
        providerName));
services.AddTransient<
    IPostConfigureOptions<FileGrainStorageOptions>,
    DefaultStorageProviderSerializerOptionsConfigurator<FileGrainStorageOptions>>();

return services.AddGrainStorage(providerName, FileGrainStorageFactory.Create);

This enables adding the file storage using the extension on ISiloBuilder:

using GrainStorage;
using Microsoft.Extensions.Hosting;

var builder = Host.CreateApplicationBuilder(args);
builder.UseOrleans(siloBuilder =>
{
    siloBuilder.UseLocalhostClustering()
        .AddFileGrainStorage("File", options =>
        {
            string path = Environment.GetFolderPath(
                Environment.SpecialFolder.ApplicationData);

            options.RootDirectory = Path.Combine(path, "Orleans/GrainState/v1");
        });
});

using var host = builder.Build();
await host.RunAsync();

Now you can select the provider using StorageProviderAttribute, for example, [StorageProvider(ProviderName = "File")], and it stores the grain state in the root directory set in the options. Consider the full implementation of FileGrainStorage:

using System.Security.Cryptography;
using System.Text;
using Microsoft.Extensions.Options;
using Orleans.Configuration;
using Orleans.Runtime;
using Orleans.Serialization.Serializers;
using Orleans.Storage;

namespace GrainStorage;

public sealed class FileGrainStorage(
    string storageName,
    FileGrainStorageOptions options,
    IOptions<ClusterOptions> clusterOptions,
    IActivatorProvider activatorProvider) : IGrainStorage, ILifecycleParticipant<ISiloLifecycle>
{
    private const int RecordHeaderLength = 24;
    private static readonly byte[] RecordMagic = "ORLFS001"u8.ToArray();
    private readonly string _serviceId = clusterOptions.Value.ServiceId;
    private readonly string _rootDirectory = Path.GetFullPath(options.RootDirectory);

    public async Task ClearStateAsync<T>(
        string stateName,
        GrainId grainId,
        IGrainState<T> grainState)
    {
        var path = GetRecordPath(stateName, grainId);
        var record = await TryReadRecordAsync(path).ConfigureAwait(false);
        if (record is not null)
        {
            ValidateETag<T>("ClearState", grainId, grainState.ETag, record.Value.ETag);
            File.Delete(path);
        }

        ResetState(grainState);
    }

    public async Task ReadStateAsync<T>(
        string stateName,
        GrainId grainId,
        IGrainState<T> grainState)
    {
        var record = await TryReadRecordAsync(GetRecordPath(stateName, grainId)).ConfigureAwait(false);
        if (record is null)
        {
            ResetState(grainState);
            return;
        }

        grainState.State = options.GrainStorageSerializer.Deserialize<T>(new BinaryData(record.Value.Payload));
        grainState.ETag = record.Value.ETag;
        grainState.RecordExists = true;
    }

    public async Task WriteStateAsync<T>(
        string stateName,
        GrainId grainId,
        IGrainState<T> grainState)
    {
        var path = GetRecordPath(stateName, grainId);
        var existingRecord = await TryReadRecordAsync(path).ConfigureAwait(false);
        if (existingRecord is not null)
        {
            ValidateETag<T>("WriteState", grainId, grainState.ETag, existingRecord.Value.ETag);
        }
        else if (grainState.ETag is not null)
        {
            throw CreateInconsistentStateException<T>("WriteState", grainId);
        }

        var etag = Guid.NewGuid().ToString("N");
        var payload = options.GrainStorageSerializer.Serialize(grainState.State).ToArray();
        await File.WriteAllBytesAsync(path, CreateRecord(etag, payload)).ConfigureAwait(false);

        grainState.ETag = etag;
        grainState.RecordExists = true;
    }

    public void Participate(ISiloLifecycle lifecycle) =>
        lifecycle.Subscribe(
            observerName: OptionFormattingUtilities.Name<FileGrainStorage>(storageName),
            stage: ServiceLifecycleStage.ApplicationServices,
            onStart: _ =>
            {
                Directory.CreateDirectory(_rootDirectory);
                return Task.CompletedTask;
            });

    private static byte[] CreateRecord(string etag, byte[] payload)
    {
        var result = new byte[RecordHeaderLength + payload.Length];
        RecordMagic.CopyTo(result, 0);
        Guid.ParseExact(etag, "N").TryWriteBytes(result.AsSpan(RecordMagic.Length, 16));
        payload.CopyTo(result, RecordHeaderLength);
        return result;
    }

    private static async Task<StoredRecord?> TryReadRecordAsync(string path)
    {
        try
        {
            var bytes = await File.ReadAllBytesAsync(path).ConfigureAwait(false);
            if (bytes.Length < RecordHeaderLength ||
                !bytes.AsSpan(0, RecordMagic.Length).SequenceEqual(RecordMagic))
            {
                throw new InvalidDataException($"The file storage record '{path}' has an invalid format.");
            }

            var etag = new Guid(bytes.AsSpan(RecordMagic.Length, 16)).ToString("N");
            return new StoredRecord(etag, bytes.AsMemory(RecordHeaderLength));
        }
        catch (FileNotFoundException)
        {
            return null;
        }
        catch (DirectoryNotFoundException)
        {
            return null;
        }
    }

    private void ResetState<T>(IGrainState<T> grainState)
    {
        grainState.State = activatorProvider.GetActivator<T>().Create();
        grainState.ETag = null;
        grainState.RecordExists = false;
    }

    private void ValidateETag<T>(
        string operation,
        GrainId grainId,
        string? currentETag,
        string storedETag)
    {
        if (!string.Equals(currentETag, storedETag, StringComparison.Ordinal))
        {
            throw CreateInconsistentStateException<T>(operation, grainId);
        }
    }

    private InconsistentStateException CreateInconsistentStateException<T>(
        string operation,
        GrainId grainId) =>
        new($"Version conflict ({operation}): ServiceId={_serviceId} ProviderName={storageName} GrainType={typeof(T)} GrainReference={grainId}.");

    private string GetRecordPath(string stateName, GrainId grainId)
    {
        using var identity = new MemoryStream();
        WriteIdentityComponent(identity, Encoding.UTF8.GetBytes(_serviceId));
        WriteIdentityComponent(identity, grainId.Type.AsSpan());
        WriteIdentityComponent(identity, grainId.Key.AsSpan());
        WriteIdentityComponent(identity, Encoding.UTF8.GetBytes(stateName));

        var hash = SHA256.HashData(identity.GetBuffer().AsSpan(0, checked((int)identity.Length)));
        return Path.Combine(_rootDirectory, $"{Convert.ToHexString(hash)}.grain");
    }

    private static void WriteIdentityComponent(Stream destination, ReadOnlySpan<byte> value)
    {
        Span<byte> length = stackalloc byte[sizeof(int)];
        System.Buffers.Binary.BinaryPrimitives.WriteInt32BigEndian(length, value.Length);
        destination.Write(length);
        destination.Write(value);
    }

    private readonly record struct StoredRecord(string ETag, ReadOnlyMemory<byte> Payload);
}