Custom grain storage

In the tutorial on declarative actor storage, you learned how to allow grains to store their state in an Azure table using one of the built-in storage providers. While Azure is a great place to store your data, many alternatives exist. There are so many that supporting them all isn’t feasible. Instead, Orleans is designed to let you easily add support for your preferred storage by writing a custom grain storage provider.

In this tutorial, you’ll walk through how to write a simple file-based grain storage provider. A file system isn’t the best place to store grain states because it’s local, can have issues with file locks, and the last update date isn’t sufficient to prevent inconsistency. However, it’s an easy example to illustrate the implementation of a grain storage provider.

Get started

An Orleans grain storage provider is a class that implements IGrainStorage, included in the Microsoft.Orleans.Core NuGet package. This sample also implements ILifecycleParticipant<T>, where the lifecycle type is ISiloLifecycle, so that it can initialize during the silo lifecycle. Start by creating a class named FileGrainStorage; the complete, compiling implementation appears at the end of the tutorial.

Each method implements the corresponding method in the IGrainStorage interface, accepting a generic type parameter for the underlying state type. The methods are:

• IGrainStorage.ReadStateAsync: Reads the state of a grain.
• IGrainStorage.WriteStateAsync: Writes the state of a grain.
• IGrainStorage.ClearStateAsync: Clears the state of a grain.

All three methods receive the same arguments:

Argument	Meaning
stateName	The logical name of this state record. For IPersistentState<T>, this is the state name configured by PersistentStateAttribute; legacy Grain<T> state uses state. A grain can have multiple named state records, so include this value in the storage key. Older versions of the interface called this argument grainType, but it doesn’t describe the state object’s .NET type.
grainId	The complete Orleans grain identity, including its grain type and primary key. Combine it with stateName to identify a record. Don’t key records using only the primary key, because different grain types can use the same key.
grainState	The state container Orleans passes to the provider. Its State property contains the application state, ETag carries the provider’s optimistic-concurrency token, and RecordExists indicates whether the record exists. A read populates these properties; a write or clear updates them to reflect the completed operation.
T	The declared .NET type of the state payload. Use T (or typeof(T)) for serialization and type metadata. It isn’t a record identifier and can be shared by many grains and named state records.

Therefore, stateName and grainState.State?.GetType().Name aren’t interchangeable. The first identifies one of a grain’s logical state records and is stable independently of the payload implementation. The second is only the runtime type name of the current payload; it can be null, can differ from typeof(T) when polymorphism is involved, and can change during a refactoring. The sample uses stateName and grainId for identity and delegates payload type handling to the configured serializer.

The ILifecycleParticipant<T>.Participate method subscribes to the silo’s lifecycle.

Before starting the implementation, create an options class containing the root directory where grain state files are persisted. Create an options file named FileGrainStorageOptions containing the following:

using Orleans.Storage;

namespace GrainStorage;

public sealed class FileGrainStorageOptions : IStorageProviderSerializerOptions
{
    public required string RootDirectory { get; set; }

    public required IGrainStorageSerializer GrainStorageSerializer { get; set; }
}

With the options class created, explore the constructor parameters of the FileGrainStorage class:

• storageName: Specifies which grains should use this storage provider through StorageProviderAttribute, for example, [StorageProvider(ProviderName = "File")].
• options: The options class just created.
• clusterOptions: The cluster options used for retrieving the ServiceId.

Initialize the storage

To initialize the storage, subscribe to the ServiceLifecycleStage.ApplicationServices stage with an onStart function. Consider the following ILifecycleParticipant<T>.Participate implementation:

public void Participate(ISiloLifecycle lifecycle) =>
    lifecycle.Subscribe(
        observerName: OptionFormattingUtilities.Name<FileGrainStorage>(_storageName),
        stage: ServiceLifecycleStage.ApplicationServices,
        onStart: (ct) =>
        {
            Directory.CreateDirectory(_options.RootDirectory);
            return Task.CompletedTask;
        });

The onStart function conditionally creates the root directory to store grain states if it doesn’t already exist.

Also, provide a common function to construct the filename, ensuring uniqueness per service, grain ID, and state name:

private string GetKeyString(string stateName, GrainId grainId) =>
    $"{_clusterOptions.ServiceId}.{grainId.Type}.{grainId.Key}.{stateName}";

Read state

To read a grain state, get the filename using the GetKeyString function and combine it with the root directory from the _options instance.

public async Task ReadStateAsync<T>(
    string stateName,
    GrainId grainId,
    IGrainState<T> grainState)
{
    var fName = GetKeyString(stateName, grainId);
    var path = Path.Combine(_options.RootDirectory, fName!);
    var fileInfo = new FileInfo(path);
    if (fileInfo is { Exists: false })
    {
        grainState.State = (T)Activator.CreateInstance(typeof(T))!;
        grainState.ETag = null;
        grainState.RecordExists = false;
        return;
    }

    using var stream = fileInfo.OpenText();
    var storedData = await stream.ReadToEndAsync();

    grainState.State = _options.GrainStorageSerializer.Deserialize<T>(new BinaryData(storedData));
    grainState.ETag = fileInfo.LastWriteTimeUtc.ToString();
    grainState.RecordExists = true;
}

Use fileInfo.LastWriteTimeUtc as an ETag, which other functions use for inconsistency checks to prevent data loss. Set RecordExists to indicate whether the read found a record.

For deserialization, use the IStorageProviderSerializerOptions.GrainStorageSerializer. This is important for correctly serializing and deserializing the state.

Write state

Writing the state is similar to reading the state.

public async Task WriteStateAsync<T>(
    string stateName,
    GrainId grainId,
    IGrainState<T> grainState)
{
    var storedData = _options.GrainStorageSerializer.Serialize(grainState.State);
    var fName = GetKeyString(stateName, grainId);
    var path = Path.Combine(_options.RootDirectory, fName!);
    var fileInfo = new FileInfo(path);
    if (fileInfo.Exists && fileInfo.LastWriteTimeUtc.ToString() != grainState.ETag)
    {
        throw new InconsistentStateException($"""
            Version conflict (WriteState): ServiceId={_clusterOptions.ServiceId}
            ProviderName={_storageName} GrainType={typeof(T)}
            GrainReference={grainId}.
            """);
    }

    await File.WriteAllBytesAsync(path, storedData.ToArray());

    fileInfo.Refresh();
    grainState.ETag = fileInfo.LastWriteTimeUtc.ToString();
    grainState.RecordExists = true;
}

Similar to reading state, use the IStorageProviderSerializerOptions.GrainStorageSerializer to write the state. The current ETag checks against the file’s last updated UTC time. If the date differs, it means another activation of the same grain changed the state concurrently. In this situation, throw an InconsistentStateException. This results in the current activation being killed to prevent overwriting the state previously saved by the other activated grain.

Clear state

Clearing the state involves deleting the file if it exists.

public Task ClearStateAsync<T>(
    string stateName,
    GrainId grainId,
    IGrainState<T> grainState)
{
    var fName = GetKeyString(stateName, grainId);
    var path = Path.Combine(_options.RootDirectory, fName!);
    var fileInfo = new FileInfo(path);
    if (fileInfo.Exists)
    {
        if (fileInfo.LastWriteTimeUtc.ToString() != grainState.ETag)
        {
            throw new InconsistentStateException($"""
                Version conflict (ClearState): ServiceId={_clusterOptions.ServiceId}
                ProviderName={_storageName} GrainType={typeof(T)}
                GrainReference={grainId}.
                """);
        }

        fileInfo.Delete();
    }

    grainState.ETag = null;
    grainState.RecordExists = false;
    grainState.State = (T)Activator.CreateInstance(typeof(T))!;

    return Task.CompletedTask;
}

For the same reason as WriteStateAsync, check for inconsistency. Before deleting the file and resetting the ETag, check if the current ETag matches the last write time UTC.

Put it all together

Next, create a factory that allows scoping the options to the provider name while creating an instance of FileGrainStorage to ease registration with the service collection.

using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Options;
using Orleans.Configuration.Overrides;
using Orleans.Storage;

namespace GrainStorage;

internal static class FileGrainStorageFactory
{
    internal static IGrainStorage Create(
        IServiceProvider services, string name)
    {
        var optionsMonitor =
            services.GetRequiredService<IOptionsMonitor<FileGrainStorageOptions>>();

        return ActivatorUtilities.CreateInstance<FileGrainStorage>(
            services,
            name,
            optionsMonitor.Get(name),
            services.GetProviderClusterOptions(name));
    }
}

Lastly, to register the grain storage, create an extension on ISiloBuilder. This extension registers the grain storage as a keyed singleton using ServiceCollectionServiceExtensions.AddKeyedSingleton.

using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Options;
using Orleans.Runtime;
using Orleans.Storage;

namespace GrainStorage;

public static class FileSiloBuilderExtensions
{
    public static ISiloBuilder AddFileGrainStorage(
        this ISiloBuilder builder,
        string providerName,
        Action<FileGrainStorageOptions> options)
    {
        builder.Services.AddFileGrainStorage(providerName, options);
        return builder;
    }

    public static IServiceCollection AddFileGrainStorage(
        this IServiceCollection services,
        string providerName,
        Action<FileGrainStorageOptions> options)
    {
        services.AddOptions<FileGrainStorageOptions>(providerName)
            .Configure(options);

        services.AddTransient<
            IPostConfigureOptions<FileGrainStorageOptions>,
            DefaultStorageProviderSerializerOptionsConfigurator<FileGrainStorageOptions>>();

        services.AddKeyedSingleton<IGrainStorage>(
            providerName,
            (sp, key) => FileGrainStorageFactory.Create(sp, key?.ToString() ?? providerName));

        services.AddKeyedSingleton<ILifecycleParticipant<ISiloLifecycle>>(
            providerName,
            (sp, key) => (ILifecycleParticipant<ISiloLifecycle>)sp.GetRequiredKeyedService<IGrainStorage>(key));

        return services;
    }
}

The FileGrainStorage implements IGrainStorage and ILifecycleParticipant<T> for ISiloLifecycle. Therefore, register two keyed singleton services, one for each interface.

services.AddKeyedSingleton<IGrainStorage>(
    providerName,
    (sp, key) => FileGrainStorageFactory.Create(sp, key?.ToString() ?? providerName));

services.AddKeyedSingleton<ILifecycleParticipant<ISiloLifecycle>>(
    providerName,
    (sp, key) => (ILifecycleParticipant<ISiloLifecycle>)sp.GetRequiredKeyedService<IGrainStorage>(key));

This enables adding the file storage using the extension on ISiloBuilder:

using GrainStorage;
using Microsoft.Extensions.Hosting;

var builder = Host.CreateApplicationBuilder(args);
builder.UseOrleans(siloBuilder =>
{
    siloBuilder.UseLocalhostClustering()
        .AddFileGrainStorage("File", options =>
        {
            string path = Environment.GetFolderPath(
                Environment.SpecialFolder.ApplicationData);

            options.RootDirectory = Path.Combine(path, "Orleans/GrainState/v1");
        });
});

using var host = builder.Build();
await host.RunAsync();

Now you can select the provider using StorageProviderAttribute, for example, [StorageProvider(ProviderName = "File")], and it stores the grain state in the root directory set in the options. Consider the full implementation of FileGrainStorage:

using Microsoft.Extensions.Options;
using Orleans.Configuration;
using Orleans.Runtime;
using Orleans.Storage;

namespace GrainStorage;

public sealed class FileGrainStorage : IGrainStorage, ILifecycleParticipant<ISiloLifecycle>
{
    private readonly string _storageName;
    private readonly FileGrainStorageOptions _options;
    private readonly ClusterOptions _clusterOptions;

    public FileGrainStorage(
        string storageName,
        FileGrainStorageOptions options,
        IOptions<ClusterOptions> clusterOptions)
    {
        _storageName = storageName;
        _options = options;
        _clusterOptions = clusterOptions.Value;
    }

    public Task ClearStateAsync<T>(
        string stateName,
        GrainId grainId,
        IGrainState<T> grainState)
    {
        var fName = GetKeyString(stateName, grainId);
        var path = Path.Combine(_options.RootDirectory, fName!);
        var fileInfo = new FileInfo(path);
        if (fileInfo.Exists)
        {
            if (fileInfo.LastWriteTimeUtc.ToString() != grainState.ETag)
            {
                throw new InconsistentStateException($"""
                    Version conflict (ClearState): ServiceId={_clusterOptions.ServiceId}
                    ProviderName={_storageName} GrainType={typeof(T)}
                    GrainReference={grainId}.
                    """);
            }

            fileInfo.Delete();
        }

        grainState.ETag = null;
        grainState.RecordExists = false;
        grainState.State = (T)Activator.CreateInstance(typeof(T))!;

        return Task.CompletedTask;
    }
    public async Task ReadStateAsync<T>(
        string stateName,
        GrainId grainId,
        IGrainState<T> grainState)
    {
        var fName = GetKeyString(stateName, grainId);
        var path = Path.Combine(_options.RootDirectory, fName!);
        var fileInfo = new FileInfo(path);
        if (fileInfo is { Exists: false })
        {
            grainState.State = (T)Activator.CreateInstance(typeof(T))!;
            grainState.ETag = null;
            grainState.RecordExists = false;
            return;
        }

        using var stream = fileInfo.OpenText();
        var storedData = await stream.ReadToEndAsync();

        grainState.State = _options.GrainStorageSerializer.Deserialize<T>(new BinaryData(storedData));
        grainState.ETag = fileInfo.LastWriteTimeUtc.ToString();
        grainState.RecordExists = true;
    }
    public async Task WriteStateAsync<T>(
        string stateName,
        GrainId grainId,
        IGrainState<T> grainState)
    {
        var storedData = _options.GrainStorageSerializer.Serialize(grainState.State);
        var fName = GetKeyString(stateName, grainId);
        var path = Path.Combine(_options.RootDirectory, fName!);
        var fileInfo = new FileInfo(path);
        if (fileInfo.Exists && fileInfo.LastWriteTimeUtc.ToString() != grainState.ETag)
        {
            throw new InconsistentStateException($"""
                Version conflict (WriteState): ServiceId={_clusterOptions.ServiceId}
                ProviderName={_storageName} GrainType={typeof(T)}
                GrainReference={grainId}.
                """);
        }

        await File.WriteAllBytesAsync(path, storedData.ToArray());

        fileInfo.Refresh();
        grainState.ETag = fileInfo.LastWriteTimeUtc.ToString();
        grainState.RecordExists = true;
    }
    public void Participate(ISiloLifecycle lifecycle) =>
        lifecycle.Subscribe(
            observerName: OptionFormattingUtilities.Name<FileGrainStorage>(_storageName),
            stage: ServiceLifecycleStage.ApplicationServices,
            onStart: (ct) =>
            {
                Directory.CreateDirectory(_options.RootDirectory);
                return Task.CompletedTask;
            });
    private string GetKeyString(string stateName, GrainId grainId) =>
        $"{_clusterOptions.ServiceId}.{grainId.Type}.{grainId.Key}.{stateName}";
}