Placement and activation balancing

Placement chooses a silo when Orleans needs a new activation. Balancing can later move existing activations. Those are separate decisions with different information and costs.

The default placement strategy is ResourceOptimizedPlacement.

For the application and operational view of scale-out, scale-in, persistence, and configuration, see Grain placement and migration.

Resource-optimized placement

ResourceOptimizedPlacementDirector receives cluster-wide SiloRuntimeStatistics from DeploymentLoadPublisher. It excludes incompatible and overloaded silos, applies a power-of-multiple-choices load-balancing strategy by sampling approximately the square root of the available candidates, normalizes their signals, adds jitter to avoid deterministic herding, and selects the lowest utilization score.

Rendering diagram.

The default relative weights are:

Signal	Weight
ResourceOptimizedPlacementOptions.CpuUsageWeight	40
ResourceOptimizedPlacementOptions.MemoryUsageWeight	20
ResourceOptimizedPlacementOptions.AvailableMemoryWeight	20
ResourceOptimizedPlacementOptions.MaxAvailableMemoryWeight	5
ResourceOptimizedPlacementOptions.ActivationCountWeight	15

ResourceOptimizedPlacementOptions.LocalSiloPreferenceMargin defaults to 5. If the local silo’s score is within that margin of the best candidate, placement can preserve locality. Before usable statistics are available, the director selects a compatible silo.

Public options: ResourceOptimizedPlacementOptions. Implementation: ResourceOptimizedPlacementDirector and the default registration in DefaultSiloServices.

Enabling LoadSheddingEnabled sets the IsOverloaded statistic when either its CPU or memory threshold is exceeded. When non-overloaded compatible silos have published statistics, resource-optimized placement scores that set using CPU, memory, capacity, and activation-count measurements. When every compatible silo is overloaded or lacks current statistics, placement selects a compatible silo so activation can proceed.

Load shedding

Set LoadSheddingEnabled to true to activate load shedding. CpuThreshold defaults to 95 percent and MemoryThreshold defaults to 90 percent. Client-gateway request rejection evaluates both thresholds. Stream providers which use LoadShedQueueFlowController evaluate CPU usage against a limit derived from CpuThreshold and pause queue reads at that limit. Crossing either configured threshold marks the silo’s published runtime statistics as overloaded, and resource-optimized placement favors compatible silos with non-overloaded statistics.

Use load shedding as one layer of admission control alongside bounded work and deadlines. Use a hosting-platform autoscaler to create capacity, activation rebalancing or repartitioning to move eligible activations, and memory-based activation shedding to deactivate activations under memory pressure.

Placement resolution and extension

PlacementStrategyResolver selects a grain-specific strategy when one is declared; otherwise it uses the default. PlacementService applies placement filters before calling the strategy’s keyed IPlacementDirector.

A custom strategy consists of:

1. an PlacementStrategy value associated with the grain type;
2. an IPlacementDirector which chooses from compatible silos; and
3. registration through PlacementStrategyExtensions.AddPlacementDirector.

Placement filters are orthogonal constraints. They can remove candidates based on metadata or another policy before the director scores them. A director should use the candidates supplied by the placement context instead of reconstructing cluster membership.

API: PlacementStrategyResolver and PlacementStrategyExtensions.AddPlacementDirector. Implementation: PlacementService, strategy resolution, and registration extensions.

Balancing long-lived activations

Resource-optimized placement only affects new activations. Long-lived activations can become unbalanced after:

• a silo joins or leaves;
• traffic changes while the activation remains alive;
• activation memory use diverges;
• a placement constraint changes the candidate set; or
• communicating grains are spread across silos.

Moving an activation has a cost: dehydrate and rehydrate work, directory updates, cold caches, and a temporary interruption. Orleans therefore exposes opt-in protocols rather than continuously moving every activation.

Valid activations continue running on their current silos across membership changes. A joining silo receives activations through later creation or opt-in migration. During graceful shutdown, ordinary activations on the departing silo deactivate; later calls create replacements on remaining compatible silos.

Experimental activation rebalancer

ActivationRebalancerExtensions.AddActivationRebalancer enables the resource rebalancer and produces compiler warning ORLEANSEXP002. A worker observes cluster statistics in sessions, estimates imbalance using entropy, and asks source silos to migrate random activations toward underloaded silos. A monitor can wake or relocate the worker after failure.

The protocol optimizes distribution of activation count and memory use. The activation repartitioner complements it by optimizing the communication graph and cross-silo hot paths.

The most operationally significant ActivationRebalancerOptions are:

Option	Default	Effect
RebalancerDueTime	60 seconds	Delay before the first balancing session.
SessionCyclePeriod	15 seconds	Time between cycles in a session. It must be at least twice the deployment-statistics refresh period.
MaxStagnantCycles	3	Stop a session after consecutive cycles whose improvement remains below the entropy quantum.
ActivationMigrationCountLimit	int.MaxValue	Maximum requested migrations per cycle. Set a finite initial limit to bound churn while evaluating the feature.

The entropy quantum, allowed deviation, and cycle and silo weights control convergence and migration rate. Keep their defaults until representative measurements justify a change. Resolve IActivationRebalancer from silo services to suspend or resume sessions, request a RebalancingReport, or subscribe to reports. Reports contain an approximate cluster imbalance and per-silo acquired and dispersed activation counts. Also observe migration rate, activation latency, state-transfer failures, memory, and cross-silo calls.

Implementation: ActivationRebalancerWorker.

Experimental activation repartitioner

ActivationRepartitioningExtensions.AddActivationRepartitioner enables communication-aware repartitioning and produces compiler warning ORLEANSEXP001. The repartitioner samples fully addressed request messages and constructs a weighted graph:

• vertices represent migratable activations;
• edge weights represent observed calls;
• anchored or non-migratable grains constrain possible moves.

Periodically, peer repartitioners exchange graph information and negotiate activation moves which improve locality while respecting an imbalance-tolerance rule. The default RebalancerCompatibleRule can incorporate the resource rebalancer’s cluster-imbalance report.

The most operationally significant ActivationRepartitionerOptions are:

Option	Default	Effect
MaxEdgeCount	10,000	Bounds the probabilistic top communication edges retained for a round.
MaxUnprocessedEdges	100,000	Bounds the pending edge buffer; the oldest entries are discarded when full.
MinRoundPeriod / MaxRoundPeriod	1 / 2 minutes	Defines the randomized interval between rounds.
RecoveryPeriod	1 minute	Prevents a silo from immediately entering another round.
AnchoringFilterEnabled	true	Reduces graph size by probabilistically collapsing well-partitioned local vertices.

Larger edge and buffer limits improve the chance of retaining useful communication data but consume more memory. Shorter round and recovery periods react faster but increase coordination and migration churn. For large clusters, allow enough time for round exchanges; the options guidance recommends adding approximately 10 seconds per anticipated silo to the maximum round period. Evaluate effectiveness using cross-silo call volume and latency together with migration rate and repartitioner logs.

Implementation: ActivationRepartitioner and RepartitionerMessageFilter.

Choosing the mechanism

Mechanism	When it acts	Primary signal	Effect
Resource-optimized placement	Activation creation	CPU, memory, capacity, activation count	Places the new activation
Activation rebalancer	Opt-in balancing sessions	Cluster resource imbalance	Random eligible activations
Activation repartitioner	Opt-in exchange rounds	Grain call graph and tolerance rule	Communication-aware activations
Load shedding	CPU or memory threshold exceeded	Local CPU and memory use	Rejects gateway requests, controls stream queue reads using CPU, and marks the silo overloaded

Resource-optimized placement is the default and is usually the first mechanism to tune. Add the activation rebalancer for persistent count or memory skew and the repartitioner for call-locality problems after measuring the workload. Enable load shedding for overload protection. The experimental movement protocols use activation migration. Apply capacity planning, admission control, and deployment health monitoring alongside these runtime mechanisms. Operational guidance belongs in the deployment section.