[Directory] Strengthen single activation guarantees even further.

[ledjon-behluli]

This PR adds safety lease holds to the new strongly-consistent grain directory to prevent split-brain scenarios after silo crashes, utilizing both specific grain leases (tombstones) and broader range leases when snapshot transfers fail. Under normal graceful shutdowns the new directory performs handovers, so no range lease holds are applied.

Users can specify the lease duration, otherwise if duration <= zero than it is computed as: 2 × ProbeTimeout × NumMissedProbesLimit which should provide a good buffer so that a network-partitioned silo has enough time to realize its isolated and shut itself down before the rest of the cluster allows those grains to reactivate elsewhere.

The impact on normal registrations in a healthy cluster should be practically unobservable.

Note, this wont work with the eventually-consistent directory!

Microsoft Reviewers: Open in CodeFlow

[ledjon-behluli]

@ReubenBond addressed the feedback

[Copilot]

Pull request overview

This PR strengthens the strongly-consistent distributed grain directory’s single-activation guarantees by introducing “safety lease holds” after ungraceful silo failures and during recovery scenarios, delaying re-registrations to reduce split-brain risk.

Changes:

• Add silo-level and range-level lease hold tracking to directory partitions, including periodic cleanup of expired leases.
• Enforce lease holds during RegisterAsync by rejecting registrations with a new DirectoryLeaseHoldException.
• Add configuration (SafetyLeaseHoldDuration) and a new test validating reactivation blocking after an ungraceful shutdown.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
test/Orleans.GrainDirectory.Tests/Orleans.GrainDirectory.Tests.csproj	Adds Microsoft.Extensions.TimeProvider.Testing package for FakeTimeProvider in tests.
test/Orleans.GrainDirectory.Tests/GrainDirectory/GrainDirectoryLeaseTests.cs	New test exercising lease-hold behavior after KillSiloAsync.
src/Orleans.Runtime/GrainDirectory/GrainDirectoryPartition.cs	Tracks/creates lease holds, applies holds on silo death, and adds cleanup logic + logging.
src/Orleans.Runtime/GrainDirectory/GrainDirectoryPartition.Interface.cs	Enforces lease holds on registration by throwing DirectoryLeaseHoldException.
src/Orleans.Runtime/GrainDirectory/DistributedGrainDirectory.cs	Computes effective lease duration, wires TimeProvider, and runs periodic lease cleanup requests.
src/Orleans.Runtime/Configuration/Options/GrainDirectoryOptions.cs	Adds SafetyLeaseHoldDuration option and documentation; switches to file-scoped namespace.
src/Orleans.Core.Abstractions/Exceptions/DirectoryLeaseHoldException.cs	Introduces a new exception type for lease-hold rejections.

Comments suppressed due to low confidence (2)

src/Orleans.Runtime/Configuration/Options/GrainDirectoryOptions.cs:88

• Typo in XML doc comment: "clcks skues" should be "clock skews".

    /// Conditional deregistration is used for lazy clean-up of activations whose prompt deregistration failed for some reason (e.g., message failure).
    /// This should always be at least one minute, since we compare the times on the directory partition, so message delays and clcks skues have
    /// to be allowed.

test/Orleans.GrainDirectory.Tests/GrainDirectory/GrainDirectoryLeaseTests.cs:65

• Grammar in comment: "its the only one alive" should be "it's the only one alive".

            // Time has expired, we can place it now, and it will be the primary as its the only one alive.
            Assert.Equal(await leaseGrain.GetAddress(), primary.SiloAddress);

[ReubenBond]

We should add some more tests. I'll ask copilot to add some more coverage.

[ReubenBond]

@ledjon-behluli I made a minor change to not throw when a lease is still held. Now, the directory returns a 'retry after delay' result instead.

[ledjon-behluli]

@ReubenBond is there something left on my side?