Coordination & role leases

Elarion has four deliberately different coordination shapes. Choose the one that matches the unit being coordinated:

A role lease answers a coarse question such as “which instance is the maintenance coordinator?” It is a heartbeat-renewed row in the application's PostgreSQL database, exposed as an IRoleLease keyed by role name. The current holder renews; other instances keep trying; graceful shutdown releases immediately; a crashed holder fails over when its lease expires.

This default targets Elarion's normal scale: roughly 1–10 application nodes on the PostgreSQL the app already runs. Beyond that tier, replace the coordination seam with dedicated infrastructure rather than growing a membership system inside the application.

Map the lease tableCopy Anchor Link

Add Elarion.Coordination.PostgreSql and opt the application's concrete DbContext into the model:

using Elarion.Coordination.PostgreSql;
using Elarion.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore;

[GenerateDbSets]
[GenerateElarionRoleLeases]
public sealed partial class AppDbContext(DbContextOptions<AppDbContext> options)
    : DbContext(options) {
    protected override void OnModelCreating(ModelBuilder modelBuilder) {
        base.OnModelCreating(modelBuilder);
        ConfigureEntities(modelBuilder); // generated, including elarion_role_leases
    }
}

The hand-written model equivalent is modelBuilder.UseElarionRoleLeases(). Apply the resulting EF Core migration before any instance starts competing for a role.

Register one lease per roleCopy Anchor Link

Each registration adds a keyed IRoleLease and its heartbeat worker:

builder.Services.AddElarionPostgreSqlRoleLease<AppDbContext>(options => {
    options.RoleName = "maintenance";
});

Duplicate registration of the same role in one process fails at startup. The defaults are sized so the old holder stops acting before a successor may legitimately take over:

LeaseDuration must be greater than RenewInterval + HeldSafetyMargin.

Role names are validated against the lease table's 128-character limit during registration. Fixed partitions validate their generated {name}:partition-N roles before adding any services.

Register a fixed role partitionCopy Anchor Link

When a stable key should select one of several coarse roles without configuring process count, register a fixed partition. One process may hold several partitions; this does not add automatic balancing:

builder.Services.AddElarionPostgreSqlRolePartition<AppDbContext>(partition => {
    partition.Name = "devices";
    partition.PartitionCount = 16;
});

Resolve the keyed IRolePartition (keyed by the partition name) and call Resolve(key) (or Resolve(scope, key)) for the locally cached role/holder/address view — no I/O on the hot path:

public sealed class DeviceRouter([FromKeyedServices("devices")] IRolePartition partition) {
    public bool OwnsDevice(string deviceId) {
        RolePartitionTarget target = partition.Resolve(deviceId);   // stable hash → one of 16 roles
        return target.IsHeld;   // this process holds the partition's role lease
        // target.Role is e.g. "devices:partition-7"; target.CurrentHolderAddress routes elsewhere.
    }
}

Roles are named devices:partition-0 through devices:partition-15. IRoleLeaseRegistry exposes all configured leases to infrastructure such as role-affine outbox claiming.

Gate every cycleCopy Anchor Link

Resolve the lease by its role name and check IsHeld at the start of every unit of work. IsHeld is local, lock-free state; it does not query PostgreSQL on the hot path.

using Elarion.Abstractions.Coordination;
using Microsoft.Extensions.DependencyInjection;

public sealed class MaintenanceWorker(
    [FromKeyedServices("maintenance")] IRoleLease lease,
    TimeProvider timeProvider)
    : BackgroundService {

    protected override async Task ExecuteAsync(CancellationToken stoppingToken) {
        while (!stoppingToken.IsCancellationRequested) {
            if (lease.IsHeld) {
                await RunOneCycleAsync(stoppingToken);
            }

            await Task.Delay(TimeSpan.FromSeconds(10), timeProvider, stoppingToken);
        }
    }
}

Do not start a permanent leader-only loop in response to a lease-change event. There is deliberately no such event: database outages fail closed, and a holder can lose the role between any two cycles. Gate each call or polling cycle instead.

CurrentHolder and CurrentHolderAddress are diagnostic/routing snapshots refreshed at heartbeat cadence. They may lag a failover by one renewal interval; correctness comes from IsHeld, not from comparing holder strings.

Route HTTP to the holderCopy Anchor Link

Some roles own live state or an ordered stream, so every public instance must accept the route while only the holder may execute it. Elarion.AspNetCore ships a small role-holder proxy for that shape:

builder.Services.AddElarionInstanceAddress();
builder.Services.AddElarionPostgreSqlRoleLease<AppDbContext>(options => {
    options.RoleName = "maintenance";
});

var app = builder.Build();

// Install before routing/auth so a non-holder never starts local execution.
app.UseElarionRoleHolderProxy("maintenance", "/maintenance", "/maintenance-streams");

app.MapMaintenanceEndpoints();

AddElarionInstanceAddress() advertises the server's bound address, replacing wildcard hosts with a non-loopback IPv4 address. Pass an explicit address behind NAT, a reverse proxy, or when nodes use HTTPS between themselves:

builder.Services.AddElarionInstanceAddress("https://app-03.internal:8443");

The proxy forwards at most one hop and supports streaming responses such as SSE. It never retries or queues a request: an unknown/unreachable holder or a failover race returns 503 with Retry-After. The loop guard prevents a forwarded request from bouncing again. When the named lease is not registered (for example in single-instance development), the call installs no middleware.

The prefix list is intentionally the same rule a future ingress or service mesh would own. When external routing can target the holder directly, remove the in-app proxy rather than layering both.

For a key-selected role, use the partition overload before routing:

app.UseElarionPartitionHolderProxy(
    "devices",
    context => context.Request.Headers["X-Device-Id"].FirstOrDefault(),
    "/device-api");

The resolver reads raw path/query/header state. Missing affinity keys return 400; holder failures use the same bounded 503 behavior as the coarse-role proxy.

When the target subsystem uses scoped affinity, pass that same stable scope. Virtual-sharded actors hash their logical actor name (the [Actor(Name = ...)] value, or the class name without Actor) and canonical key together, so actor ingress must use the scoped overload:

app.UseElarionPartitionHolderProxy(
    "actors",
    "Order",
    context => context.Request.Path.Value?.Split('/').LastOrDefault(),
    "/orders");

Actor home is a specialized roleCopy Anchor Link

AddElarionPostgreSqlActorHome<TDbContext>() composes the same mechanism under the reserved "actors" role for [Actor(Placement = ActorPlacementMode.SingleHome)] actors. Calls on other instances fail rather than forwarding through the actor runtime; host-owned HTTP prefixes can use UseElarionRoleHolderProxy("actors", ...) to reach a single home, or UseElarionPartitionHolderProxy("actors", actorName, ...) for virtual-sharded ingress. See Actors for the placement and migration boundaries.

Failure behaviorCopy Anchor Link

• Graceful shutdown: the holder releases the row, allowing immediate takeover.
• Process or node loss: another instance acquires after LeaseDuration expires.
• Database outage: no instance considers itself holder after its safety window; coordination fails closed.
• Clock skew: leases use application clocks. Keep node clocks synchronized; this is not a consensus protocol.
• Two processes on one machine: the random InstanceId suffix keeps their identities distinct.

The full rationale and proxy design are recorded in ADR-0049 and ADR-0050.

Leased work rowsCopy Anchor Link

When an application keeps its own queue-shaped table — outbound e-mail or SMS deliveries, webhook dispatches, a durable retry table — and workers on any instance must process each row one at a time, use the leased-work-row primitive from Elarion.EntityFrameworkCore.LeasedWork. It is the same claim the outbox delivers integration events with, packaged so a second table does not re-derive its invariants from memory:

• Claim is select → conditional stamp → re-read. Candidates are read without locks; one ExecuteUpdate restating the whole eligibility predicate stamps a lease token and expiry onto the rows that are still eligible; the worker gets back only the rows carrying its token. Concurrent workers never block each other and never receive the same row for the same lease period.
• Every finalize is lease-guarded. Completing, failing, releasing, or renewing a row only matches while the worker's token is still on it. A worker that stalled past its lease learns it lost the row (false) instead of overwriting the new owner.
• Recovery is by expiry. A crashed worker releases nothing; its lease lapses and the row becomes claimable again. There is no reaper and no lock to leak.
• The claim rides a partial index over the live rows only, so it stays fast while completed history accumulates.

No statement holds a row lock or a transaction across the work itself. That matters most for outbound delivery: a slow SMTP or HTTP call never pins a database connection, and a producer can still update or withdraw a row while a worker is busy with it. SELECT … FOR UPDATE SKIP LOCKED held across the external call does the opposite and is deliberately not the pattern.

Map the tableCopy Anchor Link

The entity implements ILeasedWorkRow (Id, LockId, LockedUntilUtc) as ordinary properties. Everything else — the completion marker, the attempt counter, the error, a deadline — stays your own column:

public sealed class Delivery : ILeasedWorkRow {
    public required Guid Id { get; init; }
    public required DateTimeOffset CreatedAtUtc { get; init; }
    public required DateTimeOffset ExpiresAtUtc { get; init; }
    public byte[]? Payload { get; set; }
    public int Attempts { get; set; }
    public DateTimeOffset? CompletedAtUtc { get; set; }
    public string? Error { get; set; }
    public Guid? LockId { get; set; }
    public DateTimeOffset? LockedUntilUtc { get; set; }
}

[EntityConfiguration]
public sealed class DeliveryConfiguration : IEntityTypeConfiguration<Delivery> {
    public void Configure(EntityTypeBuilder<Delivery> entity) {
        entity.ToTable("deliveries");
        // Lease columns plus the partial claim index ix_deliveries_claim over the live queue.
        entity.HasElarionLeasedWork(d => new { d.CreatedAtUtc, d.Id }, "completed_at_utc IS NULL");
    }
}

The claim index lists the queue-order column and then Id, matching the claim's ordering. The filter is required. If you keep completed rows for a while, add the complementary purge index yourself.

Claim, work, finalizeCopy Anchor Link

private static readonly TimeSpan[] RetryLadder =
    [TimeSpan.FromMinutes(1), TimeSpan.FromMinutes(15), TimeSpan.FromHours(1), TimeSpan.FromHours(12)];

public async Task<int> DeliverBatchAsync(CancellationToken ct) {
    var now = time.GetUtcNow();
    var lockId = Guid.CreateVersion7();
    var claimed = await db.Deliveries.ClaimPendingAsync(new LeasedWorkClaim<Delivery> {
        LockId = lockId,
        NowUtc = now,
        LeaseUntilUtc = now + TimeSpan.FromMinutes(2),   // well above one batch's worth of calls
        BatchSize = 20,
        Eligible = d => d.CompletedAtUtc == null && d.Attempts < 8 && d.ExpiresAtUtc > now,
        OrderBy = rows => rows.OrderBy(d => d.CreatedAtUtc),
        // Count the attempt at claim: a worker that dies mid-call still uses one up.
        OnClaim = s => s.SetProperty(d => d.Attempts, d => d.Attempts + 1),
    }, ct);

    foreach (var delivery in claimed) {
        var outcome = await transport.SendAsync(delivery, ct);   // no lock, no transaction held
        var finished = time.GetUtcNow();
        var owned = outcome.Succeeded
            ? await db.Deliveries.FinalizeClaimAsync(delivery.Id, lockId, s => s
                .SetProperty(d => d.CompletedAtUtc, finished)
                .SetProperty(d => d.Payload, (byte[]?)null), visibleAfterUtc: null, ct)
            : await db.Deliveries.FinalizeClaimAsync(delivery.Id, lockId, s => s
                .SetProperty(d => d.Error, outcome.Error),
                finished + LeasedWorkBackoff.Ladder(delivery.Attempts, RetryLadder, outcome.RetryAfter), ct);
        if (!owned) logger.LogWarning("Delivery {Id} was reclaimed by another worker.", delivery.Id);
    }

    return claimed.Count;
}

Run the method from a scheduled job or a BackgroundService loop on every instance; the claim makes that safe. A failed attempt's finalize sets the retry visibility (LeasedWorkBackoff.Exponential, or LeasedWorkBackoff.Ladder, which honors a longer remote Retry-After), so a failing row waits instead of blocking the head of the queue. ReleaseClaimAsync hands back a row without bookkeeping, and RenewClaimAsync extends the lease for work that can outlive it.

Delivery stays at least once: a worker that crashes after the remote side accepted but before the finalize leaves the row to be retried. Give the remote side the row Id as its idempotency key where it accepts one.

The invariants and the extraction history are recorded in ADR-0073.