Telemetry & observability

Elarion emits OpenTelemetry-compatible signals through System.Diagnostics.ActivitySource and System.Diagnostics.Metrics. The runtime packages do not depend on the OpenTelemetry SDK — the host chooses exporters and registers the sources and meters it wants to collect. This keeps the framework lightweight while making every framework-owned boundary observable.

Registering sources and meters

using Elarion.AspNetCore;
using Elarion.Diagnostics;
using Elarion.Abstractions.Messaging;
using Elarion.Abstractions.Scheduling;
using Elarion.Caching;
using Elarion.Connections.Diagnostics;
using Elarion.Connections.Tcp.Diagnostics;
using Elarion.Resilience;

builder.Services
    .AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .AddSource(
            JsonRpcTelemetry.ActivitySourceName,
            SchedulerTelemetry.ActivitySourceName,
            HandlerCacheTelemetry.ActivitySourceName,
            ResilienceTelemetry.ActivitySourceName,
            HandlerTelemetry.ActivitySourceName,
            EventTelemetry.ActivitySourceName)
        /* add exporters */)
    .WithMetrics(metrics => metrics
        .AddMeter(
            JsonRpcTelemetry.MeterName,
            SchedulerTelemetry.MeterName,
            HandlerCacheTelemetry.MeterName,
            ResilienceTelemetry.MeterName,
            HandlerTelemetry.MeterName,
            EventTelemetry.MeterName,
            ConnectionTelemetry.MeterName,        // hosts using Elarion.Connections
            TcpConnectionTelemetry.MeterName)     // hosts using Elarion.Connections.Tcp
        /* add exporters */);

First-party surfaces

Elarion telemetry follows the OTel semantic conventions wherever one exists, and applies their spirit where none does:

• Durations are histograms recording "seconds as a floating point number with the highest precision available" (unit s), per the OTel instrument guidance — never milliseconds — with the semconv-recommended bucket boundaries supplied as instrument advice so exporters don't fall back to millisecond-scaled defaults. (Span attributes with an explicit unit suffix like scheduler.job.duration_ms are self-describing and exempt.)
• Semconv names are adopted, not paralleled: the JSON-RPC/MCP duration metric is rpc.server.call.duration per the current RPC conventions (the retired experimental rpc.server.duration, which was defined in milliseconds, is not emitted). Where no convention exists, names and attributes use a namespaced prefix (elarion.*, handler.*, actor.*, …).
• Cardinality discipline: metric tags stay bounded (type/operation/outcome names — never keys, payloads, or user identity); high-cardinality identity (an actor key, the current user) appears on spans and log scopes only.

Handler tracing

Handler tracing follows the same "instrument always, collect on demand" model as the other surfaces: the generator wraps every handler in an ObservabilityDecorator by default, but the span is a no-op until a host registers the Elarion.Handlers source. To suppress handler telemetry, omit HandlerTelemetry.ActivitySourceName/MeterName from the OpenTelemetry registration. This mirrors how HttpClient and EF Core ship always-present ActivitySources and leave collection to the host — no opt-in attribute required. The handler span adds the application-operation boundary as the stable parent of the decorator chain, so JSON-RPC, scheduler, cache, and resilience child spans all appear inside it.

One part of the decorator is deliberately not free-when-unobserved: context enrichment runs whenever enrichers are registered (current-user support registers one by default), because its consumers are the log scope and the ambient transport span — handler log lines should carry user context whenever logging is on, listener or not. That costs a small per-call allocation (~300 B with the default user-context enricher). For per-message hot paths where that matters, the leveled [HandlerTelemetry(HandlerTelemetryMode.None)] attribute — on a handler, a module class, or the assembly, nearest wins — makes the generator skip the decorator entirely for the affected handlers: no span, execution metric, enrichment, or log scope, and no wrapper object per call (ADR-0066). Failures are unaffected. This is a compile-time composition decision, distinct from the host-side "registered but unobserved" default above — see the low-allocation dispatch profile for the full recipe.

The span carries elarion.handler, elarion.handler.request_type, elarion.handler.outcome (ok/error/exception), and elarion.handler.pipeline — the decorators actually wrapping this handler in this process, in execution order (e.g. Observability,Authorization,Validation,Transaction,Audit?). A trailing ? marks a decorator that attached through a runtime gate — a soft-attached one whose presence depends on a registered service (the audit trail, the idempotency inbox) or a [DecoratorList] decorator with an AppliesTo predicate — so it answers "why didn't caching/auditing apply here?" straight from a trace. The same list is available programmatically as HandlerMetadata.Pipeline (IHandlerPipeline — its Steps and Contains(typeof(SomeDecorator<,>))), populated once the handler is first resolved. All tags are bounded (one constant value per handler type), so they are span-safe and never carry payloads or identity.

User & request context

Every handler span is enriched with who made the call, and a matching logging scope carries that context onto every log line the handler produces — so a trace or a log search can be filtered by user. This is a first-class part of handler tracing: the same ObservabilityDecorator that opens the span also runs the enrichers, so it tags the handler span (Activity.Current) and its log scope wraps the authorization, validation, and handler chain — a denied or invalid request is still attributed to its caller. (Tracing and context enrichment were two adjacent always-on decorators before ADR-0059 merged them into one.)

Because it runs in the handler pipeline (not an ASP.NET middleware), it works the same across every transport — JSON-RPC, [HttpEndpoint], MCP, scheduler jobs, and event consumers — reading the current user the dispatch scope already seeded. The decorator itself knows nothing about "user": it drains whatever the registered IHandlerContextEnricher instances contribute. The framework ships one — UserContextEnricher — that is registered by default when current-user support is added (AddElarionClaimsCurrentUser / AddElarionCurrentUser), so user context is on out of the box; anonymous executions (scheduler, after-commit delivery) contribute nothing.

By default it emits, from ICurrentUser:

user.id and user.roles are OpenTelemetry semantic-convention attributes; user.permissions mirrors them (no semantic convention exists for permissions). Email is PII and off by default; user identity is deliberately kept off metrics (unbounded cardinality) — it rides only the span and the log scope.

Configuring or disabling the built-in enricher

The built-in UserContextEnricher is registered by current-user support. Call AddElarionUserContextEnrichment to narrow the payload, opt into email, turn it off (host-registered enrichers keep running), or enable it for a host with a custom ICurrentUser that did not go through AddElarionClaimsCurrentUser:

using Elarion.Diagnostics;

builder.Services.AddElarionUserContextEnrichment(o =>
{
    o.IncludeEmail = true;    // opt into user.email / UserEmail (PII — you own redaction/retention)
    o.MaxItems     = 32;      // cap on roles/permissions joined into a tag (default 16)
});

// or disable the built-in user enricher (a host-provided IHandlerContextEnricher still runs):
builder.Services.AddElarionUserContextEnrichment(o => o.Enabled = false);

Contributing your own context

Enrichment is an open seam. Implement IHandlerContextEnricher to add your own trace tags and log-scope items — a tenant id, a request source, a correlation value — alongside (or instead of) the built-in user context. Enrichers are resolved per handler execution, so they may inject scoped services such as ICurrentUser, and registrations compose rather than replace:

using Elarion.Abstractions.Diagnostics;
using Elarion.Abstractions.Identity;

public sealed class TenantContextEnricher(ICurrentUser user) : IHandlerContextEnricher
{
    public void Enrich(HandlerEnrichmentContext context)
    {
        if (!user.IsAuthenticated) return;                     // anonymous transports run enrichers too
        foreach (var tenant in user.GetClaimValues("tenant"))
        {
            context.SetTag("tenant.id", tenant);               // OTel-style key on the span
            context.AddScopeItem("TenantId", tenant);          // PascalCase key in the log scope
            break;
        }
    }
}

// register — composes with the built-in user enricher
builder.Services.AddElarionHandlerContextEnricher<TenantContextEnricher>();

Keep the two key styles idiomatic to their sink — OpenTelemetry semantic-convention keys on tags (tenant.id), PascalCase keys in the scope (TenantId) — and keep Enrich cheap and non-throwing: it runs on every handler invocation. See ADR-0033.

Client-side telemetry

A frontend call can join the same distributed trace as the server work it triggers. The server side needs nothing Elarion-specific: a host running OpenTelemetry with AddAspNetCoreInstrumentation() extracts the W3C traceparent header from the incoming request, so the request span — and the JSON-RPC dispatch, [HttpEndpoint], and handler-pipeline spans under it — nest beneath the client's span. The client just has to start that span and inject the header:

• JSON-RPC client — the generated TypeScript client stays OpenTelemetry-package-free and exposes an optional instrumentation hook (RpcInstrumentation) on createRpcClient/createRpcApi: supply an adapter over @opentelemetry/api (or a hand-rolled traceparent generator) to start a span per request/batch, inject trace-context headers, and record the outcome. See TypeScript client › Tracing with OpenTelemetry.
• REST / OpenAPI client — the recommended @hey-api/openapi-ts client is fetch-based, so @opentelemetry/instrumentation-fetch (browser) or @opentelemetry/instrumentation-undici (Node) auto-injects traceparent; a request interceptor can also inject it by hand. See OpenAPI › Trace client requests.

Either way the generated client never imports a tracing SDK, so client-side tracing stays a host decision.