Diagnostic packs

A diagnostic pack is an IDiagnosticAdapter implementation that turns an alert into bounded evidence queries.

Contract

Each pack must:

• match only alerts it understands
• require enough labels to scope every query
• push resource filters into the source query
• cap returned rows and findings
• use stable evidence IDs, unique inside the pack
• return collection errors without fabricating findings
• recommend no mutation
• include query translation tests
• include a real source integration test when the backend has its own query language

Built-in packs

• ActiveMQ and Artemis
• ArgoCD
• Azure Application Gateway
• ExternalSecret
• Kubernetes Job
• controller-runtime
• Kubernetes workload health
• deployment and image changes
• related firing alerts
• canonical service identity and ownership
• ingested operational changes
• Tempo trace summaries
• OpenTelemetry service dependencies

The generic log collector runs alongside matching packs. Fair-share allocation reserves findings per source.

The collector qualifies every evidence ID with the profile of the pack that produced it, so two packs cannot collide and silently evict each other's findings. A pack that throws becomes a collection error for that profile and does not fail the run or the other packs. A duplicate ID inside one pack is reported as a collection error rather than dropped silently.

Naming

A pack that matches on a service or alert name reads its candidates from the Naming settings instead of hard-coding a local convention. Label-based matching stays in the pack, because label names belong to the exporting system. See configuration.

Adding a pack

Reference Filomena.Packs and derive from DiagnosticPack. The descriptor is part of the runtime catalog, so declare the pack ID, version, category, required labels and required signals. Constructor parameters can use the public evidence-provider interfaces from Filomena.Core.

public sealed class CertificatePack(IMetricsEvidenceProvider metrics) : DiagnosticPack
{
    public override DiagnosticPackDescriptor Descriptor { get; } = new(
        "certificate",
        "Certificate expiry",
        "security",
        "1.0.0",
        "Certificate expiry and renewal evidence.",
        ["service.name"],
        ["Prometheus"]);

    public override bool Matches(AlertEvent alertEvent) =>
        alertEvent.AlertName.Contains("certificate", StringComparison.OrdinalIgnoreCase);

    public override Task<DiagnosticAdapterResult> CollectAsync(
        AlertEvent alertEvent,
        int windowMinutes,
        CancellationToken cancellationToken) =>
        Task.FromResult(new DiagnosticAdapterResult(Profile, [], []));
}

1. Implement and package the adapter in its own project.
2. Register it as IDiagnosticAdapter in the Filomena distribution.
3. Add focused applicability and query tests.
4. Document required labels, metrics and permissions.
5. Add a synthetic failure fixture suitable for offline evaluation.

GET /v1/packs is the authoritative enabled catalog for a running instance. A pack missing from that response is not registered in that distribution.

Diagnostic packs must remain read-only. Remediation belongs in an external workflow system with its own authorization and approval model.

Datadog logs

The datadog log provider queries POST /api/v2/logs/events/search, scoped to the alert's service, Kubernetes namespace, pod and container tags, restricted to Datadog's normalized error, critical, alert and emerg statuses, with the run's time window and finding limit pushed down to the request. Tag values are sent as quoted literals, so a wildcard, boolean operator or range expression arriving in an alert label cannot widen the search. A query with nothing to scope it by is not sent at all.

Tag names default to Datadog's own conventions (service, kube_namespace, pod_name, kube_container_name) under Logs:Datadog, separate from the Loki label settings, and can be overridden per deployment.

It supplies error logs only. Access-log summaries stay with the ClickHouse and Loki providers, so selecting datadog leaves access-log evidence empty rather than unavailable.

The request and response shapes follow Datadog's documented v2 logs API and are covered by unit tests against a stubbed transport. They have not been exercised against a live Datadog account.