Architecture

Filomena is an asynchronous alert diagnosis service. The webhook persists alert lifecycle events before placing them on a bounded in-memory queue. Workers recover persisted, retryable events after restarts.

Processing lifecycle

1. Authenticate and parse the Grafana webhook.
2. Normalize each alert and compute its event ID and fingerprint.
3. Insert the lifecycle event into the configured state store.
4. Resolve the canonical service identity and attach the event to an incident.
5. Deduplicate completed runs inside the configured window.
6. Load human-confirmed cases for the same alert.
7. Select matching diagnostic packs.
8. Query logs, metrics, traces, topology, infrastructure and changes concurrently with source-specific bounds.
9. Build a valid JSON prompt with separate alert, prior-case and evidence budgets.
10. Request a strict structured diagnosis from the analysis provider.
11. Reject citations to evidence that was not collected.
12. Persist the run and update the incident before notifying Slack or application logs.
13. Retry classified transient failures with exponential backoff.

Service identity and incidents

Canonical service identity uses OpenTelemetry service, deployment and Kubernetes resource attributes with compatibility aliases for existing Grafana labels. Ownership, repository and runbook metadata travel with the identity.

An explicit incident label correlates alerts across services. The fallback correlation key is the service namespace, name, environment and cluster. An incident remains open until the latest lifecycle event for every member is resolved. Kubernetes replica alerts share a workload fingerprint while the pod and container identify the incident member. Diagnosis revisions update the incident without discarding earlier triage runs.

Operational changes are immutable events keyed by a caller-supplied change ID. The change diagnostic pack selects events for the canonical service and alert window. Trace evidence uses Tempo, and topology evidence uses OpenTelemetry service-graph metrics.

Event identity

An event is one alert lifecycle transition: its fingerprint, its status and its start and end time. A repeated firing notification for the same alert start is therefore the same event, is stored once and is diagnosed once. A new start time, or a change to or from resolved, is a new event. DedupWindowMinutes suppresses a second diagnosis when the same fingerprint produces a new event inside the window.

An attempt that collected no evidence is retried inside the attempt cap, because evidence can arrive later than the alert that reports it. A suppressed repeat and a resolution are terminal and are never retried.

Prior cases

A prior case is a completed run for the same alert that a human rated, newest first, capped at five. A case counts only when its latest feedback teaches something: a correct or partial verdict, or a corrected root cause or resolution. A verdict the reviewer later withdraws stops being a prior case.

Retrieval reads every rated run for that one fingerprint rather than a recent window, so an older confirmed case cannot be hidden by newer runs whose feedback teaches nothing. The number of such runs is bounded by how often a person rates one alert. Every state provider push the fingerprint and the rating filter into a single query and share one selection function, so neither provider carries its own copy of the rule.

Offline evaluation

Export writes stored runs as cases, replay asks a candidate model for a fresh diagnosis of the same evidence, and score reports one summary. Compare scores both and returns a verdict: a metric with a known good direction may not move against it beyond a tolerance, and the candidate must cover every case the baseline did. The exit code carries the verdict, so the comparison can gate a change instead of only describing it.

Safety invariants

• Alert labels, annotations, logs and prior cases are untrusted model input.
• Diagnostic queries apply resource selectors before row limits.
• Each evidence source has a quota so one adapter cannot evict all other findings.
• Prompt truncation removes whole findings and low-priority fields while preserving valid JSON.
• The response schema restricts evidence citations to IDs collected for the current run.
• Recommended actions are limited to read-only investigation steps.
• Slack request signatures expire after five minutes.
• Notification failures do not change a completed diagnosis into a failed run.
• A completion the analysis provider cannot deliver in full is retried, not treated as a diagnosis. Truncated and malformed responses take the same path as an invalid one.

Extension points

Public domain models and provider contracts live in Filomena.Core. The API composes concrete providers at startup:

• Filomena.Storage.Postgres implements the recommended production state store.
• Filomena.Storage.Sqlite implements the local and lightweight state store.
• Filomena.Storage.ClickHouse implements the legacy ClickHouse state store.
• Filomena.Evidence.ClickHouse implements OpenTelemetry and Application Gateway log evidence.
• Filomena.Evidence.Loki implements LogQL log and Application Gateway access-log evidence.
• Filomena.Evidence.Prometheus implements instant and range metric evidence.
• Filomena.Evidence.Kubernetes implements pod, event and current or previous container-log evidence.
• Filomena.Evidence.ArgoCd implements application, condition, resource and operation evidence.
• No-op providers disable a capability without disabling unrelated diagnostic packs.

IStateStore covers event deduplication, retry recovery, prior cases, related alerts, notification delivery state, run queries and evaluation export. Its narrower inherited interfaces let consumers request only the operations they use.

Evidence contracts are split by capability: ILogEvidenceProvider, IAccessLogEvidenceProvider, IMetricsEvidenceProvider, IKubernetesEvidenceProvider and IArgoCdEvidenceProvider. The API and diagnostic packs do not reference provider clients, database contexts or source schemas.

IDiagnosticAdapter is the diagnostic-pack contract. Each adapter declares catalog metadata, decides whether it applies to an alert and returns findings with stable evidence IDs. Filomena.Packs provides the packageable base class for external pack projects.

Runtime-loaded binary plugins are not supported. Providers compile with the application so dependency and permission boundaries remain explicit.

Persistence

Every state adapter stores:

• alert_events
• triage_runs
• triage_feedback
• slack_deliveries
• operational_changes
• incidents
• incident_alerts
• run_verifications
• audit_events

PostgreSQL is the default for Compose and Helm production installations. Filomena creates and versions its tables in an existing database. The connection string is supplied through a secret.

The SQLite database must be on a durable volume. It is intended for a single Filomena instance. It runs in write-ahead logging mode, so a query never blocks the writer, and it keeps -wal and -shm files beside the database.

Version 0.1 deliberately uses one active instance. Filomena is an asynchronous, read-only investigation aid, and alert delivery and system remediation do not depend on its availability. Kubernetes restarts the instance, persisted events recover on startup, a PodDisruptionBudget blocks voluntary eviction, and every deployment upgrade has a brief documented interruption. This availability contract is acceptable for the initial internal production release. It is not an HA contract. Multiple active replicas require a state provider with distributed uniqueness and incident-update concurrency before the chart may allow replicaCount above one.

The ClickHouse state provider uses the schema under deploy/clickhouse/init. Alert and run history have a one-year TTL. Feedback has a two-year TTL. Its event deduplication is process-local because MergeTree does not enforce unique keys, so this provider must also run as a single Filomena instance.

When enabled, the ClickHouse log provider reads OpenTelemetry logs from otel.otel_logs. Filomena never writes to telemetry tables. State and log providers are configured independently.