Loki: log aggregation as the SME-friendly alternative to Elasticsearch

What is Loki?

Loki is the log-aggregation component of the Grafana ecosystem. Introduced by Grafana Labs in 2018, now in version 3.x (May 2026), and very actively developed. Core idea: logs are stored in Loki but not full-text indexed like in Elasticsearch. Instead, only metadata labels (container_name, log_level, service, environment) are indexed. The log content itself sits in chunks on S3, filesystem, or MinIO and is only read on a concrete query.

The effect: Loki typically uses 10 to 20 percent of the storage of a comparable Elasticsearch stack. A fiduciary platform with 30 days of log retention uses maybe 15 GB in Loki, 100+ GB in Elasticsearch. RAM footprint is also markedly lower: Loki single-binary runs well with 1 GB RAM, Elasticsearch needs at least 8 to 16 GB.

Licence: AGPL-3. Stricter than Apache 2.0 (Prometheus) – anyone modifying Loki and offering it as a service must release source under AGPL. For typical SME use (self-operating, own logs) AGPL is no obstacle. Anyone reselling Loki as part of a commercial SaaS should check the licence with a lawyer – Grafana Labs then also offers commercial licences.

Loki does not replace a logging framework – it is the storage and query backend. Logs are ingested via Promtail, Fluent Bit, Vector, or directly over HTTP. The frontend is always Grafana – Loki has no UI of its own beyond a slim debugging console.

Why it matters

Logs are often the most important forensic tool for an SME – not metrics. When a client login fails, an n8n workflow drops silently, or an LLM request returns a 503, the first question is always: what do the logs say? Without central log aggregation you must SSH into 25 containers and grep docker logs. With Loki the answer is back in two seconds.

CH DSG fit: Loki runs self-hosted on Hetzner Falkenstein. Logs frequently contain personal data (IP addresses, usernames, sometimes request bodies) – that cannot leave EU/CH without a transfer impact assessment. Self-hosted Loki solves the problem at the root: no data to the US, no data processing agreement debates.

SME suitability: setup is markedly simpler than Elasticsearch. A single Docker container with filesystem backend runs immediately. Promtail as sidecar or Docker logging driver collects logs automatically. For 5 to 25 containers, single-binary Loki is plenty – no cluster, no S3, no replication needed.

Audit-trail compliance: for fiduciary clients, OR 957a demands a bookkeeping audit trail; for AI systems, the EU AI Act demands documented logging of decisions. Loki is a practical backend for that – logs are centrally gathered, retained for 12 months (retention config), and queryable ad-hoc via LogQL. Read-only auditor access can be mapped through Grafana permissions.

May 2026 – bloom filters: the latest Loki 3 line introduces bloom filters for optional full-text indexing per stream. That drastically speeds substring searches across long time ranges (factor 5-10) and closes one of the last performance gaps against Elasticsearch.

How it works

Loki has three components that can run in a single binary (single-binary mode) or distributed across nodes (microservices mode). Single-binary is the SME default.

Data flow: (1) agents like Promtail, Vector, or Fluent Bit read logs (from containers, files, or syslog) and apply labels. (2) Logs are pushed to Loki via HTTP/gRPC. (3) Loki writes chunks to backend storage (filesystem or S3) and indexes only labels. (4) Grafana queries via LogQL.

docker-compose.yml example for an SME setup:

```yaml services: loki: image: grafana/loki:3.4.0 volumes: - ./loki-config.yml:/etc/loki/local-config.yaml - loki-data:/loki command: -config.file=/etc/loki/local-config.yaml ports: ["3100:3100"] restart: unless-stopped promtail: image: grafana/promtail:3.4.0 volumes: - /var/log:/var/log:ro - /var/lib/docker/containers:/var/lib/docker/containers:ro - ./promtail-config.yml:/etc/promtail/config.yml command: -config.file=/etc/promtail/config.yml restart: unless-stopped volumes: loki-data: ```

LogQL is the query language, syntactically close to PromQL. Examples: `{container="n8n"}` (all logs from the n8n container), `{container="n8n"} |= "error"` (with substring "error"), `{container="n8n"} |~ "5\d\d"` (regex on HTTP 5xx status), `{container="n8n"} | json | duration_ms > 1000` (parse JSON, then filter by duration > 1s).

Retention: Loki supports global and stream-specific retention. Default config: 30 days global plus 12 months for streams labelled `audit=true`. That lets audit logs live longer than routine logs. The compactor job runs daily and deletes expired chunks.

Bloom filters (May 2026): optional feature in Loki 3.4+. Produces a bloom filter per chunk, drastically speeding substring searches. Enable per stream – useful for high-volume streams with frequent ad-hoc searches.

Setup in 5 steps

1. 01Create docker-compose with grafana/loki:3.4 and grafana/promtail:3.4, filesystem backend, single-binary mode.
2. 02loki-config.yml with 30-day retention by default plus 12 months for streams labelled audit=true; storage_config: filesystem chunks dir.
3. 03promtail-config.yml with docker_sd_configs on /var/lib/docker/containers – automatic label extraction (container_name, log_level).
4. 04Wire Loki as a Grafana datasource, run the test query {container=~".+"}, verify JSON parsing in an Explore test.
5. 05Define alerts via Grafana Alerting on LogQL queries (e.g. rising 5xx errors), receivers to Telegram or Slack.

When to use Loki

Loki is the right choice when (a) logs from multiple containers or hosts must be centrally collected, (b) Grafana is already in the stack (or planned), (c) Elasticsearch complexity should be avoided, and (d) log volume per day sits between a few MB and around 10 GB.

Concrete use cases: a fiduciary platform with 25 Docker containers (n8n, PostgreSQL, Express APIs, cron workers, LiteLLM gateway) whose logs need central searchability. A law firm with audit-trail duties – Loki collects all access events and keeps them searchable for 12 months. A RAG system with LangChain workers whose errors happen silently – Loki delivers the line in under two seconds.

Also useful: as an audit backend for LLM requests. Every request to OpenAI/Anthropic/DeepSeek is logged with client ID, model, tokens, and response latency – compliance reports and cost analyses derive from that. At Fairlane, exactly this pattern runs in production since 2024.

When not to use

Loki is the wrong choice when (a) pure full-text search across very large log volumes (>100 GB per day) is the main use case – Elasticsearch or OpenSearch fit better, (b) the stack is OpenTelemetry-first end to end – then SigNoz is the more coherent pick, (c) the team has no Grafana and does not want one – Loki without Grafana is awkward.

Pitfalls: running Loki without a retention policy – logs grow unbounded and the host fills up. Using too many distinct labels (high cardinality) – the Loki index explodes just like Prometheus. Pushing all logs without filter – Sentry stack traces, health-check pings, debug output – burns storage without value.

Anti-patterns: running Loki and Elasticsearch in parallel – double maintenance without benefit. Loki without Promtail or Vector – direct HTTP push from each app is possible but tedious and error-prone. Loki multi-tenant without a plan – multi-tenancy exists in Loki, but it needs explicit configuration and is not on by default.

Trade-offs

FAQ

Related topics

Sources

1. Grafana Loki – Documentation · 2026-05
2. Grafana Loki 3 – Release notes and bloom filters · 2026-04
3. Grafana Labs – Loki vs Elasticsearch comparison · 2026-03
4. Promtail – Configuration reference · 2026-04