Loki-VL-proxy

📖 Project site: reliablyobserve.github.io/Loki-VL-proxy — full docs, benchmarks, runbooks, comparison matrix.

How it works (TLDR)

LogQL queries arrive → parsed into a typed AST (internal/logql) → translated into LogsQL via a typed builder (internal/logsql) → sent to VictoriaLogs.

Both parsers are hand-written recursive descent. The LogQL side handles the full Loki grammar. The LogsQL side uses a builder API that produces typed, syntactically valid LogsQL at construction time. Translation uses two tiers: stable string operations for well-understood paths (stream selectors, line filters), and typed AST construction for complex paths (stats aggregations, binary metric expressions, PipeMath/PipeStats/PipeFilter nodes).

Label metadata and cache reuse: /labels and /label/{name}/values cover the requested range, using metadata inventories and cached time buckets rather than only a recent slice. Settled buckets can be reused from memory, disk and peers; recent edges and buckets due for revalidation still read VictoriaLogs. Near-now freshness, empty answers, stale-on-error responses and optional indexed browse limits follow the configured rules in Metadata caching. A cache hit is not a guarantee of immediate visibility after retention or deletion; distributed deletion-generation metadata is not implemented.

Moving JSON metric windows: complete raw JSON count_over_time and rate answers can reuse settled evaluation points from the Loki-facing RAM/SSD/peer cache, preserving their original age and step phase. Missing edges are evaluated again; native field histograms and truncated answers retain their existing rules. See the production troubleshooting signals and 48h three-node comparison for the measured benefit and costs.

Fleet restart behavior: disk restore, startup jitter and peer warmup can reduce repeated VictoriaLogs reads. Their effect depends on surviving cache state, owner/replica availability and freshness. The fleet documentation describes the supported recovery cases; it does not guarantee fleet-wide cold-miss coordination or that only one pod ever reads the backend.

Keep your entire Loki stack — Grafana Explore, Drilldown, dashboards, API tooling — and run it on VictoriaLogs.

• Drop-in Loki API. Point your existing Grafana Loki datasource at the proxy. Zero plugin changes and no query rewrites for supported LogQL; measured gaps are listed in compatibility gaps and Known Issues.
• Measured resource difference. At 310 GiB/day ingest: VL + proxy runs on 1.4 cores and 6.1 GiB RAM. Loki's published minimum for that ingest class: 38 cores, 59 GiB. That gap is real — not a benchmark artifact.
• Proxy intelligence built in. Disk-backed label cache with keep-warm loop, full-range metadata on the first response, time-bucketed keys, adaptive parallelism, circuit breaker, rate limits, tenant isolation. Fleet restart safety: jitter + peer-first warmup keeps rolling restarts from thundering VL. One ~14 MB static binary.

Quality and performance vs Loki (current)

Tested against Loki v3.7.9, the release the compatibility stack and CI pin. Timings were measured on 2026-11-09 at commit 0843224237 on the project's A/B stack: the same data, the same hardware, the latest valid weekly run of each of 39 query-shape sets over 1h to 24h. A pair is one query shape over one time range; "comparable" means both sides answered 200 and Loki's results cache did not answer the first request. Faster and slower mean beyond 25% and 50 ms; everything else is on par. A topk or bottomk answer is compared by its ranked points and their sum, not by its series count.

Loki profile shown: loki-defaults, Loki with its own defaults (align_queries_with_step off, max_cache_freshness_per_query 10m); proxy builds run with their defaults. The aligned profile (the stack's Loki (align_queries_with_step on, max_cache_freshness_per_query 0s)) is measured too: the distribution table below shows both profiles, the rest of this section the one shown. The proxy serves Loki's read API; write endpoints such as push are not supported (ingest goes to VictoriaLogs directly). Optional deletion management is outside read API scores: its opt-in handler does not implement VictoriaLogs deletion. The alerting-rules read API remains in scope, including /api/prom/rules* for vmalert. The deprecated read aliases /api/prom/label, /api/prom/label/{name}/values, /api/prom/query, /api/prom/series and /api/prom/tail are unsupported and excluded from totals, gaps and compatibility scores (owner 2026-10-31); use the corresponding /loki/api/v1 routes. No other write endpoint is planned.

🟢 Where the proxy is better than Loki

• First request: 329 of 568 comparable pairs are faster (58%).
• Drilldown and Explore pages: 85 of 118 faster (72%); median first request 0.08 s against Loki's 0.29 s.
• Metric queries: 225 of 363 faster (62%); median first request 0.07 s against 0.19 s.
• Label and label-value calls: median repeat 0.00 s against Loki's 0.02 s, over 36 pairs (36 of them answered by Loki's results cache).
• Share of rows below Loki, first request: 57.9% (329 of 568 rows); repeat: 10.0% (76 of 762). The target is most queries below Loki (above 50%).

🟡 Where it is on par

• First request: 226 of 568 pairs are within noise (40%).
• Log queries: 60 of 79 on par; metric queries: 131 of 363; Drilldown and Explore pages: 29 of 118.
• Rows close to Loki (within the same noise): 226 on the first request, 684 on a repeat.
• Repeat of the same query, median p50: Drilldown and Explore pages 0.01 s against 0.03 s; metric queries 0.01 s against 0.02 s; log queries 0.01 s against 0.04 s.
• Answers: 716 of 782 pairs return the same result as Loki (92%).
• sum_over_time, avg_over_time, max_over_time and min_over_time over an unwrapped field answer Loki's window on query_range for a range longer or shorter than the step (some outer aggregations of them are still listed as open differences).
• An unwrap metric moved forward by a refresh or a re-run reads from VictoriaLogs only the buckets it has not read before.
• A zero divisor of / and % is NaN, like Loki.
• Binary vector matching raises Loki's duplicate-series error text at the first step where Loki's join fails.
• Eligible shared cache keys use owner plus one backup by default (-peer-replication-factor=2); RF=1 keeps only the owner. Nonowner RAM shadows are bounded to 30s and 10% of L1 capacity without extra disk copies. Three-node tests cover pooled capacity, disk refill and owner loss.

🔴 Where it is worse

• First request: 13 of 568 pairs are slower (2%): log queries 0 of 79, Drilldown and Explore pages 4 of 118, metric queries 7 of 363.
• Between 1× and 2× Loki: 8 rows on the first request, 2 on a repeat; above 2× Loki: 5 on the first request and 0 on a repeat (the 2026-11-09 distribution).
• 3 shape, range and mode rows are on the above-2× list for the loki-defaults profile (more than 2× Loki and more than 50 ms slower): 3 on the first request, 0 on a repeat, 0 on a refresh. Each is a defect being fixed: a row joins the list after two consecutive weekly runs above 2× and leaves it after two within 2×, so it can hold more or fewer rows than the day's distribution above; a change that adds or worsens a row is flagged. 29 of 108 covered registry items are locked at the 2× bound.
• Largest gaps on that list in the run shown, first request: DP2 patterns of an environment over 1h, 1.18 s against 0.21 s; LS8 json count by level_extracted, collision over 6h, 0.41 s against 0.13 s; DP1 patterns of a plain-text service over 1h, 0.26 s against 0.02 s.
• Answers: 51 of 782 pairs return a different result from Loki, and 15 could not be compared.

Distribution versus Loki in the run shown, both Loki profiles

Rows compared: both sides answered 200 and, on a first request, Loki's results cache did not answer. Bands are those of bench/ab/over2x.py: beyond 25% and 50 ms, and above 2× Loki also more than 50 ms behind. The aligned profile runs the range-metric shapes only (the queries its setting changes).

Loki profile	Request	Rows compared	🟢 Below Loki	🟡 Close to Loki	🟠 1–2× Loki	🔴 Above 2× Loki	Not judged: Loki not 200 / proxy not 200 / first request from Loki's cache
loki-defaults	first	568	329 (57.9%)	226 (39.8%)	8 (1.4%)	5 (0.9%)	20 / 0 / 194
loki-defaults	repeat	762	76 (10.0%)	684 (89.8%)	2 (0.3%)	0 (0.0%)	20 / 0 / —
aligned	first	444	308 (69.4%)	123 (27.7%)	10 (2.3%)	3 (0.7%)	19 / 0 / 82
aligned	repeat	526	15 (2.9%)	508 (96.6%)	3 (0.6%)	0 (0.0%)	19 / 0 / —

By workload in the run shown

First request and repeat of the comparable pairs of the loki-defaults run, as p50 · p90 · slowest pair: the median hides the heavy queries, the p90 and the slowest show them. The counts are of pairs, with the rule above (🟢 faster, 🟡 on par, 🔴 slower). A repeat is the recorded p50 of the repeated request, which may be a cache hit on either side.

Workload	Pairs	🟢 / 🟡 / 🔴	First request: proxy	Loki	Repeat p50: proxy	Loki
Metadata and label browser	8	0 / 6 / 2	0.08 · 1.18 · 1.18 s	0.07 · 0.27 · 0.27 s	🟡 0.02 s	0.06 s
Drilldown and Explore pages	118	85 / 29 / 4	0.07 · 0.74 · 4.57 s	0.27 · 3.58 · 9.40 s	🟡 0.01 s	0.03 s
Log queries	79	19 / 60 / 0	0.01 · 0.05 · 0.13 s	0.04 · 0.25 · 1.20 s	🟡 0.01 s	0.04 s
Metric queries	363	225 / 131 / 7	0.07 · 0.74 · 4.50 s	0.19 · 3.23 · 32.24 s	🟡 0.01 s	0.02 s

By time range, first request

How each side grows with the range a user picks: p50 · p90 · slowest pair of each workload and range.

Workload	Range	Pairs	🟢 / 🟡 / 🔴	Proxy	Loki
Drilldown and Explore pages	1h	40	25 / 13 / 2	0.03 · 0.07 · 0.27 s	0.13 · 0.20 · 0.27 s
Drilldown and Explore pages	6h	36	25 / 11 / 0	0.10 · 0.51 · 0.80 s	0.67 · 1.00 · 1.94 s
Drilldown and Explore pages	24h	40	34 / 4 / 2	0.40 · 1.57 · 4.57 s	3.32 · 4.13 · 9.40 s
Log queries	1h	31	1 / 30 / 0	0.04 · 0.06 · 0.08 s	0.03 · 0.06 · 0.83 s
Log queries	3h	8	3 / 5 / 0	0.01 · 0.02 · 0.02 s	0.04 · 0.11 · 0.11 s
Log queries	6h	20	8 / 12 / 0	0.01 · 0.03 · 0.04 s	0.04 · 0.15 · 0.25 s
Log queries	24h	20	7 / 13 / 0	0.01 · 0.04 · 0.13 s	0.04 · 1.03 · 1.20 s
Metric queries	1h	129	65 / 62 / 2	0.02 · 0.09 · 0.25 s	0.10 · 0.23 · 3.10 s
Metric queries	6h	100	75 / 23 / 2	0.12 · 0.41 · 1.35 s	0.54 · 1.08 · 11.05 s
Metric queries	24h	98	81 / 14 / 3	0.42 · 1.67 · 4.50 s	2.50 · 4.25 · 23.07 s

The heaviest queries for Loki in the run shown

The 10 comparable pairs Loki takes longest to answer on a first request: the queries a user waits for. The multiple is Loki's time over the proxy's, or the reverse when the proxy is slower.

Query shape	Range	Workload	Proxy	Loki	Proxy vs Loki
F1 sum by over a ratio of two per-stream metrics, 20s step	24h-20s	Metric queries	3.03 s	32.24 s	🟢 10.6× faster
B2 sum by over a ratio of two per-stream metrics	24h	Metric queries	2.97 s	23.07 s	🟢 7.8× faster
F3 avg by service of rate, 20s step	24h-20s	Metric queries	1.48 s	11.83 s	🟢 8.0× faster
B2 sum by over a ratio of two per-stream metrics	6h	Metric queries	0.75 s	11.05 s	🟢 14.6× faster
DB1 label breakdown pod	24h	Drilldown and Explore pages	0.90 s	9.40 s	🟢 10.4× faster
F2 count by service of count_over_time, 20s step	24h-20s	Metric queries	0.01 s ‡	9.12 s	🟢 1824.2× faster
F4 sum of max by service,app, 20s step	24h-20s	Metric queries	3.14 s	9.04 s	🟢 2.9× faster
logfmt volume	24h	Metric queries	1.04 s	5.66 s	🟢 5.4× faster
SF9 volume, logfmt label_format rename	24h	Metric queries	2.55 s	5.59 s	🟢 2.2× faster
logfmt count by level, filter, drop	24h	Metric queries	1.92 s	5.45 s	🟢 2.8× faster

‡ The proxy answered a first request over 6h or more in under 50 ms: probably from its own cache, warmed by an earlier set of the weekly run that sent the same query. The run records Loki's cache counters but not the proxy's, so this row is not a cold proxy.

Where the proxy loses most in the run shown

Query shape	Range	Workload	Proxy	Loki	Proxy vs Loki
DP1 patterns of a plain-text service	1h	Metadata and label browser	0.26 s	0.02 s	🔴 12.0× slower
DP2 patterns of an environment	1h	Metadata and label browser	1.18 s	0.21 s	🔴 5.6× slower
LF9 drilldown logs, level pill, logfmt service	24h	Drilldown and Explore pages	0.13 s	0.03 s	🔴 5.0× slower
LS8 json count by level_extracted, collision	6h	Metric queries	0.41 s	0.13 s	🔴 3.1× slower
E7 json count by level_extracted	1h	Metric queries	0.10 s	0.04 s	🔴 2.2× slower

Label browser and Drilldown metadata calls: every pair both sides answered with 200, including the first requests Loki's results cache answered (counted in the third column; a Loki time near zero there is its cache). Medians.

Call	Pairs	Loki cache-answered	First request: proxy	Loki	Repeat: proxy	Loki
Drilldown detected labels and fields	15	15	🟡 0.04 s	0.03 s	🟡 0.00 s	0.01 s
Label values	18	18	🟡 0.00 s	0.04 s	🟡 0.00 s	0.04 s
Labels (label browser)	18	18	🟡 0.00 s	0.02 s	🟡 0.00 s	0.02 s
Volume (Drilldown landing)	9	1	🔴 0.21 s	0.09 s	🟢 0.02 s	0.08 s

Quality of the Loki read API

	Now
Loki endpoints the proxy routes	25 of 27 (93%)
Endpoints with tests that compare the proxy against Loki	19 of 27 (70%)
Endpoints whose registry state is proven	12 of 27 (44%)
Routed endpoints answered natively by VictoriaLogs	6 of 25
Behaviour and feature items	6 proven, 21 partial, 3 gaps
Known differences from Loki (ranked)	108: 85 open, 13 documented, 7 owner-kept, 3 upstream
Open differences visible in Grafana (registry impact)	61 of 85: 46 Explore, 14 Logs Drilldown, 1 Grafana datasource; 10 API only, 14 edge cases

Readiness by client

The Loki endpoints each client calls (registry consumers), with the same evidence as above, and the open differences filed as visible to that client.

Client	Loki endpoints it calls	Routed	Compared with Loki in tests	Proven	Open differences it shows
Grafana Explore	10	10 (100%)	10 (100%)	8 (80%)	46
Logs Drilldown	11	11 (100%)	11 (100%)	9 (82%)	14
Grafana Loki datasource	6	6 (100%)	6 (100%)	4 (67%)	1
Loki API clients	27	25 (93%)	19 (70%)	12 (44%)	10

Read routes not implemented: /loki/api/v1/cache/generation_numbers, /loki/api/v1/index/shards (optional planner gap, low priority).

These are registry evidence counts, not a full runtime compatibility percentage. Optional deletion management is excluded from the read totals; deprecated aliases and write-only APIs are outside this score.

The 5 open differences with the most impact (all explore-visible): parsed-identity/bare-parser-metric-duplicates-extracted (bare JSON metrics cannot distinguish equal-valued metadata from ingested body fields), semantics/parse-error-labels-on-log-lines (A line a parser rejects carries error and error_details in Loki's log answer), semantics/tail-history-start-limit (native tail ignores start and limit: Grafana Live shows no history where Loki shows its last 100 lines), semantics/unwrap-per-series-identity-raw-rows (A function taken per series (avg, max, min over time, rate_counter) under sum or sum by uses the stream as the series), semantics/binary-ignoring-matching (+ ignoring(label) between two sums joins the wrong series). Differences kept on purpose are listed as documented or owner-kept.

Details, the per-workload tables and the method: performance summary · performance guide · ranked differences · coverage map · Known Issues.

Two-day (48h) ranges, measured separately on 2026-11-04 (commit 69d07019b3, Loki v3.7.9, VictoriaLogs v1.53.0, both Loki profiles): on a first request the proxy is faster than Loki on 81–83% of compared rows and above 2x Loki on at most 2 rows (1.4%), both waiting for a confirming run; a long-range load test kept /labels at 249 of 249 answers 200 with no VictoriaLogs or Loki restart. One open defect shows only at this range: answers larger than the buffered-response limit (B2, and the same shapes at a 20 s step) answer 500 where Loki answers 200. Details: two-day ranges.

---

One-glance comparison

The current position is generated from the latest weekly A/B run in Quality and performance vs Loki above: the distribution of every shape and range against Loki in both Loki profiles (🟢 below Loki, 🟡 close, 🟠 1–2×, 🔴 above 2×), the medians by workload, and the label browser and Drilldown metadata calls. The rules: a pair is one query shape over one time range, faster or slower means beyond 25% and 50 ms, and a first request Loki's results cache answered is not judged.

Method, every shape and range, the open gaps and the regression gate: performance evidence · performance guide · ranked differences. The earlier 2026-06-05, 36-query Drilldown comparison (including CPU and memory measurements) remains in the historical benchmark, with its reproduction script and raw results.

---

Query Performance

Latency against Loki by workload and for the label browser and Drilldown metadata calls is generated from the latest weekly A/B run in Quality and performance vs Loki; every shape and range is in the performance evidence.

Translation overhead

Measured on 2026-11-09 at commit 0843224237 by the weekly run: LogQL to LogsQL translation took 4.8 to 23.7 µs per query. For a 100 to 500 ms VictoriaLogs round-trip this is under 0.02% of wall time.

Query type	Time	Allocs
Selector {app="nginx"}	4.8 µs	23
rate({app="nginx",env="production"}[5m])	11.2 µs	74
sum(rate({...}[5m])) by (host)	10.8 µs	66
ip("10.0.0.0/8") line filter (native ipv4_range())	8.9 µs	41
Binary metric sum(rate) / sum(rate)	23.7 µs	116

Median of 5 runs of go test ./internal/translator/ -bench BenchmarkTranslate -benchmem with GOMAXPROCS=4, go1.27.3 on AMD EPYC 9V74 80-Core Processor (amd64), the weekly run's machine, which runs other work too: the times are an upper bound.

Throughput under load (historical)

Historical: measured on proxy v1.50.0 with Loki 3.6.x and VictoriaLogs v1.50.0 using loki-bench; not re-measured. It compares a warm, cache-assisted proxy with Loki on a synthetic mix and is not the current position. The method and the other throughput tables are in Benchmarks.

Measured with loki-bench --jitter=2h --warmup=5s --duration=30s on an Apple M5 Pro (18 cores, 64 GB RAM), ~8 M log entries across 15 services, 7-day window.

Workload	Concurrency	Loki	Proxy	VL native	Proxy / Loki
Small (metadata)	c=10	919 req/s	2,785 req/s	3,756 req/s	3.0×
Small (metadata)	c=50	726 req/s	2,099 req/s	3,793 req/s	2.9×
Small (metadata)	c=100	578 req/s	1,822 req/s	3,538 req/s	3.2×
Heavy (pipelines)	c=10	146 req/s	483 req/s	2,532 req/s	3.3×
Heavy (pipelines)	c=50	247 req/s	582 req/s	2,000 req/s	2.4×
Heavy (pipelines)	c=100	279 req/s	568 req/s	1,952 req/s	2.0×
Compute (rate, topk)	c=10	625 req/s	629 req/s	3,947 req/s	parity
Compute (rate, topk)	c=50	343 req/s†	539 req/s	4,338 req/s	1.6×
Compute (rate, topk)	c=100	410 req/s†	468 req/s	4,473 req/s	1.1×

• Small metadata: 2.9–3.2× faster — VL label/series scans outperform Loki's chunk store, amplified by proxy cache.
• Heavy pipelines: 2.0–3.3× faster — stats_query_range fast path + response caching.
• Compute: parity at c=10; proxy dominates under pressure because Loki saturates.

† Loki compute c=50: 90.55% error rate (saturated). c=100: 97.49% error rate. Proxy: 0% errors at all concurrency levels.

Dashboard load spikes — request coalescer

When many panels hit the same query simultaneously, the proxy collapses them into a single backend call. First-hit coalescing avoids the N-fan-out but pays one backend round-trip; subsequent hits are served from cache.

Full throughput tables, P90/P99 latency, CPU and RSS breakdowns: Benchmarks · Performance

---

The Cost Case

This is a production deployment, not a synthetic benchmark. The numbers below come from a real VictoriaLogs installation running at 310 GiB/day raw ingest with 800 M total log entries and 7.1 days of retention.

What VictoriaLogs actually consumed at that load:

Component	Cores	Memory
vlstorage	1.0	5.0 GiB
vlinsert	0.1	0.6 GiB
vlselect	0.1	0.25 GiB
loki-vl-proxy	~0.1–0.2	~0.15–0.26 GiB
VL + loki-vl-proxy, combined	~1.4	~6.1 GiB

For comparison, Loki's own documentation puts the minimum hardware requirement at 38 cores and 59 GiB for the same ingest class (<3 TB/day). That's the floor — a minimal, single-tenant, non-HA deployment.

Caveat: vlselect was measured at zero read concurrency — query load will add to that number. If you run heavy aggregation queries at scale, benchmark your own workload with loki-bench before sizing.

Storage: 2,201 GiB of raw logs (310 GiB/day × 7.1 days) compressed to 40.5 GiB on disk — 54.9× compression. TrueFoundry ran an independent migration and reported ~40% less storage versus their Loki deployment at the same retention.

Replication: Loki's recommended production setup uses RF=3 — tripling write load, disk, and cross-AZ egress. VictoriaLogs is designed for AZ-local deployment with no mandatory replication. If you're paying for cross-AZ data transfer today, that alone can outweigh compute savings.

Migration cost: Zero changes to Grafana, dashboards, alerts, or any Loki API client. The proxy handles translation transparently; remove it and point back at Loki if needed.

Full cost worksheet, scaling projections, and EC2/GCP sizing tables: Cost Model · Scaling

---

Supported versions

VictoriaLogs	Version
Supported	v1.49.0 (the latest v1.4x release) and every v1.5x release
Tested in CI	v1.49.0 (the latest v1.4x release) and v1.53.0 (the newest v1.5x release)
Pull-request CI pin	v1.53.0
Startup floor (-backend-min-version)	v1.49.0

The proxy refuses to start against a VictoriaLogs older than the floor unless -backend-allow-unsupported-version=true is set; such a version is unsupported and untested. CI tests two versions, the latest v1.4x release and the newest v1.5x release; a new v1.5x release replaces the newest. The weekly matrix run includes the range-metric table compared with Loki. Loki, Grafana, Logs Drilldown and the Loki datasource plugin versions are listed in the compatibility matrix; the VictoriaLogs policy and capability bands are in VictoriaLogs compatibility.

---

Quick Start

Docker

docker run -p 3100:3100 \
  ghcr.io/reliablyobserve/loki-vl-proxy:2.25.0 \
  -backend=http://victorialogs:9428

Release images are signed; see Verifying releases to check the signature and provenance before running one. Pin an explicit version tag, as above, rather than latest.

Helm

helm install loki-vl-proxy oci://ghcr.io/reliablyobserve/charts/loki-vl-proxy \
  --version <release> \
  --set extraArgs.backend=http://victorialogs:9428 \
  --set extraArgs.patterns-enabled=true

Grafana Datasource

Point your existing Loki datasource at the proxy — no other changes needed.

datasources:
  - name: Loki (via VL proxy)
    type: loki
    access: proxy
    url: http://loki-vl-proxy:3100
    jsonData:
      httpHeaderName1: X-Scope-OrgID
    secureJsonData:
      httpHeaderValue1: team-alpha

That's it. Grafana Explore, Drilldown, and all dashboards work immediately.

For StatefulSet persistence, peer-cache fleet setup, OTLP push wiring, and image source options, see Getting Started and Operations.

Peer fleet discovery — four modes, all refresh every 15 s and rebuild the hash ring live:

Mode	Flag	Best for
dns	-peer-dns=proxy-headless.ns.svc.cluster.local	Kubernetes headless service — only ready pods appear
srv	-peer-srv=_loki-vl-proxy._tcp.proxy-headless.ns.svc.cluster.local	Kubernetes StatefulSet, Consul DNS — port embedded in record
http	-peer-http-url=http://consul:8500/v1/catalog/service/loki-vl-proxy	Outside k8s: Consul, Nomad, Prometheus HTTP SD, or custom endpoint
static	-peer-static=10.0.0.1:3100,10.0.0.2:3100	Fixed fleets, development

Verify the live ring at any time: curl -H 'X-Peer-Token: <peer-auth-token>' http://proxy:3100/_cache/peers → {"peers":[...],"self":"...","count":N} (peer endpoints require the shared token unless -peer-insecure-ip-allowlist=true).

Non-Kubernetes examples (static, Consul, Prometheus SD, CoreDNS) are in examples/peers/.

---

Why It's Fast

4-tier cache:

• Tier0 — compatibility-edge cache for safe GET responses (no backend hit at all)
• L1 — in-memory hot path
• L2 — disk (bbolt), survives restarts, warms historical windows across large working sets
• L3 — peer cache, lets warm fleet replicas share results instead of all hitting the backend. Four peer discovery modes: dns (k8s headless A-records), srv (DNS SRV with embedded port, works with Consul DNS and k8s StatefulSets), http (polls any JSON endpoint — Consul catalog, Prometheus HTTP SD, custom registry), static (fixed list). Discovery refreshes every 15 s; the hash ring updates atomically so peer add/remove is live without restarts.

Window reuse. Long query_range requests are split into 1h windows. Historical windows are served from cache; only the live edge fetches from VictoriaLogs. A 7-day query with warm cache may hit the backend for a single window.

Adaptive parallelism. Parallel window fetches use EWMA-based backpressure — ramps up when VictoriaLogs is fast, backs off automatically before it becomes a problem.

Request coalescing. Concurrent identical queries collapse into one upstream request.

Lock-free hot paths. Circuit breaker, metrics histograms, and rate limiter use atomic operations instead of mutexes. Structured logging uses an async buffered handler. At 100+ concurrent requests, these eliminate the contention that dominates proxy-added latency.

---

What Works Out of the Box

• Grafana Explore — log browsing, filtering, live tail
• Grafana Logs Drilldown — patterns, service view, field breakdown
• Dashboards — supported Loki log and metric queries
• Multi-tenant — X-Scope-OrgID isolation, including Loki-style a|b multi-tenant reads
• Live tail — native WebSocket tail or synthetic polling fallback
• Rules and alerts — read bridge to vmalert (no write lifecycle)
• LogQL — stream selectors, filters, parsers, metric queries, range functions and vector operators, with measured compatibility and documented limitations
• OTel labels — dotted structured metadata exposed correctly in detected fields, underscore-safe in stream labels

---

Production Features

• Circuit breaker — opens on backend failure, closes automatically on recovery; lock-free fast path in healthy state
• Per-client rate limits — token bucket per client source IP plus a global in-flight cap (-max-concurrent, excess gets 503)
• Adaptive log sampling — below 10 req/s logs everything; above it, OK traffic becomes periodic summaries while errors are always logged
• Tenant isolation — strict X-Scope-OrgID fanout guardrails; no cross-tenant data bleed
• TLS / mTLS — configurable on both northbound (client) and southbound (backend) boundaries
• OTLP metrics push — proxy pushes its own metrics to any OTLP HTTP endpoint
• Operator dashboard — packaged Grafana dashboard covering Client → Proxy → VictoriaLogs, cache behavior, fanout, and resource utilization
• Runbook-backed alerts — 15 alert rules, each with a linked runbook
• 100+ Prometheus metrics — all under loki_vl_proxy_* prefix
• Read-only by default — /push returns 405; /loki/api/v1/delete answers 404 like Loki unless -delete-api-enabled is set, and is not functional against current VictoriaLogs either way (details); debug/admin disabled unless explicitly enabled
• Cold storage routing — time-boundary split to Victoria Lakehouse for long-range queries
• Query-length enforcement — query_range time-range limit: global default via -default-max-query-length, per-tenant override via max_query_length in -tenant-limits / -tenant-default-limits

---

How It Works

flowchart LR
    G["Grafana / API tools<br/><sub>unchanged Loki HTTP datasource</sub>"]

    subgraph Proxy["Loki-VL-proxy &nbsp;·&nbsp; one ~14 MB Go binary"]
        direction TB
        API["Loki API surface<br/><sub>query · labels · series · tail<br/>detected_fields · patterns · volume</sub>"]
        SMART["<b>Smart layer</b> — the strange stuff that makes this work<br/><sub>• disk-backed label cache + keep-warm loop (75% of labels TTL)<br/>• full-range label metadata on first request (1 h → 7 d)<br/>• time-bucketed cache keys (collapse refresh drift)<br/>• fleet jitter + peer-first warmup (≤8 VL hits on 30-pod restart)<br/>• circuit breaker · rate limits · per-tenant isolation</sub>"]
        XLATE["LogQL → LogsQL translator<br/><sub>typed AST, version-gated, hot-reload labels</sub>"]
    end

    VL[("VictoriaLogs")]
    VMA[/"vmalert · alerts/rules<br/>(optional bridge)"/]

    G -->|"Loki API, zero changes"| API
    API --> SMART --> XLATE --> VL
    SMART -. "rules surface" .-> VMA

    classDef client fill:#1f2937,stroke:#93c5fd,color:#f3f4f6,stroke-width:2px;
    classDef proxy fill:#172554,stroke:#38bdf8,color:#f0f9ff,stroke-width:2px;
    classDef smart fill:#3f1d2e,stroke:#f472b6,color:#fdf2f8,stroke-width:2px;
    classDef backend fill:#052e16,stroke:#34d399,color:#ecfdf5,stroke-width:2px;
    classDef opt fill:#1c1917,stroke:#a8a29e,color:#f5f5f4,stroke-width:1px,stroke-dasharray:4 4;

    class G client;
    class API,XLATE proxy;
    class SMART smart;
    class VL backend;
    class VMA opt;

Loading

What this picture says: Grafana keeps talking Loki. The proxy translates LogQL to LogsQL on the way down, and the Smart layer is where the work happens — every entry in that pink box is what lets a 14 MB binary serve cold Drilldown queries that the official Loki layout can't run at all on 12 GiB RAM. The full per-box exploded view (10 subsystems, every handler, every cache tier, every backend) lives in docs/architecture.md for operators and contributors.

---

Compatibility

Loki-VL-proxy is validated continuously in CI against four compatibility contracts: Loki API, Grafana Logs Drilldown, Grafana Loki datasource, and VictoriaLogs integration.

Label and Field Compatibility

Profile	Stream labels (/labels)	Detected fields / metadata	Best for
Loki-compatible (default)	underscore-only	translated underscore aliases	strict Loki UX, Grafana Explore/Drilldown
Mixed	underscore-only	dotted + translated aliases	Grafana + OTel correlation
Native-field	underscore-only (label-style=underscores)	dotted-native only	VL/OTel-native field workflows

Default flags: -label-style=underscores, -metadata-field-mode=translated. Grafana query builder works best with underscore aliases; code mode accepts dotted expressions and translates them to VL-native field matching.

Tuple safety: Default responses return strict [timestamp, line] 2-tuples. 3-tuple metadata mode activates only when the client sends X-Loki-Response-Encoding-Flags: categorize-labels. Cache keys are segregated by tuple mode.

LogQL Compatibility

CI compares supported LogQL operations with a real Loki oracle. Strict populated-fixture regressions check labels, values, timestamps and errors in addition to the broader syntax/status suite. Correcting the exhaustive helper's timestamp units exposed gaps that its earlier scores missed; passing status or nonempty results does not establish complete LogQL parity. See the measured findings and remaining limits.

Typed LogQL parser: The recursive-descent parser (internal/logql) produces a structured AST for validation, routing, and drop/keep extraction. It validates supported semantic constraints, including missing unwrap stages, invalid IP filter arguments and unclosed template delimiters. Grafana receives Loki-style error responses; the compatibility register records remaining syntax and runtime differences.

For full detail: Loki Compatibility, Translation Reference, LogQL Parser, Known Issues

---

Observability

• 100+ Prometheus metrics under loki_vl_proxy_* — cache hit ratios, window fetch latency, fanout behavior, per-tenant and per-client pressure, circuit breaker state
• Packaged operator dashboard — rows for Client-Side Loki API Visibility, Proxy Internal, and Backend-Side VictoriaLogs fanout; fast incident attribution
• 15 runbook-backed alert rules — backend latency, backend unreachable, circuit breaker open, high error rate, rate limiting, tenant error rate, tuple contract, system resources, and more
• Structured JSON logs — route-aware, semconv-aligned, with user-pattern attribution from trusted Grafana headers
• OTLP metrics push — the same proxy metrics pushed to any OTLP HTTP endpoint (-otlp-endpoint)

See Observability and Alert Runbooks Index.

---

Security

• Read-only API surface by default: /push blocked, debug/admin disabled; /loki/api/v1/delete answers 404 like Loki unless -delete-api-enabled is set, and is not functional against current VictoriaLogs (details)
• Non-root runtime image, read-only root filesystem, restricted Helm security contexts
• Hardening headers on all HTTP responses including 404s and disabled routes
• CI security gates: gitleaks, gosec, Trivy, actionlint, hadolint, OpenSSF Scorecard, OWASP ZAP, curated Nuclei
• Proxy-specific coverage: tenant isolation, cache boundary enforcement, browser-origin checks on /tail, forwarded auth handling

See Security and Security Policy.

---

UI Gallery

VictoriaLogs backend with Loki-VL-proxy as the Loki-compatible query layer.

---

Documentation Map

Core

• Getting Started
• Configuration
• Operations
• Architecture
• API Reference
• Security
• Observability
• Performance
• Scaling

Compatibility

• Compatibility Matrix
• Loki Compatibility
• Logs Drilldown Compatibility
• Grafana Loki Datasource Compatibility
• VictoriaLogs Compatibility
• Translation Modes Guide
• Translation Reference

Cache and Runtime Design

• Fleet Cache
• Peer Cache Design
• Benchmarks

Runbooks

• Alert Runbooks Index
• Deployment Best Practices
• Backend High Latency
• Backend Unreachable
• Circuit Breaker Open
• Client Bad Request Burst
• Proxy Down
• Grafana Tuple Contract
• High Error Rate
• High Latency
• Rate Limiting
• Operational Resources
• Tenant High Error Rate

Testing and Release

• Testing
• Release Info

Migration and Project Status

• Rules And Alerts Migration
• Known Issues
• Roadmap
• Changelog

---

License

Apache License 2.0. See LICENSE.