fix: line filters whose text is not UTF-8 answer like Loki

CHANGELOG.md

@@ -7,6 +7,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Fixed
+
+- A `|=` or `!=` line filter whose text is not UTF-8 (`|= "\xff\xfe"`)
+  answers like Loki instead of 400: `|=` keeps no line and `!=` every line,
+  since no stored line holds such bytes, and an `or` chain keeps its valid
+  alternatives. Error texts that quote such bytes (`|~ "\xff"`) write each one
+  as U+FFFD, as Loki does.
+
 ## [2.108.0] - 2026-11-09
 
 ### Fixed

conformance/registry/behaviours/architecture-syntax-package-layering.yaml

@@ -340,3 +340,20 @@ decisions:
     performance:
       tracked: ['BenchmarkUnwrapGateReadsBefore / BenchmarkUnwrapGateReadsAfter ns/op, B/op, allocs/op (internal/proxy)']
       bound: 'The unwrap-gate reads of one gated range request (2026-11-09, M5 Pro, GOMAXPROCS=4, count=6): 8.27 µs, 9.0 KiB, 30 allocs before; 4.36 µs, 5.1 KiB, 15 allocs after'
+  - id: line-filter-text-not-utf8
+    chosen: 'A |= or != line filter whose text is not UTF-8 is answered from the fact that no stored line holds such bytes: translate.TextLineFilter writes `|=` as match-none (`_stream_id:in()`), `!=` as no filter, and an `or` chain as the regexp of its valid alternatives; the translator and the lowering share it, and valid texts keep their exact LogsQL. Loki''s text error body is written as valid UTF-8, one U+FFFD per invalid byte, as Loki writes it'
+    over: ['writing each invalid byte as \x{FFFD} in the VictoriaLogs regexp (a superset that matches lines holding U+FFFD)', 'evaluating the filter on rows in the proxy', 'strings.ToValidUTF8 (one U+FFFD per run, not per byte as Loki)']
+    why: |
+      VictoriaLogs refuses to compile a regexp with bytes that are not UTF-8,
+      so the proxy answered 400 where Loki answers 200. Every ingestion API
+      (Loki push, OTLP, VictoriaLogs JSON lines) carries UTF-8 text, so Loki's
+      byte match of such a text never matches a stored line: the exact answer
+      needs no VictoriaLogs filter at all. Loki's 400 text for `|~ "\xff"`
+      quotes the regexp with U+FFFD per byte (stack, Loki 3.7, 2026-11-09).
+    evidence: 'TestCompat_LineFilterInvalidUTF8LikeLoki against Loki on the compose stack (|= 200 no line, != 200 every line, or chain 200 with 2 lines, |~ 400 with Loki''s bytes, metric instant equal); TestPlanShadow native 624 of 22,842 rows × profiles, 0 mismatches; TestVLOracle on the 28,663-row corpus, v1.49.0 and v1.53.0, with the invalid-UTF-8 skip removed; TestVLCapability_EmptyInMatchesNothing (the match-no-rows filter keeps no row, log and stats) on VictoriaLogs v1.49.0 and v1.53.0 (2026-11-09)'
+    code: [internal/translate/linefilter.go, internal/translator/translator.go, internal/queryplan/lower/lower.go, internal/proxy/http_utils.go]
+    tests: [TestCompat_LineFilterInvalidUTF8LikeLoki, TestVLCapability_EmptyInMatchesNothing, TestTranslateLineFilterInvalidUTF8, TestInvalidUTF8RegexpErrorTextLikeLoki, TestValidUTF8Text, FuzzErrorBodyShapes, TestPlanShadow, TestVLOracle]
+    quality: 'semantics/line-filter-invalid-utf8 fixed: the log query, the metric query and the error text answer like Loki.'
+    performance:
+      tracked: ['BenchmarkTranslateLogQL_LineFilter / BenchmarkTranslateLogQL_LineFilterInvalidUTF8 (internal/translator), BenchmarkWriteLokiErrorBody (internal/proxy) ns/op, B/op, allocs/op']
+      bound: 'M5 Pro, GOMAXPROCS=4, count=6, 2026-11-09: valid line filters unchanged (7.48 µs, 2.6 KiB, 35 allocs; base 7.57 µs, 35 allocs); invalid-UTF-8 filters 6.63 µs, 36 allocs (base 6.73 µs, 39 allocs, then rejected by VictoriaLogs); valid error body unchanged (244 ns, 10 allocs), an invalid-UTF-8 error body 454 ns, 11 allocs (base 242 ns, 10)'

conformance/registry/cases/semantics/line-filter-invalid-utf8.yaml

@@ -0,0 +1,53 @@
+id: semantics/line-filter-invalid-utf8
+title: A |= or != line filter whose text is not UTF-8 answers like Loki (fixed)
+pending: true
+proves:
+  - loki_api_v1_query_range
+  - loki_api_v1_query
+consumer: api
+why: |
+  LogQL string escapes can write bytes that are not UTF-8: `|= "\xff\xfe"`.
+  Loki matches a `|=` or `!=` filter as bytes, so it accepts the query and
+  answers it: `|=` keeps the lines that hold those bytes, `!=` the others.
+  The proxy wrote the substring match as a VictoriaLogs regexp of the quoted
+  literal, which VictoriaLogs refuses to compile ("invalid UTF-8"), so the
+  proxy answered 400 "failed to parse the log query" for the log query and
+  the metric query alike (stack, 2026-11-09).
+
+  `|~ "\xff"` is rejected by both with 400, but Loki writes each invalid
+  byte of the quoted regexp as U+FFFD, and the proxy wrote the raw bytes.
+
+  Fixed: a stored line is valid UTF-8 (VictoriaLogs and Loki's push API both
+  keep text), so a text that is not UTF-8 is held by no line. A `|=` filter
+  with such a text matches no line (`_stream_id:in()`), a `!=` filter keeps
+  every line (no filter), and in an `or` chain only the valid alternatives
+  are kept; valid literals keep their exact VictoriaLogs text. Every error
+  text the proxy writes in Loki's text shape is valid UTF-8, one U+FFFD per
+  invalid byte, as Loki's.
+request:
+  method: GET
+  path: /loki/api/v1/query_range
+  params:
+    query: '{app="ipprobe-t"} |= "\xff\xfe"'
+    start: 'now-15m'
+    end: 'now'
+  headers:
+    X-Scope-OrgID: "0"
+compare:
+  mode: observed
+  fields: [status]
+  expect: |
+    200 with no line (no stored line holds the bytes), as Loki answers; `!=`
+    200 with every line; `|~ "\xff"` 400 with Loki's text byte for byte.
+gap:
+  status: fixed
+  impact: edge
+  effort: M
+  area: 'internal/translate linefilter.go (TextAlternatives, NoTextFilter, LineFilter); internal/translator line filters; internal/proxy http_utils.go writeLokiErrorBody'
+  loki: 'line filters |= and != match bytes; error text is valid UTF-8 (U+FFFD per invalid byte)'
+  proxy: 'match-none for |= and no filter for != when the text is not UTF-8; U+FFFD per invalid byte in error text'
+  planned_test: 'TestCompat_LineFilterInvalidUTF8LikeLoki (test/e2e-compat, group core); TestTranslateLineFilterInvalidUTF8; TestInvalidUTF8RegexpErrorTextLikeLoki'
+  fixed_by: '#1236'
+  visual: none
+  visual_reason: 'Grafana cannot show the case: no stored line holds bytes that are not UTF-8, so the panels of Loki and the proxy are the same empty or full set; proven by TestCompat_LineFilterInvalidUTF8LikeLoki against Loki (status, lines, metric values and the 400 text)'
+  clusters: []

conformance/registry/generated/census.json

@@ -500,6 +500,9 @@
     "semantics/last-over-time-line-filter-offset-points | compare:observed",
     "semantics/last-over-time-line-filter-offset-points | gap:open",
     "semantics/last-over-time-line-filter-offset-points | query",
+    "semantics/line-filter-invalid-utf8 | compare:observed",
+    "semantics/line-filter-invalid-utf8 | gap:fixed",
+    "semantics/line-filter-invalid-utf8 | query",
     "semantics/line-filter-or-alternatives | compare:observed",
     "semantics/line-filter-or-alternatives | gap:fixed",
     "semantics/line-filter-or-alternatives | perf",
@@ -1006,6 +1009,7 @@
     "internal/proxy/error_layer_gate_test.go | status-500",
     "internal/proxy/error_layer_gate_test.go | status-502",
     "internal/proxy/error_layer_gate_test.go | status-503",
+    "internal/proxy/error_text_utf8_test.go | semantics/line-filter-invalid-utf8",
     "internal/proxy/extracted_probe_head_test.go | quality/extracted-label-probe-cost",
     "internal/proxy/extracted_probe_head_test.go | semantics/json-extracted-metric-pushdown",
     "internal/proxy/extracted_suffix_collision_test.go | loki-compatible-profile",
@@ -1536,6 +1540,7 @@
     "internal/translator/label_matcher_anchoring_test.go | profiles/stream-selector-regexp-anchoring",
     "internal/translator/label_matcher_regexp_test.go | profiles/label-filter-regexp-anchoring",
     "internal/translator/label_matcher_regexp_test.go | profiles/stream-selector-regexp-anchoring",
+    "internal/translator/line_filter_invalid_utf8_test.go | semantics/line-filter-invalid-utf8",
     "internal/translator/line_filter_or_test.go | semantics/line-filter-or-alternatives",
     "internal/translator/nested_vector_chain_test.go | metric-series-identity",
     "internal/translator/nested_vector_chain_test.go | semantics/nested-aggregation-outer-by-native-routes",
@@ -1746,6 +1751,9 @@
     "test/e2e-compat/line_fields_parity_test.go | loki_api_v1_query_range",
     "test/e2e-compat/line_fields_parity_test.go | profiles/detected-fields-structured-metadata-beside-json-line",
     "test/e2e-compat/line_fields_parity_test.go | profiles/parsed-fields-without-parser",
+    "test/e2e-compat/line_filter_invalid_utf8_test.go | loki_api_v1_query",
+    "test/e2e-compat/line_filter_invalid_utf8_test.go | loki_api_v1_query_range",
+    "test/e2e-compat/line_filter_invalid_utf8_test.go | semantics/line-filter-invalid-utf8",
     "test/e2e-compat/line_format_error_test.go | semantics/line-format-template-error-keeps-entry",
     "test/e2e-compat/logfmt_existence_filter_test.go | loki_api_v1_query_range",
     "test/e2e-compat/logfmt_existence_filter_test.go | parser-logfmt",
@@ -1957,6 +1965,8 @@
     "test/e2e-compat/unwrap_value_forms_test.go | semantics/unwrap-gate-parsefloat-divergences",
     "test/e2e-compat/unwrap_value_forms_test.go | semantics/unwrap-sample-validity",
     "test/e2e-compat/unwrap_value_forms_test.go | status-400",
+    "test/e2e-compat/vl_empty_in_test.go | semantics/line-filter-invalid-utf8",
+    "test/e2e-compat/vl_empty_in_test.go | vl-empty-in-matches-nothing",
     "test/e2e-compat/vl_pattern_lines_test.go | vl-phrase-filter-quoted-names",
     "test/e2e-compat/vl_pattern_lines_test.go | vl-sort-ties-by-text",
     "test/e2e-compat/vl_zstd_response_window_test.go | vl-zstd-response-window",
@@ -1981,14 +1991,14 @@
   "counts": {
     "case_consumers": {
       "any": 28,
-      "api": 43,
+      "api": 44,
       "drilldown": 50,
       "explore": 154,
       "grafana-datasource": 2,
       "operator": 4
     },
     "claiming_tests": {
-      "go": 388,
+      "go": 392,
       "playwright": 1
     },
     "page_kinds": {
@@ -2008,7 +2018,7 @@
     "shape_covers_distinct": 108,
     "shapes_path_only": 8,
     "unwired_items": 100,
-    "wired_items": 265
+    "wired_items": 267
   },
   "gates": [
     "A/B shape selection",

conformance/registry/generated/matrix-link.json

@@ -1,6 +1,6 @@
 {
   "registry": {
-    "cases_defined": 281,
+    "cases_defined": 282,
     "live_cases_blocked": 1,
     "live_cases_gap": 0,
     "live_cases_hold": 0,

conformance/registry/generated/wiring.json

@@ -357,6 +357,7 @@
       "test/e2e-compat/keep_service_name_test.go",
       "test/e2e-compat/label_matcher_regexp_anchoring_test.go",
       "test/e2e-compat/label_replace_new_label_test.go",
+      "test/e2e-compat/line_filter_invalid_utf8_test.go",
       "test/e2e-compat/marker_text_test.go",
       "test/e2e-compat/nested_vector_aggregation_test.go",
       "test/e2e-compat/non_sum_vector_aggregation_range_test.go",
@@ -464,6 +465,7 @@
       "test/e2e-compat/label_matcher_regexp_anchoring_test.go",
       "test/e2e-compat/label_replace_new_label_test.go",
       "test/e2e-compat/line_fields_parity_test.go",
+      "test/e2e-compat/line_filter_invalid_utf8_test.go",
       "test/e2e-compat/logfmt_existence_filter_test.go",
       "test/e2e-compat/logfmt_risk_candidates_test.go",
       "test/e2e-compat/marker_text_test.go",
@@ -1170,6 +1172,12 @@
       "internal/proxy/labels_inclusive_end_test.go",
       "test/e2e-compat/labels_zero_width_test.go"
     ],
+    "semantics/line-filter-invalid-utf8": [
+      "internal/proxy/error_text_utf8_test.go",
+      "internal/translator/line_filter_invalid_utf8_test.go",
+      "test/e2e-compat/line_filter_invalid_utf8_test.go",
+      "test/e2e-compat/vl_empty_in_test.go"
+    ],
     "semantics/line-filter-or-alternatives": [
       "internal/logql/log_range_alternatives_test.go",
       "internal/translator/line_filter_or_test.go",
@@ -1631,6 +1639,9 @@
     "versions/stats-bucket-label-v1.45": [
       "internal/proxy/ordered_json_extracted_pushdown_test.go"
     ],
+    "vl-empty-in-matches-nothing": [
+      "test/e2e-compat/vl_empty_in_test.go"
+    ],
     "vl-extra-stream-filters-exact-and-empty": [
       "test/e2e-compat/tenant_label_isolation_test.go"
     ],
@@ -1672,7 +1683,7 @@
       "test/e2e-compat/unwrap_ungrouped_sum_test.go"
     ]
   },
-  "registry_items": 439,
+  "registry_items": 441,
   "routes_without_registry_item": [],
   "test_to_items": {
     "cmd/proxy/main_test.go": [
@@ -1992,6 +2003,9 @@
       "status-502",
       "status-503"
     ],
+    "internal/proxy/error_text_utf8_test.go": [
+      "semantics/line-filter-invalid-utf8"
+    ],
     "internal/proxy/extracted_probe_head_test.go": [
       "quality/extracted-label-probe-cost",
       "semantics/json-extracted-metric-pushdown"
@@ -2884,6 +2898,9 @@
       "profiles/label-filter-regexp-anchoring",
       "profiles/stream-selector-regexp-anchoring"
     ],
+    "internal/translator/line_filter_invalid_utf8_test.go": [
+      "semantics/line-filter-invalid-utf8"
+    ],
     "internal/translator/line_filter_or_test.go": [
       "semantics/line-filter-or-alternatives"
     ],
@@ -3232,6 +3249,11 @@
       "profiles/detected-fields-structured-metadata-beside-json-line",
       "profiles/parsed-fields-without-parser"
     ],
+    "test/e2e-compat/line_filter_invalid_utf8_test.go": [
+      "loki_api_v1_query",
+      "loki_api_v1_query_range",
+      "semantics/line-filter-invalid-utf8"
+    ],
     "test/e2e-compat/line_format_error_test.go": [
       "semantics/line-format-template-error-keeps-entry"
     ],
@@ -3559,6 +3581,10 @@
       "semantics/unwrap-sample-validity",
       "status-400"
     ],
+    "test/e2e-compat/vl_empty_in_test.go": [
+      "semantics/line-filter-invalid-utf8",
+      "vl-empty-in-matches-nothing"
+    ],
     "test/e2e-compat/vl_pattern_lines_test.go": [
       "vl-phrase-filter-quoted-names",
       "vl-sort-ties-by-text"
@@ -3594,7 +3620,7 @@
       "profiles/structured-metadata-keys-per-profile"
     ]
   },
-  "tests_with_markers": 389,
+  "tests_with_markers": 393,
   "unknown_ids": {},
   "unwired_items": [
     "api_prom_label",
@@ -3698,5 +3724,5 @@
     "versions/capabilities-floor-vs-matrix",
     "versions/rejected-stats-query-wording-vl-1.47"
   ],
-  "wired_items": 265
+  "wired_items": 267
 }