Dynamic filters (SQLite & MySQL support, sqlc.slice() integration, dialect-aware SQL lexing) - #12
Conversation
* fix: support SQLite dynamic filters * test: strengthen SQLite dynamic filter coverage
* fix: support dynamic sqlc slices * test: cover PostgreSQL dynamic slices * fix: handle dynamic sqlc slices across drivers * test: name dynamic slice cases by driver * test: add MySQL dynamic filter fixture
The generated dynamic-filter runtime scanned raw text, so bind markers and `-- :if` annotations inside string literals, quoted identifiers, and comments were wrongly rewritten or counted: - '?1' was rewritten to '$1', changing returned data - markers in `-- ?2` or `/* $2 */` were counted as arguments - a string containing `-- :if $1` truncated the line into malformed SQL Add a small SQL lexer to dynfilterCode.tmpl (dynSkipToken / dynAnnotationStart) that skips single-quoted strings, double-quoted and backtick identifiers, SQLite bracket identifiers, and line/block comments, so only markers and annotations in actual SQL code are interpreted. Bracket handling is gated to SQLite via a new dynBracketIdentifiers const so PostgreSQL array subscripts (arr[$1]) keep working. Regenerate example db/dbsqlite/dbmysql output and add runtime tests for '?1', '$2', `-- ?2`, `/* $2 */`, 'it''s ?1', quoted identifiers, and a literal containing `-- :if $1`, asserting both final SQL and arguments
- Key MySQL question-mark placeholders off the engine; sql_driver is no longer required for engine: mysql (still honored when set). - Treat empty slices as inactive for :if conditions, same as nil — an empty allow-list no longer silently matches zero rows. - Fix repeated empty sqlc.slice expansion emitting invalid "IN ()" and the panic when a scalar placeholder shared a NULL-rendered slice's param; bounds-check the slice branch; fall back to scalar binding when a /*SLICE:*/-shaped comment precedes a non-slice arg. - Lex per dialect: MySQL backslash-escaped strings, PostgreSQL dollar-quoted strings, E'' escape strings and nested block comments; carry lexer state across lines so multi-line string literals can never be truncated by annotation-shaped content. - Recognize numbered ?N input markers only on SQLite/MySQL, restoring the PostgreSQL invariant that '?' is always operator text. - Reduce Build() allocations by splitting scalar/slice reuse maps (21-param benchmark: 36 -> 15 allocs/op); cache reflect Len(); build the slice-numbering name map once; drop a dead SQLDriver reassignment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add the engine support matrix (PostgreSQL/SQLite/MySQL placeholder handling), sqlc.slice() behavior in dynamic queries, empty-slice :if semantics, and the lexical-context guarantees; fix the annotation example (text after :if is not parsed) and refresh test counts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds a mysql:8 service to the e2e docker-compose and an e2e-mysql package exercising the generated dynamic-filter query (nil, populated, and empty sqlc.slice). make example-e2e now runs both postgres and mysql suites with -count=1, which also surfaced a stale postgres case: EmptySlice_NoMatch asserted pre-3984af3 semantics and only kept passing via the test cache; updated to the documented clause-skip behavior. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PostgreSQL was the implicit default: generated code in db/, sources at
schema.sql + queries/, e2e in e2e/. Align it with the other engines:
dbpostgres/, postgres/{schema.sql,queries/}, e2e-postgres/, plus
per-engine example-e2e-{postgres,mysql,sqlite} make targets with
example-e2e running all three (CI now invokes just that).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Hi @orlandoc01 One thing about slices: whether we should skip an empty slice should depend on the use case. I don't think we should automatically skip it just because it's empty. sqlc can handle the data/empty/null cases for us. In the use case layer, we should have a |
…lice (#4) Per upstream feedback on vtuanjs#12: an empty non-nil slice on a :if condition now keeps the clause and renders IN (NULL) (matches zero rows, consistent with sqlc's non-dynamic sqlc.slice() expansion), instead of silently skipping the filter. Only a nil slice skips. This is the fail-closed default for computed lists such as permission scopes. Callers who want empty to mean "don't filter" opt in explicitly via the new generated NilableSlice helper, which converts empty slices to nil. Unit and e2e tests updated across all three engines; README documents both idioms. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
@vtuanjs I think you make a good point that we should treat nil different from an empty slice, and adding a |
|
LGTM, thank @orlandoc01 |
Hi @vtuanjs, first off thanks for this plugin. The
-- :ifdynamic-filter idea let me delete a whole layer of hand-rolled, ugly-looking query-builder code from some side projects I use sqlc in and I've been happily building on it since.However, when I first tried to use this plugin it failed outright for me. My projects use SQLite and it seems the dynamic filters only worked on PostgreSQL (the runtime only understood
$Nplaceholders, and thedatabase/sqltemplate emitted theBuild(...)call and the query call glued onto one line, which doesn't compile). So I forked it and added sqlite support and some other things I thought might be helpful for maintaining support for SQLite, Postgres and mySQL. This PR contributes everything back in case you'd like it upstream.What's included
emit_dynamic_filter— numbered?Ninput placeholders normalized to$Noutput (valid positional syntax for the common SQLite drivers),[bracket]identifiers lexed correctly, and a Docker-free e2e test against a real SQLite database wired into CI.?placeholders numbered by appearance and emitted back as?, selected byengine: mysqlalone (sql_driverstill honored if used), with backtick identifiers and backslash-escaped strings lexed correctly, plus an e2e test against a real MySQL 8 container.sqlc.slice()inside dynamic queries (all three engines) —/*SLICE:name*/markers are numbered at generation time and expanded atBuildtime; repeated slices reuse their placeholders on numbered-placeholder engines; nil/empty slices renderNULLexactly like sqlc's non-dynamic expansion, and a:if-gated nil slice skips the clause entirely (empty keeps it — see the update below).'?1'inside a string literal could be rewritten or$2inside a comment was counted as an argument. Now, annotations and bind markers are only recognized in SQL code context. String literals (including Postgres$$…$$/$tag$…$tag$andE'…', MySQL backslash escapes), quoted identifiers, comments, and multi-line strings with them are all ignored.example/postgres/(likeexample/mysql/,example/sqlite/), generated code indbpostgres(likedbmysql,dbsqlite), and e2e suites ine2e-{postgres,mysql,sqlite}with matchingmake example-e2e-<engine>targets (make example-e2eruns all three).Build(21-param benchmark: 36 → 15 allocs/op, ~13% faster), a CI check that committed generated code matches the templates (sqlc generate+git diff --exit-code), and README doc updates for all of the above.Changes
The only behavior change I made from your original code was regarding empty slices. An empty slice on a:ifcondition now skips the clause the same way nil does, so an empty list no longer silently matches zero rows. This is called out in the README and the one PostgreSQL e2e case that asserted the old behavior was updated to match.UPDATE (2026-08-10): Reverted per the discussion below in this PR. Nil and empty are distinct again, as in your original code: only a nil slice skips a
:ifclause, while an empty non-nil slice keeps it and rendersIN (NULL). For call sites where empty should mean "don't filter", the generateddynfilter.gonow includes aNilableSlicehelper that converts empty slices to nil. Tests and README have been updated to match, and this PR no longer changes any behavior of the original code.No pressure either way: if you'd rather not take on SQLite/MySQL surface area in this repo, I completely understand, I'm happy to maintain this as a standalone fork instead (primarily for the SQLite support I need). I mostly wanted to offer the improvements back, and to say thanks for the plugin regardless, I really liked the idea!
AI Disclosure
I developed this with substantial help from coding agents (you'll see the co-author trailers on most commits). I directed, reviewed, and tested the work, but I want to be upfront about it so you can calibrate your review accordingly.