Skip to content

Dynamic filters (SQLite & MySQL support, sqlc.slice() integration, dialect-aware SQL lexing) - #12

Merged
vtuanjs merged 10 commits into
vtuanjs:mainfrom
orlandoc01:main
Aug 11, 2026
Merged

vtuanjs merged 10 commits into
vtuanjs:mainfrom
orlandoc01:main

Conversation

@orlandoc01

@orlandoc01 orlandoc01 commented Aug 9, 2026 •

Copy link
Copy Markdown

Hi @vtuanjs, first off thanks for this plugin. The -- :if dynamic-filter idea let me delete a whole layer of hand-rolled, ugly-looking query-builder code from some side projects I use sqlc in and I've been happily building on it since.

However, when I first tried to use this plugin it failed outright for me. My projects use SQLite and it seems the dynamic filters only worked on PostgreSQL (the runtime only understood $N placeholders, and the database/sql template emitted the Build(...) call and the query call glued onto one line, which doesn't compile). So I forked it and added sqlite support and some other things I thought might be helpful for maintaining support for SQLite, Postgres and mySQL. This PR contributes everything back in case you'd like it upstream.

What's included

  • SQLite support for emit_dynamic_filter — numbered ?N input placeholders normalized to $N output (valid positional syntax for the common SQLite drivers), [bracket] identifiers lexed correctly, and a Docker-free e2e test against a real SQLite database wired into CI.
  • MySQL support — bare ? placeholders numbered by appearance and emitted back as ?, selected by engine: mysql alone (sql_driver still honored if used), with backtick identifiers and backslash-escaped strings lexed correctly, plus an e2e test against a real MySQL 8 container.
  • sqlc.slice() inside dynamic queries (all three engines) — /*SLICE:name*/ markers are numbered at generation time and expanded at Build time; repeated slices reuse their placeholders on numbered-placeholder engines; nil/empty slices render NULL exactly like sqlc's non-dynamic expansion, and a :if-gated nil slice skips the clause entirely (empty keeps it — see the update below).
  • Dialect-aware lexing — Previously '?1' inside a string literal could be rewritten or $2 inside a comment was counted as an argument. Now, annotations and bind markers are only recognized in SQL code context. String literals (including Postgres $$…$$ / $tag$…$tag$ and E'…', MySQL backslash escapes), quoted identifiers, comments, and multi-line strings with them are all ignored.
  • Per-engine example layout — the example module now treats PostgreSQL like the other engines instead of as the implicit default. Sources live in example/postgres/ (like example/mysql/, example/sqlite/), generated code in dbpostgres (like dbmysql, dbsqlite), and e2e suites in e2e-{postgres,mysql,sqlite} with matching make example-e2e-<engine> targets (make example-e2e runs all three).
  • Hardening and performance — fixes for panics and invalid SQL on slice edge cases (repeated empty slices, marker text without a placeholder), fewer per-call heap allocations in Build (21-param benchmark: 36 → 15 allocs/op, ~13% faster), a CI check that committed generated code matches the templates (sqlc generate + git diff --exit-code), and README doc updates for all of the above.

Changes
The only behavior change I made from your original code was regarding empty slices. An empty slice on a :if condition now skips the clause the same way nil does, so an empty list no longer silently matches zero rows. This is called out in the README and the one PostgreSQL e2e case that asserted the old behavior was updated to match.

UPDATE (2026-08-10): Reverted per the discussion below in this PR. Nil and empty are distinct again, as in your original code: only a nil slice skips a :if clause, while an empty non-nil slice keeps it and renders IN (NULL). For call sites where empty should mean "don't filter", the generated dynfilter.go now includes a NilableSlice helper that converts empty slices to nil. Tests and README have been updated to match, and this PR no longer changes any behavior of the original code.

No pressure either way: if you'd rather not take on SQLite/MySQL surface area in this repo, I completely understand, I'm happy to maintain this as a standalone fork instead (primarily for the SQLite support I need). I mostly wanted to offer the improvements back, and to say thanks for the plugin regardless, I really liked the idea!

AI Disclosure
I developed this with substantial help from coding agents (you'll see the co-author trailers on most commits). I directed, reviewed, and tested the work, but I want to be upfront about it so you can calibrate your review accordingly.

orlandoc01 and others added 9 commits August 4, 2026 23:59
* fix: support SQLite dynamic filters

* test: strengthen SQLite dynamic filter coverage
* fix: support dynamic sqlc slices

* test: cover PostgreSQL dynamic slices

* fix: handle dynamic sqlc slices across drivers

* test: name dynamic slice cases by driver

* test: add MySQL dynamic filter fixture
The generated dynamic-filter runtime scanned raw text, so bind markers and `-- :if` annotations inside string literals, quoted identifiers, and comments were wrongly rewritten or counted:
  - '?1' was rewritten to '$1', changing returned data
  - markers in `-- ?2` or `/* $2 */` were counted as arguments
  - a string containing `-- :if $1` truncated the line into malformed SQL

Add a small SQL lexer to dynfilterCode.tmpl (dynSkipToken / dynAnnotationStart) that skips single-quoted strings, double-quoted and backtick identifiers, SQLite bracket identifiers, and line/block comments, so only markers and annotations in actual SQL code are interpreted. Bracket handling is gated to SQLite via a new dynBracketIdentifiers const so PostgreSQL array subscripts (arr[$1]) keep working.

Regenerate example db/dbsqlite/dbmysql output and add runtime tests for '?1', '$2', `-- ?2`, `/* $2 */`, 'it''s ?1', quoted identifiers, and a literal containing `-- :if $1`, asserting both final SQL and arguments
- Key MySQL question-mark placeholders off the engine; sql_driver is no
  longer required for engine: mysql (still honored when set).
- Treat empty slices as inactive for :if conditions, same as nil — an
  empty allow-list no longer silently matches zero rows.
- Fix repeated empty sqlc.slice expansion emitting invalid "IN ()" and
  the panic when a scalar placeholder shared a NULL-rendered slice's
  param; bounds-check the slice branch; fall back to scalar binding
  when a /*SLICE:*/-shaped comment precedes a non-slice arg.
- Lex per dialect: MySQL backslash-escaped strings, PostgreSQL
  dollar-quoted strings, E'' escape strings and nested block comments;
  carry lexer state across lines so multi-line string literals can
  never be truncated by annotation-shaped content.
- Recognize numbered ?N input markers only on SQLite/MySQL, restoring
  the PostgreSQL invariant that '?' is always operator text.
- Reduce Build() allocations by splitting scalar/slice reuse maps
  (21-param benchmark: 36 -> 15 allocs/op); cache reflect Len(); build
  the slice-numbering name map once; drop a dead SQLDriver reassignment.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add the engine support matrix (PostgreSQL/SQLite/MySQL placeholder
handling), sqlc.slice() behavior in dynamic queries, empty-slice :if
semantics, and the lexical-context guarantees; fix the annotation
example (text after :if is not parsed) and refresh test counts.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds a mysql:8 service to the e2e docker-compose and an e2e-mysql
package exercising the generated dynamic-filter query (nil, populated,
and empty sqlc.slice). make example-e2e now runs both postgres and
mysql suites with -count=1, which also surfaced a stale postgres case:
EmptySlice_NoMatch asserted pre-3984af3 semantics and only kept passing
via the test cache; updated to the documented clause-skip behavior.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PostgreSQL was the implicit default: generated code in db/, sources at
schema.sql + queries/, e2e in e2e/. Align it with the other engines:
dbpostgres/, postgres/{schema.sql,queries/}, e2e-postgres/, plus
per-engine example-e2e-{postgres,mysql,sqlite} make targets with
example-e2e running all three (CI now invokes just that).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@vtuanjs

vtuanjs commented Aug 10, 2026

Copy link
Copy Markdown
Owner

Hi @orlandoc01
Thanks for the contribution! Overall, it looks good to me.

One thing about slices: whether we should skip an empty slice should depend on the use case. I don't think we should automatically skip it just because it's empty. sqlc can handle the data/empty/null cases for us.

In the use case layer, we should have a nilableSlice or nilableArray function to handle this explicitly.

…lice (#4)

Per upstream feedback on vtuanjs#12: an empty non-nil slice on
a :if condition now keeps the clause and renders IN (NULL) (matches zero
rows, consistent with sqlc's non-dynamic sqlc.slice() expansion), instead
of silently skipping the filter. Only a nil slice skips. This is the
fail-closed default for computed lists such as permission scopes.

Callers who want empty to mean "don't filter" opt in explicitly via the
new generated NilableSlice helper, which converts empty slices to nil.

Unit and e2e tests updated across all three engines; README documents
both idioms.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
@orlandoc01

Copy link
Copy Markdown
Author

@vtuanjs I think you make a good point that we should treat nil different from an empty slice, and adding a NilableSlice function by default to get the behavior of skipping a clause when the slice is empty is also a good suggestion. I've revised the code as such and amended the top level PR body message to reflect that. Feel free to use if it helps and let me know if you have any more feedback to share!

@vtuanjs

vtuanjs commented Aug 11, 2026

Copy link
Copy Markdown
Owner

LGTM, thank @orlandoc01
Let me test it a bit more, and after that, I’ll release it.

@vtuanjs
vtuanjs merged commit e95083f into vtuanjs:main Aug 11, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants