feat(pg_search): allow extension function schema to be configured

[Sanderhoff-alt]

Overview

Fixes #3757.

This PR introduces configuration support for the PostgreSQL pg_search function schema via the HINDSIGHT_API_TEXT_SEARCH_EXTENSION_PG_SEARCH_FUNCTION_SCHEMA environment variable (default: paradedb).

Certain managed PostgreSQL cloud providers (and customized PostgreSQL installations) install pg_search extension functions (score, boolean, match) under custom schemas such as pgsearch rather than the default paradedb schema. This change enables full compatibility with those environments without requiring custom Docker builds or database-level wrapper forwarding functions.

---

Architectural Workflow & Comparison

flowchart TD
    subgraph Config ["1. Configuration & Startup"]
        ENV["ENV: HINDSIGHT_API_TEXT_SEARCH_EXTENSION_PG_SEARCH_FUNCTION_SCHEMA"] --> Val{"PG Identifier Validation"}
        Val -->|"Valid (e.g. pgsearch)"| ConfObj["HindsightConfig.text_search_extension_pg_search_function_schema = 'pgsearch'"]
        Val -->|"Empty / Unset"| DefaultObj["HindsightConfig.text_search_extension_pg_search_function_schema = 'paradedb'"]
        Val -->|"Invalid"| Err["Raise ValueError (Prevent Injection)"]
    end

    subgraph MemoryRecall ["2. Memory Recall BM25 Query Generation"]
        ConfObj --> Dialect1["PostgreSQLDialect.build_bm25_arm"]
        DefaultObj --> Dialect1
        Dialect1 --> SQL1["SELECT ... schema.score(id) AS bm25_score<br/>WHERE id @@@ schema.boolean(should => ARRAY[schema.match('text', $4), ...])"]
    end

    subgraph KnowledgePages ["3. Knowledge Pages BM25 Query Generation"]
        ConfObj --> Dialect2["knowledge_bm25_arm"]
        DefaultObj --> Dialect2
        Dialect2 --> SQL2["SELECT ... schema.score(mm.id) AS score<br/>WHERE mm.id @@@ schema.boolean(should => ARRAY[schema.match('name', $3), ...])"]
    end

Loading

---

Query Generation Comparison

Component	Default (paradedb)	Configured (pgsearch)
BM25 Score Calculation	paradedb.score(id)	pgsearch.score(id)
Boolean Multi-field Match	id @@@ paradedb.boolean(...)	id @@@ pgsearch.boolean(...)
Field Match Clauses	paradedb.match('text', $4) paradedb.match('context', $4) paradedb.match('text_signals', $4)	pgsearch.match('text', $4) pgsearch.match('context', $4) pgsearch.match('text_signals', $4)
Knowledge Page Match	paradedb.match('name', $3) paradedb.match('content', $3)	pgsearch.match('name', $3) pgsearch.match('content', $3)

---

Configuration Reference

Environment Variable	Description	Type	Default	Example Values
HINDSIGHT_API_TEXT_SEARCH_EXTENSION_PG_SEARCH_FUNCTION_SCHEMA	PostgreSQL schema containing pg_search functions (score, boolean, match).	String (PG Identifier)	paradedb	paradedb, pgsearch

Fallback & Validation Behavior

Input Value	Parsed Output	Behavior
unset	paradedb	Standard ParadeDB behavior (100% backwards-compatible)
"" (Empty string) / " "	paradedb	Graceful fallback to default
"pgsearch" / "PgSearch"	pgsearch	Normalized to lowercase valid schema identifier
"123schema" / "pg-search" / "bad;DROP TABLE"	ValueError	Rejected during startup validation (SQL injection safe)

---

Design Note: Function Schema (paradedb) vs. Tokenizer Schema (pdb)

ParadeDB pg_search uses a dual-schema architecture by design:

1. Target / Function Schema (paradedb by default): Hosts query-time functions (score(), boolean(), match()). This is the extension's target schema specified via CREATE EXTENSION pg_search SCHEMA <name> or configured by cloud distributions.
2. Tokenizer Type Schema (pdb): Hosts pseudo-types for index-time column casting (e.g. (text::pdb.icu), (text::pdb.jieba)). This is an auxiliary schema created by ParadeDB's internal installation DDL (CREATE SCHEMA IF NOT EXISTS pdb;), independent of the target schema.

Why the Tokenizer Schema (pdb) is Kept Fixed

• Empirical Cloud Behavior: As verified in [pg_search] Allow the extension function schema to be configured #3757, managed PostgreSQL distributions that install query functions under pgsearch still provide standard tokenizer types under pdb (e.g. pdb.icu).
• Standard Compatibility: No known PostgreSQL distribution alters the internal pdb type namespace, as doing so would break ParadeDB's canonical cast syntax across tooling.
• Minimal Surface Area: Making only the function schema configurable fully resolves the compatibility blocker without introducing unnecessary configuration complexity.

---

Technical Changes Summary

Layer	File	Description
Configuration	hindsight-api-slim/hindsight_api/config.py	Added env constants, HindsightConfig field, PostgreSQL identifier regex validation, and normalization.
SQL Abstraction	hindsight-api-slim/hindsight_api/engine/sql/base.py	Extended SQLDialect.build_bm25_arm interface with pg_search_function_schema.
SQL Dialects	hindsight-api-slim/hindsight_api/engine/sql/postgresql.py hindsight-api-slim/hindsight_api/engine/sql/oracle.py	Parameterized schema references in PostgreSQLDialect.build_bm25_arm and knowledge_bm25_arm.
Dispatchers	hindsight-api-slim/hindsight_api/engine/search/retrieval.py hindsight-api-slim/hindsight_api/engine/memory_engine.py	Forwarded configured schema from HindsightConfig to dialect arm builders.
Documentation & Env	.env.example docker/docker-compose/pg_search/docker-compose.yaml hindsight-docs/docs/developer/configuration.md	Updated templates and reference documentation.
Test Suite	hindsight-api-slim/tests/test_config_validation.py hindsight-api-slim/tests/test_db_abstraction.py hindsight-api-slim/tests/test_knowledge_bm25_dispatch.py	Added comprehensive test coverage for validation, recall SQL generation, and knowledge pages SQL generation.

---

Test & Validation Matrix

Test Suite	Scope	Result
test_config_validation.py	Default value, env loading, whitespace fallback, identifier validation & injection prevention	✅ Passed
test_db_abstraction.py	Memory recall BM25 arm SQL generation with custom schema	✅ Passed
test_knowledge_bm25_dispatch.py	Knowledge pages BM25 arm SQL generation with custom schema	✅ Passed
test_env_template.py	Embed template sync verification	✅ Passed
ty check hindsight_api/	Type checker across all modified API modules	✅ Passed
lint.sh	Project linting & formatting hooks	✅ Passed

[strix-security]

Strix Security Review

No security issues found.

_{Updated for 24a5c31.}

---

Reviewed by Strix
_{Re-run review · Configure security review settings}