Make the database portable and encryptable (#3848) - #5526

Open

shai-almog wants to merge 213 commits into

feature/portable-encryptable-database

Commits on Aug 6, 2026

Define the portable Database contract and add the encryption API

shai-almog
and
claude
committed
5b0f0b0
Make the simulator conformant and able to open encrypted databases

shai-almog
and
claude
committed
a29da0d
Make the iOS database conformant and fix three defects in it

shai-almog
and
claude
committed
85f6127
Bring Android to the database contract and add encrypted databases

shai-almog
and
claude
committed
0fedd67
Give the native Windows and Linux ports a database

shai-almog
and
claude
committed
b0e2de8
Replace WebSQL with SQLite on WebAssembly and gate the engine per app

shai-almog
and
claude
committed
dcf1f1f
Add device conformance tests, documentation and the interop proof

shai-almog
and
claude
committed
ce77b83
Fix the three CI failures from the first run

shai-almog
and
claude
committed
70dc81f
Exclude the SQLCipher package from the Ant Android build too

shai-almog
and
claude
committed
d595bd9
Address the review findings and the second round of CI failures

shai-almog
and
claude
committed
7f2f2c7
Address the second batch of review findings

shai-almog
and
claude
committed
766ab43
Address the third batch of review findings

shai-almog
and
claude
committed
1b78be8
Satisfy the forbidden PMD rules

shai-almog
and
claude
committed
c8ca4a5
Compile the engine's ciphers without assuming CPU features are enabled

shai-almog
and
claude
committed
dbcceec
Add copyright headers to the new guide snippet files

shai-almog
and
claude
committed
ba13354
Address the fourth review batch, the guide prose gate and CodeQL noise

shai-almog
and
claude
committed
ecad03f
Fix failed-commit recovery, parameter counts and the JS key probe

shai-almog
and
claude
committed
efbab6c
Report unbound parameters from executeQuery(String) with the shared message

shai-almog
and
claude
committed
f5767b0
Default Ant projects to database compatibility mode

shai-almog
and
claude
committed
e282819
Roll back through the engine when an Android commit fails

shai-almog
and
claude
committed
730a69a
Close out the fourth review round

shai-almog
and
claude
committed
cb0456a
Make Android encryption buildable, and actually exercise it

shai-almog
and
claude
committed
c6f57d4
Make encrypt and decrypt work on Android

shai-almog
and
claude
committed
452c6cf
Stop the JavaScript port hanging when the SQLite engine will not start

shai-almog
and
claude
committed
952f337
Do not assert a filesystem path where databases are not filesystem backed

shai-almog
and
claude
committed
7a70937
Close out the fifth review round

shai-almog
and
claude
committed
62e2271
Stop every JavaScript native binding from throwing across the bridge

shai-almog
and
claude
committed
4a11dc9
Fix three ways the previous round's fixes could fail

shai-almog
and
claude
committed
d4d3c89
Complete the migration recovery, and stop OPFS trouble losing data

shai-almog
and
claude
committed
483acb2
Type-check the SQLCipher package, which nothing else compiles

shai-almog
and
claude
committed
81cbb36

Commits on Aug 7, 2026

Close the remaining native and dispatch error paths

shai-almog
and
claude
committed
4c32755
Resolve the cipher check's dependencies from the reactor, not a repository

shai-almog
and
claude
committed
13cef92
Sweep the error paths I had fixed in only one implementation

shai-almog
and
claude
committed
2b527c3
Fix three defects the previous round's fixes introduced or left

shai-almog
and
claude
committed
ce68bdd
Move migration artifacts out of the database namespace entirely

shai-almog
and
claude
committed
79aceb2
Make delete and exists aware of interrupted conversions, and finish the JS sweep

shai-almog
and
claude
committed
ecf667b
Prove ownership of migration files instead of picking a location

shai-almog
and
claude
committed
14c1630
Report close failures, and track trigger bodies by position

shai-almog
and
claude
committed
be69b0b
Merge remote-tracking branch 'origin/master' into feature/portable-encryptable-database
shai-almog
committed
da4d201
Hoist casts out of catch regions and exchange SQL text as UTF-8 bytes

shai-almog
and
claude
committed
2420d98
Detect the port from the database, not from Display

shai-almog
and
claude
committed
26dcced
Record the export before writing it, and refuse a passphrase SQLite would cut short

shai-almog
and
claude
committed
c1fc047
Count named and numbered placeholders the way the engine does

shai-almog
and
claude
committed
f3e483b
Read a whole SQLite parameter name, not just its identifier characters

shai-almog
and
claude
committed
26a59d8
Merge remote-tracking branch 'origin/master' into feature/portable-encryptable-database
shai-almog
committed
643699a
Hold a port to its own legacy behaviour, not to another port's

shai-almog
and
claude
committed
018356f
Gate the SQLite payloads on the application, not on the framework

shai-almog
and
claude
committed
06b22a6
Keep windows.h from dragging winsock.h into the database bindings

shai-almog
and
claude
committed
8dc65cf
Finish gating the cipher, and stop recovery touching files it does not own

shai-almog
and
claude
committed
3d74b04
Document what using a database, and encrypting one, costs a build

shai-almog
and
claude
committed
a6bee19
Track transactions opened by script, and stop three ways to lose data

shai-almog
and
claude
committed
9b8733a

Commits on Aug 8, 2026

Count every Android connection, and read a script the way the engine does

shai-almog
and
claude
committed
aabde20
See a transaction whichever entry point opened it

shai-almog
and
claude
committed
15865f2
Read ROLLBACK TO as tokens, and hold the database for the whole conversion

shai-almog
and
claude
committed
b4b662e
Take the connection slot before the engine opens, and share one classifier

shai-almog
and
claude
committed
59d3853
Ask the engine whether a transaction is open, instead of inferring it

shai-almog
and
claude
committed
ca1cc1a
Claim the database for an in-place key rotation too

shai-almog
and
claude
committed
4712b47
Merge remote-tracking branch 'origin/master' into feature/portable-encryptable-database
shai-almog
committed
c7b0fe9
Keep the last readable copy, and make the claim cover recovery

shai-almog
and
claude
committed
14555d1
Hold a key change to one connection, and load the JS engine's wasm

shai-almog
and
claude
committed
266d6e8
Report an unavailable managed key as a note, not a skipped group

shai-almog
and
claude
committed
406b094
Hold the rekey claim for the whole rewrite, and ship a wasm that compiles

shai-almog
and
claude
committed
9f409ea
Key the simulator registry by the resolved file

shai-almog
and
claude
committed
b39c725
Reserve the slot before opening, and key the browser's engine through a VFS

shai-almog
and
claude
committed
e842054
Track savepoint transactions, and read the BEGIN mode as a token

shai-almog
and
claude
committed
27c2584
Claim the simulator's file before JDBC opens it, and stop a vacuous check

shai-almog
and
claude
committed
37e28b7
Read whole savepoint names, and hold the claim until the handle is gone

shai-almog
and
claude
committed
6a35664
Calibrate the ciphertext check, and stack savepoints of one name

shai-almog
and
claude
committed
57167c6
Ask the no-key question on a database nothing else depends on

shai-almog
and
claude
committed
a134e50
End a savepoint transaction through SQL on Android too

shai-almog
and
claude
committed
5f0bbb5
Return a Java long exactly from the browser, and count nested transactions

shai-almog
and
claude
committed
9be5fb3
Merge master into feature/portable-encryptable-database
shai-almog
committed
bf23c85
Move Google sign-in to a SDK that links on a simulator

shai-almog
and
claude
committed
3a1e67c
Track a legacy execute, and close the constructor nobody could satisfy

shai-almog
and
claude
committed
185633d
Keep a database an unkeyed open could not read, and nest only where nesting works

shai-almog
and
claude
committed
26bc092

Commits on Aug 9, 2026

Report a database that will not open as an IOException

shai-almog
and
claude
committed
9d325d9
Keep a savepoint stack, and hold the simulator to SQLite's grammar

shai-almog
and
claude
committed
3f8e3bd
Unwind the savepoint stack on ROLLBACK TO, and clean up what recovery displaces

shai-almog
and
claude
committed
3150fa1
Ask which config factory was called, and refresh a rewound cursor's count

shai-almog
and
claude
committed
74c115c
Key a managed alias on the resolved file on the remaining ports

shai-almog
and
claude
committed
c538064
Scan library jars and the database package, and say what count costs

shai-almog
and
claude
committed
75e3952
Retry the suite build past a Maven Central 403

shai-almog
and
claude
committed
fb4ad8c
Resolve a managed key the same way wherever it is looked up

shai-almog
and
claude
committed
02b051d
Track a parameterized savepoint, and stop the wrapper deadlocking on itself

shai-almog
and
claude
committed
2d72a0d
Destroy one managed key, not two, and refresh the count on any backward seek

shai-almog
and
claude
committed
f664689
End a savepoint transaction through SQL from a script too

shai-almog
and
claude
committed
b91482b
Merge remote-tracking branch 'origin/master' into feature/portable-encryptable-database
shai-almog
committed
0826b63
Fold savepoint names as SQLite does, and ask the engine after every statement

shai-almog
and
claude
committed
8a33f40
Check the worker before the lock, and read the engine on the failure path too

shai-almog
and
claude
committed
3bf5d9c

Commits on Aug 10, 2026

Restore the browser's engine read, and let isInTransaction skip the lock

shai-almog
and
claude
committed
f0840c9
Clear Android's flag when a rollback finds nothing to roll back

shai-almog
and
claude
committed
89a4517
Register a wrapped connection, and let queued callers hear that the database closed

shai-almog
and
claude
committed
a32ef74
Read a SQLite URI connection, and clear state when there is nothing left to end

shai-almog
and
claude
committed
63f20b4
Match the raw-key literal that is actually written, and free the cursor close

shai-almog
and
claude
committed
594b8df
Quote by the exact literal, rekey a nameless handle, and stop killing the worker

shai-almog
and
claude
committed
a051ebb

Commits on Aug 11, 2026

Merge remote-tracking branch 'origin/master' into feature/portable-encryptable-database
shai-almog
committed
5d47d7e
Resolve opaque SQLite URIs, and count a connection that cannot name its file

shai-almog
and
claude
committed
bb549e4
Keep the constructors a compiled library can still call

shai-almog
and
claude
committed
34894b8
Refuse a compatibility handle on a database that is being converted

shai-almog
and
claude
committed
fc1620a
Scan for database usage before the port joins the tree

shai-almog
and
claude
committed
241339d
Merge remote-tracking branch 'origin/master' into feature/portable-encryptable-database
shai-almog
committed
94db2ed
Carry the refusal instead of throwing it from a compatibility constructor

shai-almog
and
claude
committed
6f6d54d

Commits on Aug 12, 2026

Reconcile transaction state after a failed cursor step, and refuse to delete an open database

shai-almog
and
claude
committed
2e4e9a4
Do not raise for a rollback the engine already performed

shai-almog
and
claude
committed
262ed99
Count transaction control in SQL where the engine counts it

shai-almog
and
claude
committed
27a21b1
Decide iOS cipher availability by asking the engine

shai-almog
and
claude
committed
ff0f7a8
Keep engine settings that only one engine can honour out of the bundled build

shai-almog
and
claude
committed
da8eaf4

Commits on Aug 13, 2026

Never re-run a writing statement to move a cursor, and refuse to delete an open database

shai-almog
and
claude
committed
4473e02
Ask each port how many connections it has open, and treat a writing pragma as a write

shai-almog
and
claude
committed
c7822ed
Build the ciphers only when asked, and find that iOS had stopped encrypting

shai-almog
and
claude
committed
6579c12
Take the delete claim before the count is read, and let recovery hold the conversion claim

shai-almog
and
claude
committed
3898e19
Claim a delete in one step, and recover only for an open that holds the file alone

shai-almog
and
claude
committed
3920937
Rewind only when the cursor has somewhere to go back from

shai-almog
and
claude
committed
aeb3ae8
Refuse a delete under an unnamed connection or a running conversion

shai-almog
and
claude
committed
9661aa2
Record a conversion as unproven before installing it, not after it fails

shai-almog
and
claude
committed
45b16d5
Fix the duplicated package line in DatabaseTest

shai-almog
and
claude
committed
d3df098
Close a converted handle that cannot be proven, and name the displaced file first

shai-almog
and
claude
committed
0e656f6
Run the database checks when their own scripts change

shai-almog
and
claude
committed
6606910
Going back to a place the cursor has not left is a no-op

shai-almog
and
claude
committed
6ef695c
Keep transaction control out of executeQuery, and make the Android key probe run

shai-almog
and
claude
committed
d0784c7
Bind an empty value as empty rather than as null

shai-almog
and
claude
committed
cec6df0
Let the worker stop after it drains, instead of parking for the process

shai-almog
and
claude
committed
1bde0d4
Stop the worker even when the close fails, and read numbers the way SQLite does

shai-almog
and
claude
committed
1493c06
Validate an Android query without walking it, and mark a killed thread finished

shai-almog
and
claude
committed
525295f
Refuse work the moment a stop is asked for, and use a filter form that works

shai-almog
and
claude
committed
64171b8
Keep an Android cursor inside the rows it has, and refuse async work after a stop

shai-almog
and
claude
committed
6a68f2d
Read a row without counting the result set, and fix the wrapped indentation

shai-almog
and
claude
committed
ee0d4f0
Give each Windows and Linux application its own database directory

shai-almog
and
claude
committed
681388d
Prove a converted database opens, leave in-memory connections alone, clear stale engine files

shai-almog
and
claude
committed
aae711a
Refuse an open that recovery has to run before

shai-almog
and
claude
committed
76d61b3

Commits on Aug 14, 2026

Give the connection slot back even when the close fails

shai-almog
and
claude
committed
dbf51f0
Let the no-file answer decide, whatever the URL looks like

shai-almog
and
claude
committed
e04c2f2
Recognise the memory URLs exactly, not every name that starts like one

shai-almog
and
claude
committed
258e6a8
Validate an Android query by preparing it, not by moving its cursor

shai-almog
and
claude
committed
fde0b68
Keep marker entries inside their directory, and report engine failures from the cursor

shai-almog
and
claude
committed
ee17d11
Report a failed value read from the Android cursor as an IOException too

shai-almog
and
claude
committed
d4bc70f
Register a colon-named database under its file, from one reading of the URL

shai-almog
and
claude
committed
e780626
Let a writing cursor read the rows its own run produced

shai-almog
and
claude
committed
143441e
Say why a repeated write is refused even in compatibility mode

shai-almog
and
claude
committed
b6de9e0
Take the conformance suite out of the framework

shai-almog
and
claude
committed
53f2ab0

Commits on Aug 15, 2026

Do not run the statement to answer a rewind, and take the sidecars on delete

shai-almog
and
claude
committed
070fb52
Delete a database with its working files, on every port

shai-almog
and
claude
committed
0e368af
Report a failed sidecar unlink, and close the primary when its anchor fails

shai-almog
and
claude
committed
8764948
Give a native desktop application a home of its own, and a database under it

shai-almog
and
claude
committed
1fbafa4
Let a declared skip stand, and say when a delete left files behind

shai-almog
and
claude
committed
d3bdd70
Read a JavaScript column as text, and ask its type without fetching it

shai-almog
and
claude
committed
453453f
Do not leave a plaintext copy behind when the source will not close

shai-almog
and
claude
committed
7d628b6
Treat a colon as a drive only where drives exist

shai-almog
and
claude
committed
26d17f9

Commits on Aug 16, 2026

Key an iOS managed key on where the database sits, not where iOS put it

shai-almog
and
claude
committed
fb71af6
Refuse a seek that would have to run the write, and tell a failed read from a null

shai-almog
and
claude
committed
357a3e5
Ask the filesystem which file a Windows connection is on

shai-almog
and
claude
committed
c60d1c7
Let the registry and the managed alias be different questions

shai-almog
and
claude
committed
ea081eb
Merge master
shai-almog
committed
3eb98f7
Close the Windows peer when it cannot be registered, and fold names as the filesystem does

shai-almog
and
claude
committed
8dd918f
Re-key an iOS database under the alias its next open will look for

shai-almog
and
claude
committed
4445a2e
Key Windows databases on the folded name alone, and keep a UNC root

shai-almog
and
claude
committed
1c0c732
Keep reading a library after one entry inside it cannot be read

shai-almog
and
claude
committed
47ebe0f
Register an iOS connection under the identity a delete asks about

shai-almog
and
claude
committed
ba1139d
Close a simulator statement whatever goes wrong before a cursor takes it

shai-almog
and
claude
committed
fc50e81
Clear the Linux keyring secret, refuse to empty a link, and say the writes are done

shai-almog
and
claude
committed
4d301cf
Keep the token when the keyring will not give the secret up

shai-almog
and
claude
committed
b3aa28e
Never generate a managed key on a lookup that only failed

shai-almog
and
claude
committed
ae3abfe
Count an attached database as open, and stop reading existence as readability

shai-almog
and
claude
committed
d1d3f69
Track only the attachments this can actually identify

shai-almog
and
claude
committed
35275d9
Reserve an attached database before the engine attaches it

shai-almog
and
claude
committed
7386f3f
Ask the engine what is attached instead of reading the SQL

shai-almog
and
claude
committed
a327fbd

Commits on Aug 17, 2026

Track an attachment by its file, and see what a parameter attaches

shai-almog
and
claude
committed
1daca64
Undo an attachment nothing can hold, and refuse one a cursor would hide

shai-almog
and
claude
committed
cd6795d
Detach every schema an unholdable file was attached under

shai-almog
and
claude
committed
ca19e5d
Read an ATTACH past the comments around it, without a cast that cannot fail

shai-almog
and
claude
committed
ea93c1f
Follow links into a database key where the engine follows them

shai-almog
and
claude
committed
8fbb057
Refuse an ATTACH whose filename this cannot resolve before it runs

shai-almog
and
claude
committed
968acf4
Refuse a relative ATTACH where the engine resolves it elsewhere

shai-almog
and
claude
committed
54f130e
Merge remote-tracking branch 'origin/master' into feature/portable-encryptable-database
shai-almog
committed
6ca3d04
Take an ATTACH placeholder's own slot, not always the first parameter

shai-almog
and
claude
committed
71f0cc8
Write out the SQLite native entry points the verifier has to read

shai-almog
and
claude
committed
2446f8e
Carry auto_vacuum across a conversion, and report a dead RNG as KEY_UNAVAILABLE

shai-almog
and
claude
committed
cac2c8b
Place the GC gate's hazard inside a mark instead of sleeping through one

shai-almog
and
claude
committed
0ced137
Resolve a database path through a native, not java.io.File

shai-almog
and
claude
committed
0a4f256
Carry the journal mode across an Android key conversion too

shai-almog
and
claude
committed
12ff173
Put the connection's own settings back after an Android conversion

shai-almog
and
claude
committed
9de42e9

Commits on Aug 18, 2026

Run every pragma through a cursor, and bind #name like the other three

shai-almog
and
claude
committed
912fb34
Restore every connection pragma a conversion loses, not the three noticed so far

shai-almog
and
claude
committed
78abc46
Identify a created Windows file after it exists, and derive the pragma list from the engine

shai-almog
and
claude
committed
cf18208
Separate the registry key from the managed key alias, and refuse a URI that means another file

shai-almog
and
claude
committed
0a0a9d8
Read the class file with ASM instead of searching it for bytes

shai-almog
and
claude
committed
30deb2b
Say what an unreadable class costs, and prove Java 17 is not one

shai-almog
and
claude
committed
47b227e
Read a lambda, an empty file and a stopping worker for what they are

shai-almog
and
claude
committed
9e4eb8d
Say in close()'s contract that it waits for work the worker took

shai-almog
and
claude
committed
b0fd7d3
Refuse an ATTACH whose URI carries an authority

shai-almog
and
claude
committed
b21d923
Retry the two steps that died on Maven Central, and only for that

shai-almog
and
claude
committed
17cafc4
Refuse to open a browser database whose data is in the old WebSQL store

shai-almog
and
claude
committed
3250a30
Answer a key change honestly on the two Android connections

shai-almog
and
claude
committed
9f37e3a
Let a plaintext Android database be asked for no key at all

shai-almog
and
claude
committed
5027ea4
Stop reading an unreachable browser engine as an empty one

shai-almog
and
claude
committed
eb01f38
Refuse an unidentified connection while any file is claimed

shai-almog
and
claude
committed
6fad0e7
Delete a database last, after the files that hold its pages

shai-almog
and
claude
committed
1b0f3b0
Key a managed secret to the application, not to what it is called

shai-almog
and
claude
committed
6908b8d
Retry the receipt-sync tests when Central refuses, and only then

shai-almog
and
claude
committed
81aee05
Separate one simulator project's secrets from another's

shai-almog
and
claude
committed
d1692b9
Wait for the inspector's details form before capturing it

shai-almog
and
claude
committed
5d73fd1
Give the simulator verifier the header every source here carries

shai-almog
and
claude
committed
0334a8d
Wait for the inspector panel to settle, rather than for the state it passes through

shai-almog
and
claude
committed
3c41c32
Give the Central retries a window the size of the outages

shai-almog
and
claude
committed
ad096aa
Widen the Central window on the Linux core build too

shai-almog
and
claude
committed
acff084
Retry the archetype build when Central refuses the install plugin

shai-almog
and
claude
committed
f33c50e

Commits on Aug 19, 2026

Create a managed key rather than writing one, and keep namespaces distinct

shai-almog
and
claude
committed
6cf8164
Merge remote-tracking branch 'origin/master' into feature/portable-encryptable-database
shai-almog
committed
3c27e69
Decide a first managed key where the operating system decides it

shai-almog
and
claude
committed
a3d9274
Stop apt hanging a job for six hours

shai-almog
and
claude
committed
961f177
Bound the apt calls that run inside the container too

shai-almog
and
claude
committed
70ad716
Give the Larger Text test the header this repository requires

shai-almog
and
claude
committed
b0e037c
Report what the database tests actually did, and refuse to publish a test that never ran

shai-almog
and
claude
committed
ba7f0b4