Harden namespace directory ownership and lifecycle cleanup

README.md

@@ -33,8 +33,14 @@
 
 ---
 
-> [!NOTE]
-> This repository contains libSQL, a fork of SQLite developed by Turso. For the full SQLite rewriten in Rust (also by Turso), please visit [tursodatabase/turso](https://github.com/tursodatabase/turso).
+> [!IMPORTANT]
+> **Turso database and libSQL are two different projects from the same team.**
+>
+> **libSQL** (this repository) is an open-source fork of SQLite. It extends SQLite with features like embedded replicas and remote access, but inherits SQLite's fundamental limitations such as the single-writer model.
+>
+> **[Turso database](https://github.com/tursodatabase/turso)** is a SQLite-compatible database rewritten from scratch in Rust. It is **not** a fork of SQLite — it is a completely new implementation that goes beyond what any SQLite fork can offer, including concurrent writes and bi-directional sync with offline support. Turso is currently in beta.
+>
+> **If you're starting a new project, you probably want to look into [Turso](https://github.com/tursodatabase/turso).** libSQL is actively maintained, but new features are being developed in Turso.
 
 ## Documentation
 

bottomless-cli/src/replicator_extras.rs

@@ -382,9 +382,8 @@ impl Replicator {
             let frame = tokio::fs::File::open(&obj).await?;
             let frame_buf_reader = BufReader::new(frame);
 
-            let mut frameno = first_frame_no;
             let mut reader = bottomless::read::BatchReader::new(
-                frameno,
+                first_frame_no,
                 frame_buf_reader,
                 page_size as usize,
                 compression_kind,
@@ -411,7 +410,6 @@ impl Replicator {
                     );
                     pending_pages.flush(db).await?;
                 }
-                frameno += 1;
                 last_received_frame_no += 1;
             }
             db.flush().await?;

docs/ADMIN_API.md

@@ -9,13 +9,91 @@ To enable the admin API, and manage namespaces, two extra flags need to be passe
 - `--admin-listen-addr <addr>:<port>`: the address and port on which the admin API should listen. It must be different from the user API listen address (which defaults to port 8080).
 - `--enable-namespaces`: enable namespaces for the instance. By default namespaces are disabled.
 
+## Namespace names
+
+Namespace names must be non-empty single filesystem components. `.` and `..`,
+forward/backward slashes, and NUL are rejected, including percent-encoded path
+parameters after HTTP decoding. Safe existing names with spaces, punctuation,
+and Unicode remain supported. On Windows, invalid Win32 characters, trailing
+ASCII dots/spaces, and reserved device names are also rejected. These rules
+also apply to fork source/destination and shared-schema names. Invalid names
+return `400 Bad Request` on the admin API.
+
+On upgrade, invalid namespace names or configs in the metastore (including
+invalid shared-schema names) prevent startup instead of being treated as absent
+or allowing their rows to be overwritten; this holds even
+with `--meta-store-destroy-on-error`. Filesystem recovery skips invalid and
+symlinked directory entries without deleting them. An invalid persisted
+migration job/task stops its scheduler without marking that work complete.
+Back up and inspect metastore and namespace files before repairing these entries
+explicitly.
+
+Validation prevents path traversal *through a namespace string*. Directory
+ownership checks additionally reserve new namespace/fork directories atomically,
+reject any existing entry (including aliases and orphan directories), and
+require an unloaded persisted namespace's actual directory entry to match its
+stored name. A legacy alias or symlink is refused rather than opened or
+deleted. These checks assume the data directory is trusted; they do not make
+filesystem operations atomic against a privileged external process replacing
+paths or symlinks outside the server's coordination locks.
+
+A cancelled create/fork or failed cleanup can retain a newly reserved directory
+as quarantine after metadata has been removed, so delayed writes cannot reach
+a retry. Inspect it and the metastore after the work stops before repairing or
+retrying. Per-name operation locks allow unrelated namespace administration to
+continue during slow restores. Shutdown signals in-flight create/fork work to
+stop and permits a bounded drain before reporting an error. A replica detecting
+an incompatible log moves its old files to `replica-log-quarantine/` outside
+`dbs/`, retaining the namespace directory identity, then retries once. Inspect
+quarantined files before removal. Destroy/reset confirms remote backups before
+moving a directory to `namespace-teardown-quarantine/`; backup failure before
+confirmation leaves the old directory and metastore row in place. After
+confirmation, independently draining workers own the name lock even if the HTTP
+request is cancelled. Destroy then removes metadata and the old directory.
+Before deleting metadata, destroy atomically publishes a fully written `namespace-destroy-intents/` record identifying the
+old directory; unpublished temporary records are discarded on startup.
+Reconciliation rolls back an uncommitted intent or finishes a committed delete
+before namespaces are served. A mismatched/missing live inode or unexpected
+quarantine fails startup for operator repair rather than opening a blank DB.
+Incomplete intents also fence create, fork, replica load, and reset until
+recovered. Reset separately publishes a `namespace-reset-intents/` record with
+the old inode and persisted config before detaching the old directory. The old
+files remain quarantined while the replacement is set up. A pending reset
+rejects changing its shared-schema membership: the original link must keep the
+old schema available, and a second link would incorrectly enlist the partial
+database in schema migrations. A reset waits for the schema registration lock
+and refuses an existing migration; new migrations are rejected while a linked
+tenant or the schema namespace itself has a pending reset intent, avoiding
+scheduler retry exhaustion. Only a
+durable commit marker permits release of this restriction and deletion of the
+old files. Before that marker, a crash
+restores the old files and config and preserves partial new files under
+`namespace-reset-abandoned/` for inspection. A setup error keeps the name
+fenced until process restart, when all late setup workers have stopped;
+rollback during the live process could otherwise redirect late path opens into
+the old database. Invalid identities or interrupted recovery fail closed.
+Unix directory fsync orders the intent ahead of the SQLite commit;
+sudden power-loss durability is not guaranteed on Windows (which has no
+portable directory fsync here), nor is macOS physical flush guaranteed by
+ordinary fsync. Process-crash recovery is supported on both. Short filesystem
+identity scans and renames remain synchronous under the filesystem lock:
+offloading only the syscalls would allow cancellation to release a name lock
+while late writes are still running. Large directories can therefore briefly
+stall a current-thread runtime pending a separately coordinated offload. Stale
+config/replication handles from a prior incarnation cannot reinsert rows or
+links after deletion; a fresh create/reset uses a new generation. Unexpected
+identity changes are preserved for explicit operator repair. Inspect remaining
+quarantine files after interrupted teardown.
+
 ## Routes
 
 ```HTTP
 POST /v1/namespaces/:namespace/create
 ```
 
-Create a namespace named `:namespace`.
+Create a namespace named `:namespace`. Explicit creation rejects an existing
+namespace, including `default`; internal startup/lazy loading of `default`
+reuses its persisted config rather than replacing it.
 body:
 
 ```json

libsql-ffi/bundled/SQLite3MultipleCiphers/src/sqlite3.c

@@ -211731,7 +211731,7 @@ static int vectorParseSqliteText(
       continue;
     }
     if( this != ',' && this != ']' ){
-      if( iBuf > MAX_FLOAT_CHAR_SZ ){
+      if( iBuf >= MAX_FLOAT_CHAR_SZ ){
         *pzErrMsg = sqlite3_mprintf("vector: float string length exceeded %d characters: '%s'", MAX_FLOAT_CHAR_SZ, valueBuf);
         goto error;
       }

libsql-ffi/bundled/src/sqlite3.c

@@ -211731,7 +211731,7 @@ static int vectorParseSqliteText(
       continue;
     }
     if( this != ',' && this != ']' ){
-      if( iBuf > MAX_FLOAT_CHAR_SZ ){
+      if( iBuf >= MAX_FLOAT_CHAR_SZ ){
         *pzErrMsg = sqlite3_mprintf("vector: float string length exceeded %d characters: '%s'", MAX_FLOAT_CHAR_SZ, valueBuf);
         goto error;
       }

libsql-server/src/admin_shell.rs

@@ -2,7 +2,6 @@ use std::fmt::Display;
 use std::pin::Pin;
 use std::str::FromStr;
 
-use bytes::Bytes;
 use dialoguer::BasicHistory;
 use rusqlite::types::ValueRef;
 use tokio_stream::{Stream, StreamExt as _};
@@ -37,10 +36,9 @@ impl AdminShell {
 
     async fn with_namespace(
         &self,
-        ns: Bytes,
+        namespace: NamespaceName,
         queries: impl Stream<Item = Result<rpc::Query, tonic::Status>>,
     ) -> anyhow::Result<impl Stream<Item = Result<rpc::Response, tonic::Status>>> {
-        let namespace = NamespaceName::from_bytes(ns).unwrap();
         let connection_maker = self
             .namespace_store
             .with(namespace, |ns| ns.db.connection_maker())
@@ -107,6 +105,61 @@ fn try_run_one(conn: &mut rusqlite::Connection, q: String) -> anyhow::Result<rpc
     })
 }
 
+fn namespace_from_metadata(
+    metadata: &tonic::metadata::MetadataMap,
+) -> Result<NamespaceName, tonic::Status> {
+    let namespace = metadata
+        .get_bin("x-namespace-bin")
+        .ok_or_else(|| tonic::Status::invalid_argument("missing namespace"))?;
+    let bytes = namespace
+        .to_bytes()
+        .map_err(|_| tonic::Status::invalid_argument("bad namespace encoding"))?;
+    NamespaceName::from_bytes(bytes)
+        .map_err(|_| tonic::Status::invalid_argument("invalid namespace name"))
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn rejects_invalid_namespace_metadata() {
+        let mut metadata = tonic::metadata::MetadataMap::new();
+        assert_eq!(
+            namespace_from_metadata(&metadata).unwrap_err().code(),
+            tonic::Code::InvalidArgument
+        );
+        for name in [
+            b"".as_slice(),
+            b"..",
+            b"../outside",
+            b"/outside",
+            b"a\\b",
+            b"a\0b",
+            b"\xff",
+        ] {
+            metadata.insert_bin("x-namespace-bin", BinaryMetadataValue::from_bytes(name));
+            assert_eq!(
+                namespace_from_metadata(&metadata).unwrap_err().code(),
+                tonic::Code::InvalidArgument,
+                "{name:?}"
+            );
+        }
+    }
+
+    #[test]
+    fn accepts_safe_namespace_metadata() {
+        let mut metadata = tonic::metadata::MetadataMap::new();
+        for name in ["default", "tenant-1.example", "tenant café"] {
+            metadata.insert_bin(
+                "x-namespace-bin",
+                BinaryMetadataValue::from_bytes(name.as_bytes()),
+            );
+            assert_eq!(namespace_from_metadata(&metadata).unwrap().as_str(), name);
+        }
+    }
+}
+
 #[async_trait::async_trait]
 impl AdminShellService for AdminShell {
     type ShellStream = Pin<Box<dyn Stream<Item = Result<rpc::Response, tonic::Status>> + Send>>;
@@ -115,20 +168,9 @@ impl AdminShellService for AdminShell {
         &self,
         request: tonic::Request<tonic::Streaming<rpc::Query>>,
     ) -> std::result::Result<tonic::Response<Self::ShellStream>, tonic::Status> {
-        let Some(namespace) = request.metadata().get_bin("x-namespace-bin") else {
-            return Err(tonic::Status::new(
-                tonic::Code::InvalidArgument,
-                "missing namespace",
-            ));
-        };
-        let Ok(ns_bytes) = namespace.to_bytes() else {
-            return Err(tonic::Status::new(
-                tonic::Code::InvalidArgument,
-                "bad namespace encoding",
-            ));
-        };
+        let namespace = namespace_from_metadata(request.metadata())?;
 
-        match self.with_namespace(ns_bytes, request.into_inner()).await {
+        match self.with_namespace(namespace, request.into_inner()).await {
             Ok(s) => Ok(tonic::Response::new(Box::pin(s))),
             Err(e) => Err(tonic::Status::new(
                 tonic::Code::FailedPrecondition,

libsql-server/src/connection/config.rs

@@ -65,30 +65,36 @@ impl Default for DatabaseConfig {
     }
 }
 
-impl From<&metadata::DatabaseConfig> for DatabaseConfig {
-    fn from(value: &metadata::DatabaseConfig) -> Self {
-        DatabaseConfig {
+impl TryFrom<&metadata::DatabaseConfig> for DatabaseConfig {
+    type Error = crate::Error;
+
+    fn try_from(value: &metadata::DatabaseConfig) -> Result<Self, Self::Error> {
+        Ok(DatabaseConfig {
             block_reads: value.block_reads,
             block_writes: value.block_writes,
             block_reason: value.block_reason.clone(),
             max_db_pages: value.max_db_pages,
-            heartbeat_url: value.heartbeat_url.as_ref().map(|s| Url::parse(s).unwrap()),
+            heartbeat_url: value
+                .heartbeat_url
+                .as_ref()
+                .map(|s| Url::parse(s))
+                .transpose()?,
             bottomless_db_id: value.bottomless_db_id.clone(),
             jwt_key: value.jwt_key.clone(),
             txn_timeout: value.txn_timeout_s.map(Duration::from_secs),
             allow_attach: value.allow_attach,
             max_row_size: value.max_row_size.unwrap_or_else(default_max_row_size),
             is_shared_schema: value.shared_schema.unwrap_or(false),
-            // namespace name is coming from primary, we assume it's valid
             shared_schema_name: value
                 .shared_schema_name
                 .clone()
-                .map(NamespaceName::new_unchecked),
+                .map(NamespaceName::from_string)
+                .transpose()?,
             durability_mode: match value.durability_mode {
                 None => DurabilityMode::default(),
                 Some(m) => DurabilityMode::from(metadata::DurabilityMode::try_from(m)),
             },
-        }
+        })
     }
 }
 
@@ -112,6 +118,47 @@ impl From<&DatabaseConfig> for metadata::DatabaseConfig {
     }
 }
 
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn replicated_config_rejects_unsafe_shared_schema_names() {
+        for name in ["", ".", "..", "../schema", "/schema", "a\\b", "a\0b"] {
+            let config = metadata::DatabaseConfig {
+                shared_schema_name: Some(name.into()),
+                ..Default::default()
+            };
+            assert!(
+                matches!(
+                    DatabaseConfig::try_from(&config),
+                    Err(crate::Error::InvalidNamespace)
+                ),
+                "{name:?}"
+            );
+        }
+    }
+
+    #[test]
+    fn replicated_config_preserves_safe_shared_schema_names() {
+        for name in [None, Some("schema-1.example"), Some("tenant café")] {
+            let config = metadata::DatabaseConfig {
+                shared_schema_name: name.map(str::to_owned),
+                ..Default::default()
+            };
+            let decoded = DatabaseConfig::try_from(&config).unwrap();
+            assert_eq!(
+                decoded.shared_schema_name.as_ref().map(|n| n.as_str()),
+                name
+            );
+            assert_eq!(
+                metadata::DatabaseConfig::from(&decoded).shared_schema_name,
+                config.shared_schema_name
+            );
+        }
+    }
+}
+
 /// Durability mode specifies the `PRAGMA SYNCHRONOUS` setting for the connection
 #[derive(PartialEq, Clone, Copy, Debug, Deserialize, Serialize, Default)]
 #[serde(rename_all = "lowercase")]

libsql-server/src/error.rs

@@ -70,6 +70,8 @@ pub enum Error {
     NamespaceAlreadyExist(String),
     #[error("Invalid namespace")]
     InvalidNamespace,
+    #[error("Invalid persisted namespace config for `{namespace}`: {reason}. Repair the persisted config before restarting; no data was removed")]
+    InvalidPersistedNamespaceConfig { namespace: String, reason: String },
     #[error("Invalid namespace bytes: `{0}`")]
     InvalidNamespaceBytes(Box<dyn std::error::Error + Sync + Send + 'static>),
     #[error("Replica meta error: {0}")]
@@ -192,6 +194,9 @@ impl IntoResponse for &Error {
             PrimaryConnectionTimeout => self.format_err(StatusCode::INTERNAL_SERVER_ERROR),
             NamespaceAlreadyExist(_) => self.format_err(StatusCode::BAD_REQUEST),
             InvalidNamespace => self.format_err(StatusCode::BAD_REQUEST),
+            InvalidPersistedNamespaceConfig { .. } => {
+                self.format_err(StatusCode::INTERNAL_SERVER_ERROR)
+            }
             InvalidNamespaceBytes(_) => self.format_err(StatusCode::BAD_REQUEST),
             LoadDumpError(e) => e.into_response(),
             InvalidMetadataBytes(_) => self.format_err(StatusCode::INTERNAL_SERVER_ERROR),