From 7e20634dc4ad0297c5a9159163969db2cfc1f609 Mon Sep 17 00:00:00 2001 From: Vincenzo Palazzo Date: Mon, 21 Sep 2026 20:15:33 -0700 Subject: [PATCH 1/2] docs: add production ldk-server update skill Record the host update path so a server bump always installs the matching CLI. The July CLI signs only the timestamp and fails against the body HMAC with Invalid credentials. Written with AI assistance (Goose). --- .goose/skills/ldk-server-update/SKILL.md | 225 +++++++++++++++++++++++ 1 file changed, 225 insertions(+) create mode 100644 .goose/skills/ldk-server-update/SKILL.md diff --git a/.goose/skills/ldk-server-update/SKILL.md b/.goose/skills/ldk-server-update/SKILL.md new file mode 100644 index 00000000..e4ebec08 --- /dev/null +++ b/.goose/skills/ldk-server-update/SKILL.md @@ -0,0 +1,225 @@ +--- +name: ldk-server-update +description: > + Update the production ldk-server on vincent@65.108.246.14 to a specific + commit or tag, and install the matching ldk-server and ldk-server-cli + together. Trigger on: update ldk-server, deploy ldk-server, restart + ldk-server, bump ldk-server, install a commit, install a tag, + ldk-cli auth failed after upgrade. +--- + +# Update production ldk-server + +Update the live node to one commit or tag. Always install **both** +binaries from that same build. A new server with the old CLI fails +closed: `Error (Authentication Error): Invalid credentials`. + +This is the Hetzner node, not a generic install. Do not invent a +systemd unit. The process is started by hand and reparented to PID 1. + +## Host + +| Item | Value | +|---|---| +| SSH | `vincent@65.108.246.14` | +| Config | `/home/vincent/ldk-server-mainnet.toml` | +| Storage | `/home/vincent/.ldk-server/` | +| Secrets | `keys_seed` or `keys_mnemonic`, `bitcoin/api_key`, `tls.crt`, `tls.key` | +| gRPC | `65.108.246.14:3536` (TLS, not loopback) | +| P2P | `127.0.0.1:9735` | +| Build checkout | `/home/vincent/src/ldk-server` | +| Install dir | `/home/vincent/.local/bin` | +| Commands | `ldk-server`, `ldk-server-cli`, `ldk-cli` → `ldk-server-cli` | +| Rev file | `/home/vincent/.local/bin/ldk-server.rev` | +| Fork remote | `git@github.com:vincenzopalazzo/ldk-server.git` | +| Upstream | `git@github.com:lightningdevkit/ldk-server.git` | +| Root disk | `/` — build here. `/mnt/HC_Volume_103194752` stays near full; do not build there. | + +`~/.local/bin` is already on `PATH`. + +## Why both binaries + +Since `819bed8` the server HMAC is `HMAC-SHA256(api_key, timestamp || body)`. +The July 2026 CLI signs only the timestamp. The header still parses, so +the only error is `Invalid credentials`. The `api_key` file did not change. + +The CLI hex-encodes the 32 raw bytes in `bitcoin/api_key` itself. Do not +pass `-a $(xxd -p api_key)` to a current CLI. That double-encodes the key +and fails the same way. + +Never leave `ldk-cli` pointing at a checkout under +`/mnt/HC_Volume_103194752`. That symlink is how the July CLI survived the +last upgrade. + +## Rules + +- Ask before restarting. Building and installing does not require a restart. +- Never delete or overwrite `keys_seed`, `keys_mnemonic`, channel storage, + `api_key`, or the TLS files. +- If both `keys_seed` and `keys_mnemonic` exist, stop. They are different + secrets. Do not choose one. +- Do not `cargo clean` on the volume checkout. +- Do not force-push `main`. +- One SSH command per purpose. A `====` banner breaks the remote zsh. + +## 1. Resolve the revision + +`` is a commit, branch, or tag. Prefer the fork if the commit is +only there (`feat/keys-seed-compat` and similar). Use upstream for a +stock release. + +```bash +ssh vincent@65.108.246.14 'set -euo pipefail +cd /home/vincent/src/ldk-server +git fetch origin +git fetch https://github.com/lightningdevkit/ldk-server.git main:refs/remotes/upstream/main +git rev-parse --verify "^{commit}" +git log -1 --oneline "" +' +``` + +If `` is not on the fork, fetch the URL or ref that contains it. +Do not guess. + +Record the running identity before changing anything: + +```bash +ssh vincent@65.108.246.14 'set -euo pipefail +pgrep -a ldk-server +tr "\0" " " < /proc/$(pgrep -n -x ldk-server)/cmdline; echo +sha256sum /proc/$(pgrep -n -x ldk-server)/exe +/home/vincent/.local/bin/ldk-cli -b 65.108.246.14:3536 get-node-info +ls -la /home/vincent/.ldk-server/keys_seed /home/vincent/.ldk-server/keys_mnemonic /home/vincent/.ldk-server/bitcoin/api_key +' +``` + +Save the node id. A successful update must print the same one. + +## 2. Build both binaries + +Build in `/home/vincent/src/ldk-server`, on `/`. Check `df -h /` first. +Need several GB free. `rustc` on this host is 1.95, which is enough for +current `main`. + +```bash +ssh vincent@65.108.246.14 'set -euo pipefail +cd /home/vincent/src/ldk-server +git fetch origin +git checkout --detach "" +cargo build --release -p ldk-server -p ldk-server-cli +git rev-parse HEAD +sha256sum target/release/ldk-server target/release/ldk-server-cli +' +``` + +Detached HEAD is intentional. Do not commit on the server. Do not switch +the operator laptop checkout to do this. + +## 3. Install both, atomically + +Copy the binaries. Do not symlink them back into `target/release`. A +later build would change the CLI under a running server, or the reverse. + +```bash +ssh vincent@65.108.246.14 'set -euo pipefail +SRC=/home/vincent/src/ldk-server/target/release +INSTALL=/home/vincent/.local/bin +STAGE=$(mktemp -d) +install -m 0755 "$SRC/ldk-server" "$STAGE/ldk-server" +install -m 0755 "$SRC/ldk-server-cli" "$STAGE/ldk-server-cli" +# publish cli first, then the server binary, then the ldk-cli name +mv -f "$STAGE/ldk-server-cli" "$INSTALL/ldk-server-cli" +mv -f "$STAGE/ldk-server" "$INSTALL/ldk-server" +ln -sfn "$INSTALL/ldk-server-cli" "$INSTALL/ldk-cli" +rmdir "$STAGE" +REV=$(git -C /home/vincent/src/ldk-server rev-parse --short=12 HEAD) +printf "%s %s\n" "$(date -u +%Y%m%dT%H%M%SZ)" "$REV" > "$INSTALL/ldk-server.rev" +sha256sum "$INSTALL/ldk-server" "$INSTALL/ldk-server-cli" +cmp "$INSTALL/ldk-server" "$SRC/ldk-server" +cmp "$INSTALL/ldk-server-cli" "$SRC/ldk-server-cli" +' +``` + +Stop here unless the operator asked to restart. The old process keeps +running the previous server binary. The new CLI may already fail against +it if the auth scheme changed. Say that, and do not "fix" it by restarting +unasked. + +## 4. Restart only when asked + +There is no systemd unit. The live process has PPID 1. `SIGHUP` reopens +the log. It does **not** exec a new binary. + +```bash +ssh vincent@65.108.246.14 'set -euo pipefail +OLD=$(pgrep -n -x ldk-server) +kill -TERM "$OLD" +for _ in 1 2 3 4 5 6 7 8 9 10; do + kill -0 "$OLD" 2>/dev/null || break + sleep 1 +done +if kill -0 "$OLD" 2>/dev/null; then + echo "old process still alive; not sending KILL" >&2 + exit 1 +fi +nohup /home/vincent/.local/bin/ldk-server /home/vincent/ldk-server-mainnet.toml \ + >> /home/vincent/.ldk-server/ldk-server.stdout 2>&1 & +echo $! > /home/vincent/.ldk-server/ldk-server.pid +disown || true +' +``` + +Wait until the log says the gRPC listener is up before calling the CLI. +If it does not come up, do not start a second copy. + +## 5. Prove it + +```bash +ssh vincent@65.108.246.14 'set -euo pipefail +pgrep -a ldk-server +tr "\0" " " < /proc/$(pgrep -n -x ldk-server)/cmdline; echo +readlink -f /proc/$(pgrep -n -x ldk-server)/exe +sha256sum /proc/$(pgrep -n -x ldk-server)/exe /home/vincent/.local/bin/ldk-server +/home/vincent/.local/bin/ldk-cli -b 65.108.246.14:3536 get-node-info +/home/vincent/.local/bin/ldk-cli -b 65.108.246.14:3536 list-channels +' +``` + +Pass only if all of these are true: + +- The process exe is `/home/vincent/.local/bin/ldk-server`. +- Its sha256 matches `ldk-server-cli`'s sibling from the same install. +- `ldk-cli get-node-info` returns the node id recorded in step 1. +- `keys_seed` / `keys_mnemonic` were not replaced. A new `keys_mnemonic` + next to an old `keys_seed` means the process generated a second identity. + Stop and report it. +- `list-channels` still shows the channels that were usable before. + +Also check the log for `Loaded node entropy from` and +`Starting ldk-server version`. The version line includes the git commit. + +## Rollback + +Keep the previous pair. Before installing, if those files exist: + +```bash +cp -a ~/.local/bin/ldk-server ~/.local/bin/ldk-server.prev +cp -a ~/.local/bin/ldk-server-cli ~/.local/bin/ldk-server-cli.prev +``` + +Rollback is the same restart, with the `.prev` files moved back into +place, then `ln -sfn` for `ldk-cli`. Never roll the server back and leave +the new CLI, or the reverse. + +Channel storage is not part of the binary rollback. Do not restore an +older sqlite/postgres snapshot over the one the node just wrote. + +## Auth failure after an update + +If `get-node-info` returns `Invalid credentials`: + +1. Confirm `ldk-cli` resolves to `~/.local/bin/ldk-server-cli`. +2. Confirm that file's mtime matches `~/.local/bin/ldk-server`. +3. Do not hex-encode `api_key` again. +4. If the CLI is older than the server, reinstall the pair from the + running commit. Do not rotate `api_key`. From 600f0715c2829ad4eb4c4f02d90e75799aa4fb22 Mon Sep 17 00:00:00 2001 From: Vincenzo Palazzo Date: Mon, 21 Sep 2026 20:19:57 -0700 Subject: [PATCH 2/2] docs: describe a paired ldk-server and CLI update An old CLI signs only the timestamp. Current servers also sign the request body, so a server-only upgrade fails closed with Invalid credentials. Keep the procedure generic: no host addresses, install both binaries from the same revision, and restart only when asked. Written with AI assistance (Goose). --- .goose/skills/ldk-server-update/SKILL.md | 245 +++++++++++------------ 1 file changed, 122 insertions(+), 123 deletions(-) diff --git a/.goose/skills/ldk-server-update/SKILL.md b/.goose/skills/ldk-server-update/SKILL.md index e4ebec08..6727fd7b 100644 --- a/.goose/skills/ldk-server-update/SKILL.md +++ b/.goose/skills/ldk-server-update/SKILL.md @@ -1,157 +1,164 @@ --- name: ldk-server-update description: > - Update the production ldk-server on vincent@65.108.246.14 to a specific - commit or tag, and install the matching ldk-server and ldk-server-cli - together. Trigger on: update ldk-server, deploy ldk-server, restart - ldk-server, bump ldk-server, install a commit, install a tag, - ldk-cli auth failed after upgrade. + Update a running ldk-server to a specific commit or tag, and install the + matching ldk-server and ldk-server-cli together. Trigger on: update + ldk-server, deploy ldk-server, restart ldk-server, bump ldk-server, + install a commit, install a tag, ldk-cli auth failed after upgrade. --- -# Update production ldk-server +# Update ldk-server -Update the live node to one commit or tag. Always install **both** -binaries from that same build. A new server with the old CLI fails -closed: `Error (Authentication Error): Invalid credentials`. +Update one running node to one commit or tag. Always install **both** +binaries from that same build. A new server with an old CLI fails closed: -This is the Hetzner node, not a generic install. Do not invent a -systemd unit. The process is started by hand and reparented to PID 1. +```text +Error (Authentication Error): Invalid credentials +``` + +The API key file did not change. The signature scheme did. -## Host +## Inputs -| Item | Value | +Ask if any of these are not already known. Do not invent them. + +| Input | Example | |---|---| -| SSH | `vincent@65.108.246.14` | -| Config | `/home/vincent/ldk-server-mainnet.toml` | -| Storage | `/home/vincent/.ldk-server/` | -| Secrets | `keys_seed` or `keys_mnemonic`, `bitcoin/api_key`, `tls.crt`, `tls.key` | -| gRPC | `65.108.246.14:3536` (TLS, not loopback) | -| P2P | `127.0.0.1:9735` | -| Build checkout | `/home/vincent/src/ldk-server` | -| Install dir | `/home/vincent/.local/bin` | -| Commands | `ldk-server`, `ldk-server-cli`, `ldk-cli` → `ldk-server-cli` | -| Rev file | `/home/vincent/.local/bin/ldk-server.rev` | -| Fork remote | `git@github.com:vincenzopalazzo/ldk-server.git` | -| Upstream | `git@github.com:lightningdevkit/ldk-server.git` | -| Root disk | `/` — build here. `/mnt/HC_Volume_103194752` stays near full; do not build there. | - -`~/.local/bin` is already on `PATH`. +| SSH host | `user@host` | +| Revision | commit, branch, or tag | +| Build checkout | `~/src/ldk-server` | +| Config | path passed to the running process | +| Install dir | a directory on `PATH`, such as `~/.local/bin` | +| gRPC address | the address in the config, not a guessed loopback | +| TLS cert | `/tls.crt` | + +Read the live command line before choosing paths: + +```bash +ssh HOST 'tr "\0" " " < /proc/$(pgrep -n -x ldk-server)/cmdline; echo' +``` + +The first argument after the binary is the config. The storage directory +is in that file. Do not assume systemd. A hand-started process reparents +to PID 1 and will not pick up a new binary on `SIGHUP`. ## Why both binaries -Since `819bed8` the server HMAC is `HMAC-SHA256(api_key, timestamp || body)`. -The July 2026 CLI signs only the timestamp. The header still parses, so -the only error is `Invalid credentials`. The `api_key` file did not change. +Since commit `819bed8` the server HMAC is: -The CLI hex-encodes the 32 raw bytes in `bitcoin/api_key` itself. Do not -pass `-a $(xxd -p api_key)` to a current CLI. That double-encodes the key -and fails the same way. +```text +HMAC-SHA256(api_key, timestamp || raw gRPC body) +``` + +Older CLIs sign only the timestamp. The `x-auth` header still parses, so +the server returns `Invalid credentials`. -Never leave `ldk-cli` pointing at a checkout under -`/mnt/HC_Volume_103194752`. That symlink is how the July CLI survived the -last upgrade. +A current CLI hex-encodes the 32 raw bytes in `/api_key` itself. +Do not pass `-a $(xxd -p api_key)` to that CLI. That double-encodes the +key and fails the same way. + +Install copies. Do not symlink `ldk-cli` or `ldk-server` at +`target/release`. The next build would change one binary and leave the +other. ## Rules -- Ask before restarting. Building and installing does not require a restart. +- Ask before restarting. Building and installing does not require one. - Never delete or overwrite `keys_seed`, `keys_mnemonic`, channel storage, `api_key`, or the TLS files. - If both `keys_seed` and `keys_mnemonic` exist, stop. They are different secrets. Do not choose one. -- Do not `cargo clean` on the volume checkout. +- Do not `cargo clean` a checkout that holds the running binary. - Do not force-push `main`. -- One SSH command per purpose. A `====` banner breaks the remote zsh. - -## 1. Resolve the revision +- One SSH command per purpose. -`` is a commit, branch, or tag. Prefer the fork if the commit is -only there (`feat/keys-seed-compat` and similar). Use upstream for a -stock release. +## 1. Record the running node ```bash -ssh vincent@65.108.246.14 'set -euo pipefail -cd /home/vincent/src/ldk-server -git fetch origin -git fetch https://github.com/lightningdevkit/ldk-server.git main:refs/remotes/upstream/main -git rev-parse --verify "^{commit}" -git log -1 --oneline "" -' -``` - -If `` is not on the fork, fetch the URL or ref that contains it. -Do not guess. - -Record the running identity before changing anything: - -```bash -ssh vincent@65.108.246.14 'set -euo pipefail +ssh HOST 'set -euo pipefail pgrep -a ldk-server -tr "\0" " " < /proc/$(pgrep -n -x ldk-server)/cmdline; echo -sha256sum /proc/$(pgrep -n -x ldk-server)/exe -/home/vincent/.local/bin/ldk-cli -b 65.108.246.14:3536 get-node-info -ls -la /home/vincent/.ldk-server/keys_seed /home/vincent/.ldk-server/keys_mnemonic /home/vincent/.ldk-server/bitcoin/api_key +EXE=$(readlink -f /proc/$(pgrep -n -x ldk-server)/exe) +sha256sum "$EXE" +CLI=$(command -v ldk-cli || command -v ldk-server-cli) +"$CLI" -b GRPC_HOST:PORT get-node-info ' ``` -Save the node id. A successful update must print the same one. +Save the node id. A successful update must print the same one. If the CLI +cannot authenticate against the current server, record that and continue +with the log. Do not rotate `api_key` to make the check pass. + +Also record whether `keys_seed`, `keys_mnemonic`, or both exist, and their +sizes. A 64-byte `keys_seed` is a legacy identity. Replacing it creates a +different node. ## 2. Build both binaries -Build in `/home/vincent/src/ldk-server`, on `/`. Check `df -h /` first. -Need several GB free. `rustc` on this host is 1.95, which is enough for -current `main`. +Check free space on the filesystem that holds the build checkout. A +release build needs several GB. ```bash -ssh vincent@65.108.246.14 'set -euo pipefail -cd /home/vincent/src/ldk-server -git fetch origin -git checkout --detach "" +ssh HOST 'set -euo pipefail +cd BUILD_CHECKOUT +git fetch --tags ORIGIN +git checkout --detach REV cargo build --release -p ldk-server -p ldk-server-cli git rev-parse HEAD sha256sum target/release/ldk-server target/release/ldk-server-cli ' ``` -Detached HEAD is intentional. Do not commit on the server. Do not switch -the operator laptop checkout to do this. +`REV` must resolve before checkout. Detached HEAD is intentional. Do not +commit on the server. -## 3. Install both, atomically +If the revision is only on a fork, fetch that remote. Do not guess which +fork contains it. -Copy the binaries. Do not symlink them back into `target/release`. A -later build would change the CLI under a running server, or the reverse. +## 3. Install both + +Keep the previous pair for rollback, then publish the CLI first and the +server second. `mv` on the same filesystem replaces each name atomically. ```bash -ssh vincent@65.108.246.14 'set -euo pipefail -SRC=/home/vincent/src/ldk-server/target/release -INSTALL=/home/vincent/.local/bin +ssh HOST 'set -euo pipefail +SRC=BUILD_CHECKOUT/target/release +INSTALL=INSTALL_DIR STAGE=$(mktemp -d) install -m 0755 "$SRC/ldk-server" "$STAGE/ldk-server" install -m 0755 "$SRC/ldk-server-cli" "$STAGE/ldk-server-cli" -# publish cli first, then the server binary, then the ldk-cli name +if [ -f "$INSTALL/ldk-server" ]; then + cp -a "$INSTALL/ldk-server" "$INSTALL/ldk-server.prev" +fi +if [ -f "$INSTALL/ldk-server-cli" ]; then + cp -a "$INSTALL/ldk-server-cli" "$INSTALL/ldk-server-cli.prev" +fi mv -f "$STAGE/ldk-server-cli" "$INSTALL/ldk-server-cli" mv -f "$STAGE/ldk-server" "$INSTALL/ldk-server" ln -sfn "$INSTALL/ldk-server-cli" "$INSTALL/ldk-cli" rmdir "$STAGE" -REV=$(git -C /home/vincent/src/ldk-server rev-parse --short=12 HEAD) +REV=$(git -C BUILD_CHECKOUT rev-parse --short=12 HEAD) printf "%s %s\n" "$(date -u +%Y%m%dT%H%M%SZ)" "$REV" > "$INSTALL/ldk-server.rev" -sha256sum "$INSTALL/ldk-server" "$INSTALL/ldk-server-cli" cmp "$INSTALL/ldk-server" "$SRC/ldk-server" cmp "$INSTALL/ldk-server-cli" "$SRC/ldk-server-cli" ' ``` -Stop here unless the operator asked to restart. The old process keeps -running the previous server binary. The new CLI may already fail against -it if the auth scheme changed. Say that, and do not "fix" it by restarting -unasked. +Stop here unless restart was requested. The old process keeps running the +previous server. A new CLI may already fail against it if the auth scheme +changed. Say that. Do not restart to make the CLI work unless asked. ## 4. Restart only when asked -There is no systemd unit. The live process has PPID 1. `SIGHUP` reopens -the log. It does **not** exec a new binary. +`SIGHUP` reopens the log. It does not exec a new binary. + +If a systemd unit actually supervises the process, restart that unit and +skip the manual launch. Confirm with `systemctl status`, not by assuming +the sample unit in `contrib/ldk-server.service` is installed. + +Otherwise: ```bash -ssh vincent@65.108.246.14 'set -euo pipefail +ssh HOST 'set -euo pipefail OLD=$(pgrep -n -x ldk-server) kill -TERM "$OLD" for _ in 1 2 3 4 5 6 7 8 9 10; do @@ -162,64 +169,56 @@ if kill -0 "$OLD" 2>/dev/null; then echo "old process still alive; not sending KILL" >&2 exit 1 fi -nohup /home/vincent/.local/bin/ldk-server /home/vincent/ldk-server-mainnet.toml \ - >> /home/vincent/.ldk-server/ldk-server.stdout 2>&1 & -echo $! > /home/vincent/.ldk-server/ldk-server.pid +nohup INSTALL_DIR/ldk-server CONFIG \ + >> STORAGE_DIR/ldk-server.stdout 2>&1 & +echo $! > STORAGE_DIR/ldk-server.pid disown || true ' ``` -Wait until the log says the gRPC listener is up before calling the CLI. +Wait until the log says the gRPC listener is bound before calling the CLI. If it does not come up, do not start a second copy. ## 5. Prove it ```bash -ssh vincent@65.108.246.14 'set -euo pipefail +ssh HOST 'set -euo pipefail pgrep -a ldk-server -tr "\0" " " < /proc/$(pgrep -n -x ldk-server)/cmdline; echo readlink -f /proc/$(pgrep -n -x ldk-server)/exe -sha256sum /proc/$(pgrep -n -x ldk-server)/exe /home/vincent/.local/bin/ldk-server -/home/vincent/.local/bin/ldk-cli -b 65.108.246.14:3536 get-node-info -/home/vincent/.local/bin/ldk-cli -b 65.108.246.14:3536 list-channels +sha256sum /proc/$(pgrep -n -x ldk-server)/exe INSTALL_DIR/ldk-server +INSTALL_DIR/ldk-cli -b GRPC_HOST:PORT get-node-info +INSTALL_DIR/ldk-cli -b GRPC_HOST:PORT list-channels ' ``` Pass only if all of these are true: -- The process exe is `/home/vincent/.local/bin/ldk-server`. -- Its sha256 matches `ldk-server-cli`'s sibling from the same install. -- `ldk-cli get-node-info` returns the node id recorded in step 1. -- `keys_seed` / `keys_mnemonic` were not replaced. A new `keys_mnemonic` - next to an old `keys_seed` means the process generated a second identity. - Stop and report it. +- The process executable is the installed `ldk-server`, not a leftover + checkout binary. +- Its sha256 matches the installed file. +- `ldk-cli` resolves to the CLI installed in the same step. +- `get-node-info` returns the node id recorded in step 1. +- `keys_seed` and `keys_mnemonic` were not replaced. A new + `keys_mnemonic` next to an old `keys_seed` means the process generated + a second identity. Stop and report it. - `list-channels` still shows the channels that were usable before. -Also check the log for `Loaded node entropy from` and -`Starting ldk-server version`. The version line includes the git commit. +The log line `Starting ldk-server version` includes the git commit. Match +it to `ldk-server.rev`. ## Rollback -Keep the previous pair. Before installing, if those files exist: - -```bash -cp -a ~/.local/bin/ldk-server ~/.local/bin/ldk-server.prev -cp -a ~/.local/bin/ldk-server-cli ~/.local/bin/ldk-server-cli.prev -``` - -Rollback is the same restart, with the `.prev` files moved back into -place, then `ln -sfn` for `ldk-cli`. Never roll the server back and leave -the new CLI, or the reverse. +Move `ldk-server.prev` and `ldk-server-cli.prev` back into place, refresh +the `ldk-cli` symlink, and restart only if asked. Never roll one binary +back and leave the other. Channel storage is not part of the binary rollback. Do not restore an -older sqlite/postgres snapshot over the one the node just wrote. +older sqlite or postgres snapshot over the one the node just wrote. ## Auth failure after an update -If `get-node-info` returns `Invalid credentials`: - -1. Confirm `ldk-cli` resolves to `~/.local/bin/ldk-server-cli`. -2. Confirm that file's mtime matches `~/.local/bin/ldk-server`. +1. Confirm `ldk-cli` is the CLI from the same install as the server. +2. Compare mtimes and the rev file. They must name one commit. 3. Do not hex-encode `api_key` again. 4. If the CLI is older than the server, reinstall the pair from the running commit. Do not rotate `api_key`.