fix(0141): build and verify the Lambdas before every deploy - #325
Merged
Merged
Conversation
`make deploy-production-compute` compiled the CDK TypeScript only and zipped whatever sat in target/lambda/. On 2026-08-03 that shipped a stale prices-api behind a clean diff and a green deploy; on 2026-08-12 the same directory held eleven 10-byte `#!/bin/sh` stubs. Every target that can ship a Lambda (deploy-production, -compute, -eventbridge) now depends on `build-lambdas`. Cargo decides freshness — a 1 s no-op when the artifacts are current — and each bootstrap is then refused unless it is an executable aarch64 ELF distinct from every other asset's. CI runs the same script instead of its own inline copy. Every per-stack deploy target passes --exclusively, so a deploy of ApiGateway can no longer deploy Compute as a side effect.
The 0072 runbook's step 6 records the Rust-build prerequisite and warns that "heals the 0132 drift" is only true from a current tree; step 7 and the infra README say why GET /health — a gateway mock — proves nothing about a deploy. The ledger-processor runbook builds through `make build-lambdas` rather than a hand-typed single-crate cargo command. The task records the four live runs behind its acceptance criteria, what was not verified (a real `cdk deploy --exclusively`, the changed CI steps), and that the first Compute deploy may re-hash every Lambda in the stack because the build is now the group build CI uses.
`node --test tools/scripts/` only works on newer Node: on the 22.22.0 CI uses, a directory argument is loaded as a module and the step fails. The tests now run from a glob, as the infra project's Nx `test` target, so CI's `nx run-many` and the verify:staged / verify:push hooks run them on the pinned Node. The guard could still vouch for files nobody ships. With an `*_ASSET_DIR` override exported, CDK packages that directory instead; with CARGO_TARGET_DIR or build.target-dir set, cargo writes the fresh bootstraps elsewhere and the old ones under target/lambda/ verify. Both are now refused, the second before anything is built. `diff-production` builds the Lambdas first, so the diff an operator approves is of the artifacts the deploy ships. The branch and dirty state are printed again after verification, directly above `cdk deploy`. The Makefile tests no longer go blind if the asset-dir literals move out of the stack files, and no longer run `make` at import.
The ledger-processor runbook no longer recommends LEDGER_PROCESSOR_ASSET_DIR, and says that a change spanning Compute and ApiGateway needs Compute deployed first now that neither target deploys the other. The CI wiki page footnotes the two steps that became one. The task carries all ten findings with their outcomes, including the one kept as is: --exclusively on deploy-production-apigateway.
Collaborator
ReviewNo correctness bugs found. One low-severity risk. The new tests ( Finding
Every Lambda deploy target and So after a routine
It fails loudly and ships nothing, but no Lambda can be deployed until someone rolls the compiler back. A Checked and found correct
|
adamkoot
added a commit
that referenced
this pull request
Sep 21, 2026
0141's target globs tools/scripts/**/*.test.mjs, so since it merged it already runs these too: 30 cases in one step, 12 + 18, on the pinned Node 22.22.0. The separate step was doing the second half twice. This is decision 13's follow-up, which #325 landing unblocked.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
make deploy-production,-compute,-eventbridgeanddiff-productionnow depend onmake build-lambdas: onecargo lambda buildover every asset the CDK app references (cargo decides freshness — ~1 s when current), followed by a check that eachtarget/lambda/<name>/bootstrapis an executable aarch64 ELF distinct from every other asset's. Before this, the deploy zipped whatever was on disk: a staleprices-apireached production on 2026-08-03, and on 2026-08-12 the directory held eleven 10-byte#!/bin/shstubs.*_ASSET_DIRoverride the CDK reads is set, or if cargo's target dir is not the one CDK packages.deploy-production-*target passes--exclusively, so deploying ApiGateway no longer deploys Compute as a side effect. A change spanning both stacks needs Compute first — noted at the target and in the runbook.node:testguards run as the infra project's Nxtesttarget (CI +verify:staged/verify:push).deploy-ledger-processor.md),infra/README.mdand the CI wiki page updated;GET /healthis documented as a gateway mock that proves nothing about a deploy.Verified locally by staling and stubbing real artifacts (results table in the task file). Not verified: a real
cdk deploy --exclusivelyagainst the account — the first operator deploy is that check, and it may show a new S3Key for every Lambda in the stack, since the build is now the group build CI uses. The task staysactiveuntil then.