fix(0204): alarm descriptions must fit CloudWatch's 1024-char limit - #229
Merged
Conversation
The first production deploy failed mid-flight. Three usd-stranded rungs carried ~1250-character descriptions and CloudWatch rejected them after other alarms in the stack had already been created. Nothing local catches this. cdk synth renders an over-long description happily, the template is valid CloudFormation, cdk diff showed the expected 12 additions and 0 deletions, and CI was green — the limit lives only in the CloudWatch API. So the guard has to run at synth: a construct-tree walk that throws with the offending alarm names and lengths, reading the resolved property so token-built descriptions are measured as CloudWatch sees them. The descriptions are shortened to <= 937 chars, and the text that pushed them over was text this same day falsified — the "priced by 30 h of age" baseline and the "check the two-hop chain rather than assuming the sweep" instruction. Both hops were measured healthy while the leg was dark, so that guidance pointed a responder at a dead end. The rewrite names the real cause, says the alarm stays latched until 0209 is fixed, and warns to verify on _1m rather than a coarse tier. Long runbook-style descriptions are worth keeping — an operator reading Slack at 03:00 has nothing else — which is why this needs a guard rather than discipline. Guard verified by inducing it: padding a description to 2533 chars fails synth with all three names; restoring yields a byte-identical template.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What happened
The first production deploy of the 0204 alarms failed mid-flight. Three
usd-strandedrungs carried ~1250-character descriptions; CloudWatch capsAlarmDescriptionat 1024, and rejected them after other alarms in the stack had already been created.Why nothing caught it
cdk synthcdk diffThe limit is enforced only by the service. This is the task's own acceptance-criterion lesson — verified by inducing the condition, not by reading the CDK — landing on the deploy itself.
Fixes
1. Descriptions shortened to ≤ 937 chars.⚠️ The text that pushed them over was text the same day's investigation had falsified: the "MEASURED BASELINE … priced by 30 h of age" claim and the "check the TWO-HOP chain rather than assuming the sweep" instruction. Both hops were measured healthy while the leg was dark, so that guidance pointed a responder at a dead end. The rewrite names the real cause (the USDT pivot has never priced a
_1mrow), states the alarm stays latched until 0209 is fixed, and warns to verify on_1mrather than a coarse tier.2.
assertAlarmDescriptionsFitCloudWatch()runs at synth — walks the construct tree, throws with offending alarm names and lengths, and reads the resolved CloudFormation property so token-built descriptions are measured as CloudWatch will see them.These alarms carry deliberately long runbook-style descriptions because an operator reading Slack at 03:00 has nothing else. That's worth keeping, which is precisely why the ceiling needs a guard rather than discipline.
Verification
Refs: task 0204, and 0209 / 0212 for the condition the rewritten description now points at.