Add structured debug logging to the configuration merge pipeline

[coderabbitai]

Summary

The configuration merge pipeline in src/cli/merge.rs silently accumulates errors and makes no observability available to operators trying to diagnose configuration failures. Adding structured debug logging at each layer boundary would make the merge process auditable.

Context

PR #273 introduced merge_with_config and resolve_merged_diag_json in src/cli/merge.rs. The pipeline traverses four layers (defaults, file, environment, CLI) but emits no log output at discovery or merge time.

Proposed work

• Add tracing::debug! calls in merge_with_config for each layer push (defaults applied, file discovered, environment merged, CLI overrides applied)
• Log config file discovery attempts (path tried, found/not found)
• Log validation rejections with structured fields (key, reason)
• Ensure log output is suppressed when diag_json mode is active (to avoid polluting machine-readable output)

Backlinks

• PR: Complete CLI config struct implementation with OrthoConfig (0.8.0) (3.11.1) #273
• Requested by @leynos

Rescope (2026-08-12)

The discovery half of this work has landed: src/cli/discovery.rs now uses tracing::{debug, debug_span} (emissions at lines 101 and 168) with a dedicated src/cli/discovery_tracing_tests.rs. Remaining scope: src/cli/merge.rs itself still has zero tracing calls — no per-layer logging in merge_with_config (defaults / file / env / CLI) and no structured validation-rejection logging. Note also the rename since filing: resolve_merged_diag_json is now resolve_merged_json, and "diag_json mode" is now the --json flag.

[coderabbitai]

🔗 Related PRs

leynos/weaver#146 - Trace compile YAML pipeline (#115) [merged]
#329 - Instrument config path resolution (#292) [merged]
#330 - Inject config environment access (#293) [merged]
#379 - Add metrics for config-load failures and startup latency (#303) [open]
#548 - Cache config file layer discovery (#319) [open]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[leynos]

@coderabbitai looking at the codebase, is this issue still relevant? If the issue has been fixed in the code, does the documentation reflect this?

If it is still relevant, please provide an assessment of the issue criticality (critical, high, medium or low), based on risk of the issue impacting the user and the anticipated blast radius.\n\nOnce a decision has been made, apply the appropriate label.

If the issue is documentation ONLY (i.e., no code changes needed), apply the documentation label in addition to the appropriate criticality label.

Please also provide an impact of the required changes.

[coderabbitai]

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Fetch issue `#318` details
gh issue view 318 --repo leynos/netsuke --json title,body,labels,state,comments

Length of output: 3997

---

🏁 Script executed:

#!/bin/bash
# Check current state of merge.rs for tracing/logging
cat src/cli/merge.rs

echo "---"
# Check if tracing is used anywhere in src/cli
rg -n "tracing" src/cli/

echo "---"
# Check documentation for logging/observability mentions
fd -e md -e rst -e txt . docs/ 2>/dev/null | head -20
rg -rn "tracing\|debug log\|observabilit" docs/ 2>/dev/null | head -30

Length of output: 1491

---

🏁 Script executed:

#!/bin/bash
# Find merge.rs anywhere in repo
fd merge.rs

echo "---"
# Find any cli directory
fd -t d cli

echo "---"
# Search for merge_with_config function
rg -rn "merge_with_config\|resolve_merged_diag_json" --type rust

echo "---"
# Search for tracing usage in rust files
rg -rn "tracing::" --type rust | head -40

echo "---"
# Check docs for execplans related to merge/logging
fd -e md . docs/execplans/ | xargs grep -l "merge\|log\|tracing" 2>/dev/null

Length of output: 5346

---

🏁 Script executed:

#!/bin/bash
# Read config_merge.rs
cat src/cli/config_merge.rs

echo "=== CLI module files ==="
ls src/cli/

echo "=== tests/cli_tests/merge.rs ==="
cat tests/cli_tests/merge.rs

echo "=== Check main.rs around config loading ==="
cat src/main.rs

echo "=== Search for ndebug/ninfo in cli directory ==="
rg -rn "ndebug\|ninfo\|nwarn\|nerror" src/cli/

Length of output: 29127

---

🏁 Script executed:

#!/bin/bash
# Check available labels on the repo
gh label list --repo leynos/netsuke --limit 50

echo "---"
# Check if ndebug!/ninfo! macros are defined somewhere (to understand the tracing abstraction)
rg -rn "macro_rules! ndebug\|macro_rules! ninfo\|macro_rules! nwarn" --type rust | head -20

Length of output: 1020

---

🏁 Script executed:

#!/bin/bash
gh issue edit 318 --repo leynos/netsuke --add-label "medium"

Length of output: 162

---

---

@leynos, here is the assessment.

---

Issue #318 — Relevance Assessment

The issue is still fully open. Inspection of src/cli/config_merge.rs (the module has been renamed from merge.rs but is otherwise the file described in the issue) confirms zero tracing calls anywhere in src/cli/. Neither ndebug!, ninfo!, nwarn!, nor any tracing:: call exists in that directory.

Status of each proposed item

Proposed work	Status
tracing::debug! in merge_with_config for each layer push	❌ Not implemented
Log config file discovery attempts (path tried, found/not-found)	❌ Not implemented
Log validation rejections with structured fields	❌ Not implemented
Suppress log output in diag_json mode	✅ Implicitly resolved — main.rs does not call init_tracing at all when in JSON mode (configure_runtime gates init_tracing behind !mode.is_json()), so any future debug! calls will naturally be silenced

Three of the four items remain unimplemented. Code changes are required in src/cli/config_merge.rs.

---

Criticality: medium

Risk to user: Low probability of production impact — this is a missing observability feature, not a crash or data-loss defect. However, operators diagnosing configuration failures (wrong file loaded, unexpected layer override, env variable not picked up) have no structured audit trail to follow through the four-layer pipeline (defaults → file → env → CLI).

Blast radius: Confined to developer and operator experience. End users are unaffected unless they are actively debugging a misconfiguration; at that point, the absence of structured debug output forces them to resort to trial-and-error.

---

Impact of required changes

Changes are scoped entirely to src/cli/config_merge.rs:

• Add ndebug! (or tracing::debug!) calls at each composer.push_* site in merge_with_config
• Emit structured fields (e.g. path, source, layer_count) from push_file_layers for explicit-path and discovery branches
• Log validation rejections in cli_overrides_from_matches with key and reason fields
• No documentation changes are needed; diag_json suppression requires no additional code

The label medium has been applied.

[leynos]

@CodeRabbit plan

[coderabbitai]

Coding Plan

Summary

• Add structured tracing::debug! calls to src/cli/config_merge.rs following the existing codebase pattern of key-value fields before the message string
• Log at each layer boundary in merge_with_config and at each discovery/load point in push_file_layers
• Log validation rejections with key and reason fields for structured filtering
• Rely on the existing diag_json suppression mechanism — no changes needed to main.rs or the tracing subscriber setup

Implementation Steps

Add Structured Debug Logging to the Merge Pipeline

This phase adds tracing::debug! calls throughout config_merge.rs to make the merge process auditable. All logging follows the existing codebase pattern of structured key-value fields before the message string.

Task 1: Add Layer Push Logging in merge_with_config

Add debug logging after each layer is successfully pushed to the MergeComposer, providing visibility into which layers are active during a merge operation.

• Import tracing::debug at the top of src/cli/config_merge.rs
• After the defaults layer push (line ~334), log that defaults were applied
• After the environment layer push (line ~349), log that environment variables were merged, including the prefix used (NETSUKE_)
• After the CLI layer push (line ~355), log that CLI overrides were applied; include field count or similar metric if readily available
• Use structured fields: layer = "defaults", layer = "environment", layer = "cli" to enable filtering

Task 2: Add Config File Discovery Logging in push_file_layers

Log config file discovery attempts to make it clear which paths were tried and whether they succeeded.

• In the explicit path branch of push_file_layers, log the resolved path before calling load_layers_from_path, using ?path Debug formatting
• After discovery completes in the auto-discovery branch, log the number of layers found and their source paths (available via MergeLayer::path())
• Log when the project-scope second pass is triggered, including the expected project file path
• Log each successfully loaded layer with its source path

Task 3: Add Validation Rejection Logging

Log validation and loading failures with structured fields containing the key and reason, enabling operators to diagnose configuration problems.

• When sanitize_value(&Cli::default()) fails, log the error before pushing to the errors vector; use error = %err for Display formatting
• When file loading fails in push_layers_result, log the path attempted and the error reason
• When environment extraction fails, log that env layer extraction failed with the error details
• When cli_overrides_from_matches fails, log the validation rejection with key and message fields
• Follow the existing pattern: tracing::debug!(key = %key, reason = %message, "validation rejected")

Task 4: Add Discovery Error Logging

Log discovery errors that are currently silently accumulated, particularly optional errors that get discarded when layers are successfully found.

• Log required_errors from discovery before appending them to the error vector
• Log optional_errors when they are about to be discarded (i.e., when file_layers.value is non-empty)
• Use a lower log level (tracing::trace!) for discarded optional errors if the volume is a concern, otherwise tracing::debug!

🤖 Prompt for AI agents

Add structured `tracing::debug!` (and `tracing::trace!` where appropriate) calls
throughout `src/cli/config_merge.rs` to instrument the full merge pipeline. The
goal is to make every layer push, file discovery attempt, validation rejection,
and silently-discarded error visible via structured log output.

**Import**
- Add `use tracing::debug;` (and `use tracing::trace;` if needed) at the top of
`src/cli/config_merge.rs`

**Layer push logging in `merge_with_config`**
- After the defaults layer push (~line 334), emit a debug log: `layer =
"defaults"`
- After the environment layer push (~line 349), emit a debug log: `layer =
"environment"`, and include the prefix `NETSUKE_`
- After the CLI layer push (~line 355), emit a debug log: `layer = "cli"`;
include a field count or similar metric if readily available
- All structured fields must appear before the message string, following the
existing codebase pattern

**Config file discovery logging in `push_file_layers`**
- In the explicit path branch, log the resolved path before calling
`load_layers_from_path`; use `?path` Debug formatting
- In the auto-discovery branch, after discovery completes, log the number of
layers found and their source paths via `MergeLayer::path()`
- Log when the project-scope second pass is triggered, including the expected
project file path
- Log each successfully loaded layer with its source path

**Validation rejection logging**
- When `sanitize_value(&Cli::default())` fails, log the error before pushing to
the errors vector; use `error = %err` for Display formatting
- When file loading fails in `push_layers_result`, log the path attempted and
the error reason
- When environment extraction fails, log that the env layer extraction failed
with error details
- When `cli_overrides_from_matches` fails, log with `key` and `reason` fields:
`tracing::debug!(key = %key, reason = %message, "validation rejected")`

**Discovery error logging**
- Log `required_errors` from discovery before appending them to the error vector
- Log `optional_errors` when they are about to be discarded (i.e., when
`file_layers.value` is non-empty)
- Use `tracing::trace!` for discarded optional errors if volume is a concern;
otherwise use `tracing::debug!`

**Do not** introduce any changes to `main.rs` or the tracing subscriber setup —
rely entirely on the existing `diag_json` suppression mechanism.

Research

The configuration merge pipeline lives in src/cli/config_merge.rs (note: the ticket references src/cli/merge.rs but the actual file is config_merge.rs). The merge_with_config function implements a four-layer merge: defaults → file(s) → environment → CLI. Errors are accumulated into a Vec<Arc<OrthoError>> rather than failing fast. Tracing is already configured to be suppressed when diag_json mode is active — the subscriber is never initialised in JSON mode — so no additional suppression logic is required.

---

🚀 Next Steps

🤖 All AI agent prompts combined

Add structured `tracing::debug!` (and `tracing::trace!` where appropriate) calls
throughout `src/cli/config_merge.rs` to instrument the full merge pipeline. The
goal is to make every layer push, file discovery attempt, validation rejection,
and silently-discarded error visible via structured log output.

**Import**
- Add `use tracing::debug;` (and `use tracing::trace;` if needed) at the top of
`src/cli/config_merge.rs`

**Layer push logging in `merge_with_config`**
- After the defaults layer push (~line 334), emit a debug log: `layer =
"defaults"`
- After the environment layer push (~line 349), emit a debug log: `layer =
"environment"`, and include the prefix `NETSUKE_`
- After the CLI layer push (~line 355), emit a debug log: `layer = "cli"`;
include a field count or similar metric if readily available
- All structured fields must appear before the message string, following the
existing codebase pattern

**Config file discovery logging in `push_file_layers`**
- In the explicit path branch, log the resolved path before calling
`load_layers_from_path`; use `?path` Debug formatting
- In the auto-discovery branch, after discovery completes, log the number of
layers found and their source paths via `MergeLayer::path()`
- Log when the project-scope second pass is triggered, including the expected
project file path
- Log each successfully loaded layer with its source path

**Validation rejection logging**
- When `sanitize_value(&Cli::default())` fails, log the error before pushing to
the errors vector; use `error = %err` for Display formatting
- When file loading fails in `push_layers_result`, log the path attempted and
the error reason
- When environment extraction fails, log that the env layer extraction failed
with error details
- When `cli_overrides_from_matches` fails, log with `key` and `reason` fields:
`tracing::debug!(key = %key, reason = %message, "validation rejected")`

**Discovery error logging**
- Log `required_errors` from discovery before appending them to the error vector
- Log `optional_errors` when they are about to be discarded (i.e., when
`file_layers.value` is non-empty)
- Use `tracing::trace!` for discarded optional errors if volume is a concern;
otherwise use `tracing::debug!`

**Do not** introduce any changes to `main.rs` or the tracing subscriber setup —
rely entirely on the existing `diag_json` suppression mechanism.

💡 Iterate on the plan with: `@coderabbitai` <feedback>

Example Feedback
- `@coderabbitai` You can skip phase 3. Add a simple unit test case for phase 2.
- `@coderabbitai` For design choice 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord!

[leynos]

Important

Constraint added 2026-08-01: no in-process environment mutation.

AGENTS.md now forbids in-process environment mutation in tests and disqualifies process-wide locks as a fallback.

The merge pipeline this issue instruments includes an environment layer, and asserting the debug output for that layer requires controlling NETSUKE_CONFIG and the environment-sourced values. src/cli/discovery.rs:111 offers no injection point.

This issue is blocked on #483, which threads &impl mockable::Env through discovery and the layer-collection functions.

Sequence this after #483 and alongside #292, which instruments the adjacent discovery functions — the two touch overlapping call sites and are cheaper to land together than consecutively.

[leynos]

Rescoped (see the dated section appended to the description): the discovery half has landed on main (discovery.rs tracing plus dedicated tests); remaining scope is src/cli/merge.rs itself, which still has zero tracing calls. Also noted the diag_json → --json rename.