[ISSUE #9865]✨Complete remoting V2 security, observability, and lifecycle contracts

[mxsm]

Which Issue(s) This PR Fixes(Closes)

• Fixes [Enhancement✨] Complete remoting V2 security, observability, and lifecycle contracts #9865

Brief Description

Complete the remoting V2 security, observability, and lifecycle contracts across transport, broker maintenance handling, runtime shutdown reporting, and the observability metric catalog.

• Authenticate network maintenance requests from the canonical V2 request origin and keep embedded broker proxy identity explicit.
• Add body-free response observation contracts, bounded remoting metrics, and exactly-once coverage for direct, queued, legacy, and deferred response paths.
• Make connection retirement server-owned and ordered, expose narrow push/request/close capabilities, isolate listener panics, and report incomplete shutdown cleanup as typed failures.
• Update the V2 public API contract, migration ledger, metric catalog, and focused security/lifecycle regression coverage.

How Did You Test This Change?

• cargo fmt -p rocketmq-transport -p rocketmq-runtime -p rocketmq-broker -- --check - passed.
• cargo clippy --workspace --no-deps --all-targets --all-features -- -D warnings - passed.
• cargo clippy -p rocketmq-transport -p rocketmq-runtime -p rocketmq-broker --all-targets --all-features -- -D warnings - passed.
• cargo test -p rocketmq-transport - passed, including 671 library tests, integration tests, and 95 doc tests.
• Repeated the ordered deferred owner-cutoff regression 100 times - passed 100/100.
• cargo test -p rocketmq-transport --lib --features observability - passed, 671/671.
• Broker maintenance V2 authentication and embedded identity focused tests - passed, 5/5.
• RocketMQ observability Clippy and unit-test matrix for default, metrics, Prometheus, traces, logs, and combined features - passed; the package-level architecture guard reports the same pre-existing subscriber site on this branch and main.
• python scripts/generate_metric_catalog.py --check - passed.
• cargo doc -p rocketmq-transport --no-deps - passed with one pre-existing private-link warning in tls.rs.
• .\scripts\runtime-audit.ps1 -SkipBaseline -EnforceBoundaryBaseline - passed.
• .\scripts\check-error-hygiene.ps1 - reports the same pre-existing findings as main; this change adds no finding.
• git diff --check - passed.

Summary by CodeRabbit

• New Features

  • Added detailed transport metrics for request outcomes, response handling, deferred requests, queue wait times, duplicates, and abandoned responses.
  • Added V2 response observation data for monitoring request and response lifecycles.
  • Improved shutdown reporting with task, waiter, writer, and cleanup health details.

• Bug Fixes

  • Strengthened maintenance-request authorization and rejected forged or unsupported requests.
  • Improved session-close handling, including timeout detection, cleanup failures, and listener panics.

• Documentation

  • Added V1-to-V2 migration guidance and marked legacy transport APIs as deprecated, with recommended replacements.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 4665bffa-4247-4c3b-977a-6b5b7cb6e3cf

📥 Commits

Reviewing files that changed from the base of the PR and between b4b5f96 and c40c97d.

⛔ Files ignored due to path filters (1)

• rocketmq-observability/tests/fixtures/metric_catalog_descriptors.tsv is excluded by !**/*.tsv

📒 Files selected for processing (57)

• rocketmq-broker/src/processor/maintenance_request_processor.rs
• rocketmq-doc/en/remoting-processor-v2-migration-ledger.md
• rocketmq-observability/src/metrics/catalog/generated.rs
• rocketmq-observability/src/metrics/catalog/schema/rust/catalog.toml
• rocketmq-observability/src/metrics/catalog/tests.rs
• rocketmq-observability/src/metrics/remoting.rs
• rocketmq-observability/src/metrics/tests/remoting_metric_isolation.rs
• rocketmq-observability/src/semantic.rs
• rocketmq-observability/tests/metric_catalog_contract.rs
• rocketmq-runtime/src/shutdown_report.rs
• rocketmq-transport/src/clients/client.rs
• rocketmq-transport/src/clients/rocketmq_tokio_client/api.rs
• rocketmq-transport/src/connection.rs
• rocketmq-transport/src/connection/issue_9754_tests.rs
• rocketmq-transport/src/dispatch.rs
• rocketmq-transport/src/dispatch/authorized_dispatcher.rs
• rocketmq-transport/src/dispatch/authorized_dispatcher/v2.rs
• rocketmq-transport/src/dispatch/authorized_dispatcher/v2/tests/boundary_identity.rs
• rocketmq-transport/src/dispatch/authorized_dispatcher/v2/tests/harness.rs
• rocketmq-transport/src/dispatch/authorized_dispatcher/v2/tests/legacy_adapter.rs
• rocketmq-transport/src/dispatch/authorized_dispatcher/v2/tests/outcomes_deadlines.rs
• rocketmq-transport/src/dispatch/deferred_registry/internal.rs
• rocketmq-transport/src/dispatch/deferred_registry/tests.rs
• rocketmq-transport/src/dispatch/deferred_responder.rs
• rocketmq-transport/src/dispatch/deferred_response.rs
• rocketmq-transport/src/dispatch/deferred_resume.rs
• rocketmq-transport/src/dispatch/deferred_resume/terminal_ownership_tests.rs
• rocketmq-transport/src/dispatch/deferred_resume/tests/stop.rs
• rocketmq-transport/src/dispatch/deferred_session_cleanup.rs
• rocketmq-transport/src/dispatch/legacy_processor_adapter.rs
• rocketmq-transport/src/dispatch/legacy_processor_adapter/tests.rs
• rocketmq-transport/src/dispatch/remoting_request/builder.rs
• rocketmq-transport/src/dispatch/request_control.rs
• rocketmq-transport/src/dispatch/response.rs
• rocketmq-transport/src/dispatch/response_plan.rs
• rocketmq-transport/src/dispatch/response_sink.rs
• rocketmq-transport/src/dispatch/response_sink/plan.rs
• rocketmq-transport/src/prelude.rs
• rocketmq-transport/src/public_api.rs
• rocketmq-transport/src/public_api_v2.rs
• rocketmq-transport/src/remoting.rs
• rocketmq-transport/src/remoting_server/rocketmq_tokio_server/connection_handler.rs
• rocketmq-transport/src/remoting_server/rocketmq_tokio_server/v2_server/tests.rs
• rocketmq-transport/src/runtime/connection_handler_context.rs
• rocketmq-transport/src/runtime/processor_v2.rs
• rocketmq-transport/src/server.rs
• rocketmq-transport/src/session_executor.rs
• rocketmq-transport/src/telemetry.rs
• rocketmq-transport/src/v2_session_registry.rs
• rocketmq-transport/src/write_strategy.rs
• rocketmq-transport/src/writer_runtime.rs
• rocketmq-transport/src/writer_runtime/issue_9754_tests.rs
• rocketmq-transport/tests/authorized_dispatch_conformance_tests.rs
• rocketmq-transport/tests/public_api_contract.rs
• rocketmq-transport/tests/public_api_v1.rs
• rocketmq-transport/tests/v1_processor_contract_tests.rs
• scripts/generate_metric_catalog.py

---

Walkthrough

The pull request completes remoting V2 security, response observation, transport metrics, session shutdown reporting, API deprecation, and migration documentation. It adds end-to-end authorization tests and expands lifecycle, telemetry, queue-wait, and public API contract coverage.

Changes

Security and request ingress

Layer / File(s)	Summary
Trusted maintenance authorization rocketmq-broker/src/processor/maintenance_request_processor.rs	Authorization now distinguishes network and embedded requests. Network requests use verified credentials and channel context. Embedded requests use the authenticated principal. Other origins are rejected.
Legacy channel boundary rocketmq-transport/src/runtime/connection_handler_context.rs, rocketmq-transport/src/remoting.rs, rocketmq-transport/src/remoting_server/..., rocketmq-transport/src/dispatch/legacy_processor_adapter.rs	Internal V1 paths use legacy_channel(). The general channel() and legacy response methods are deprecated.
V2 request contracts rocketmq-transport/src/dispatch/remoting_request/builder.rs, rocketmq-transport/src/dispatch/request_control.rs	Request builders expose received time, origin, and authentication. Boundary responses use control without the request deadline.

V2 response observation

Layer / File(s)	Summary
Observation types and processor hook rocketmq-transport/src/runtime/processor_v2.rs, rocketmq-transport/src/public_api_v2.rs	V2 adds typed response metadata and terminal outcomes. Processors can receive full response observations while retaining write projections.
Dispatch and deferred response flow rocketmq-transport/src/dispatch/authorized_dispatcher/v2.rs, rocketmq-transport/src/dispatch/legacy_processor_adapter.rs, rocketmq-transport/src/dispatch/deferred_responder.rs, rocketmq-transport/src/dispatch/deferred_resume.rs	Dispatch carries observations through inline, deferred, rejected, cancelled, failed, and legacy response paths. Completion is recorded once. Request spans are restored for deferred resume work.
Observation validation rocketmq-transport/src/dispatch/authorized_dispatcher/v2/tests/*, rocketmq-transport/src/dispatch/deferred_resume/terminal_ownership_tests.rs	Tests cover boundary rejection, legacy response ambiguity, deferred completion, cancellation, processor ownership, and span re-entry.

Session lifecycle

Layer / File(s)	Summary
Close report and task draining rocketmq-transport/src/server.rs, rocketmq-transport/src/session_executor.rs, rocketmq-runtime/src/shutdown_report.rs	Session close now reports cleanup, task, writer, lease, panic, timeout, and failure state. Inline and resume tasks have separate counters.
Deferred cleanup accounting rocketmq-transport/src/dispatch/deferred_session_cleanup.rs, rocketmq-transport/src/dispatch/deferred_registry/internal.rs	Cleanup reports registered and removed waiters, remaining permits, and panicking targets. Cleanup targets return removed waiter counts.
Session and registry integration rocketmq-transport/src/v2_session_registry.rs, rocketmq-transport/src/remoting_server/rocketmq_tokio_server/connection_handler.rs, rocketmq-transport/src/clients/client.rs	Close requests wait for full completion. Server outbound leases prevent new pushes and requests after close begins. Registration rolls back after connect-listener panics.
Lifecycle tests rocketmq-transport/src/remoting_server/rocketmq_tokio_server/v2_server/tests.rs, rocketmq-transport/src/dispatch/deferred_resume.rs	Tests cover concurrent close, listener panics, deferred cleanup, writer completion, task draining, and pending server requests.

Transport metrics

Layer / File(s)	Summary
Metric schema and catalog rocketmq-observability/src/metrics/catalog/schema/rust/catalog.toml, rocketmq-observability/src/metrics/catalog/generated.rs, scripts/generate_metric_catalog.py	Request labels and seven remoting metrics were added. Generated label indexes and catalog counts were updated.
Metric recording API rocketmq-observability/src/metrics/remoting.rs, rocketmq-observability/src/semantic.rs	Fixed request, outcome, response mode, result, code, and abandonment labels are recorded for request and response lifecycle events.
Response queue wait rocketmq-transport/src/write_strategy.rs, rocketmq-transport/src/connection.rs, rocketmq-transport/src/writer_runtime.rs	Queued writes identify response traffic. The writer records queue wait only for eligible responses.
Metric validation rocketmq-observability/src/metrics/tests/remoting_metric_isolation.rs, rocketmq-observability/src/metrics/catalog/tests.rs, rocketmq-observability/tests/metric_catalog_contract.rs, rocketmq-transport/src/connection/issue_9754_tests.rs, rocketmq-transport/src/writer_runtime/issue_9754_tests.rs	Tests validate classified metrics, signed deferred gauges, response outcomes, queue wait, and catalog totals.

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant V2ConnectionHandler
  participant AuthorizedCommandDispatcherV2
  participant V2RequestObservation
  participant SessionHandle
  participant SessionExecutor
  participant SessionWriter
  Client->>V2ConnectionHandler: send authenticated request
  V2ConnectionHandler->>AuthorizedCommandDispatcherV2: dispatch RemotingRequest
  AuthorizedCommandDispatcherV2->>V2RequestObservation: start and classify request
  AuthorizedCommandDispatcherV2->>V2RequestObservation: complete response or deferred registration
  V2RequestObservation-->>Client: response observation and response write
  SessionHandle->>SessionHandle: request close
  SessionHandle->>SessionExecutor: drain inline and resume tasks
  SessionHandle->>SessionWriter: retire writer
  SessionHandle-->>Client: publish close completion

Loading

Poem

A rabbit checks the guarded gate

Signed requests arrive in order and state
Metrics bloom with labels bright
Deferred paths return one result right
Tasks drain and writers sleep
Clean close reports their count to keep

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch mxsm/issue-9865-xct

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[rocketmq-rust-bot]

🔊@mxsm 🚀Thanks for your contribution🎉！

💡CodeRabbit(AI) will review your code first🔥！

Note

🚨The code review suggestions from CodeRabbit are to be used as a reference only, and the PR submitter can decide whether to make changes based on their own judgment. Ultimately, the project management personnel will conduct the final code review💥.