feat(deployment): add automated rollback and disaster recovery manage…

[lukawisdom06]

Summary

This PR implements an automated rollback and disaster recovery framework for model deployments to improve system reliability and minimize downtime during deployment failures. It introduces state snapshot management, rollback orchestration, recovery procedures, and supporting documentation.

Changes

• Added automated rollback triggers for failed model deployments.
• Implemented state snapshot creation and management to enable safe rollback operations.
• Added disaster recovery procedures for restoring deployments from snapshots.
• Introduced recovery testing automation to validate rollback and recovery workflows.
• Added rollback approval workflow support for controlled recovery operations.
• Implemented Recovery Time Objective (RTO) tracking to monitor recovery performance.
• Updated disaster recovery documentation with recovery procedures and operational guidance.

Testing

• Added unit tests covering rollback, snapshot management, and recovery workflows.
• Simulated deployment failures to verify automated rollback behavior.
• Validated disaster recovery scenarios and state restoration.
• Confirmed all tests pass with pytest tests/.
• Verified linting passes with pre-commit run --all-files.

Documentation

• Added and updated docs/disaster-recovery.md with rollback procedures, recovery workflow, and operational best practices.

Related Issue

Closes #651
Closes #650
Closes #649
Closes #648

[github-actions]

📏 Pull request size warning

This pull request exceeds the review-friendly size limits. CI is not blocked — this is guidance, not a gate.

Metric	This PR	Limit
Lines changed	1015	1000
Files changed	9	10

Why it matters

Review quality drops sharply with diff size: defects slip through, review turnaround grows, and reverts become riskier because unrelated changes are bundled together.

What to do

• Split the change into a stack of smaller, independently reviewable PRs.
• Land refactors and behaviour changes separately.
• Hide incomplete work behind a feature flag and merge it incrementally.
• Move generated files, lockfiles, or vendored code into their own PR.

Legitimately large?

• Add the refactor:large label to raise the file ceiling to 50 files.
• Add [large PR] to the title to skip this check entirely.

Breached limits

• 1015 lines changed (+1015/-0) exceeds the 1000-line limit.

[drips-wave]

@lukawisdom06 Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits