[Zxf-xufeng]

Which issue or RFC does this PR close?

Follow-up to #1945 and #2243. No issue is automatically closed.

Rationale for this change

OceanBase CI repeats general regressions, repeatedly initializes full Server schemas, and can terminate before collecting failure evidence or saving caches. This change reduces duplicate work and gives each bounded test command time to finish and preserve diagnostics.

What changes are included in this PR?

• Run ordinary E2E coverage once, with Hook cases in their separate serial pass. Reserve setup and cache time for every supported Python version.
• Use two duration-balanced OceanBase migration workers with exactly-once coverage checks. Keep migration recovery cases isolated; reuse only the schema for search authorization scenarios, with separate Scopes, idempotency keys and Server lifecycles.
• Bound cold image pulls, container startup, SQL readiness probes and cleanup. Preserve lock, transaction, resource and lifecycle evidence before deadlines without exposing SQL text or row keys.
• Cover slow passing cases and teardown within explicit budgets: 360 seconds per clean-migration case, a 160-minute clean step, a 40-minute long-query step and a 280-minute migration job. The 240 minutes of test-step ceilings leave 40 minutes for setup and evidence. Preserve both legacy and native FTS coverage.
• Retain matching native acceptance JUnit evidence independently of replay. Reuse task-image Python during ACP bootstrap, pin its SDK/uv dependencies and save bootstrap logs.
• Cache the pinned license checker, reduce preflight checkout transfers, and provide a bounded, checksum-verified mise fallback shared with Desktop Linux.
• Wait for the matching OTLP failure state; accommodate measured DSH/background startup latency; preserve OpenDAL projection fields and use explicit synthetic compatibility evidence in desktop lifecycle tests. Production compatibility declarations remain unchanged.
• Download pinned GitHub Release inputs through a bounded official API fallback when the release-download route fails. Both paths retain the locked SHA256 and share one total deadline. The same helper prepares pinned skill-up and Windows NSIS tools before their consumers run.

Are there any user-facing changes?

No public API or persisted-format changes. OpenDAL's projection behavior is preserved with a type-safe validation call. Other changes affect CI, diagnostics and test fixtures.

How was this change tested?

• make check: passed after merging master 7fef43d6.
• Managed-resource acquisition and resource-store tests: 62 passed, 6 native-platform skips. The fallback tests retain digest rejection and the overall download budget; a local live archive download hit the local network deadline, so native download validation remains with CI.
• Master subscription/SDK integration regressions: 516 passed; SQLite authorization: 20 passed.
• SQL scoring and native FTS capability tests: 40 passed. The scoring fixture implements the synchronous connection adapter required by capability probing, retaining all score/order assertions.
• Shared download/resource and CI workflow/path regressions: 220 passed, 8 platform-dependent skips.
• Post-merge CI and SQLite search authorization selection: 175 passed, 2 skipped, 48 deselected.
• ACP bootstrap tests: 2 passed, including preserved installation-failure evidence.
• DSH real-server E2E: 8 passed. OpenDAL connector tests: 10 passed using verified wheels in an isolated temporary dependency directory; the full make opendal-test bootstrap hit a PyPI timeout. Workflow action pins and diff checks passed.
• Both PR branches merge with master and with each other locally. Full CI on the pushed head is pending; native engine acceptance is not yet confirmed. Local Rust dependency downloads were blocked by network timeouts, so desktop tests require CI validation.

AI usage statement

OpenAI Codex (GPT-6) assisted with analysis, implementation, regression tests and PR preparation.

[hidb4ai]

Merge readiness blocker: this head conflicts with current master (9697d5f264adfafd9f7b15f19e1ca6b0176ab8c6) in .github/workflows/e2e-harness.yml, at Maintain the registered Server chain on real OceanBase without skips. git merge-tree --write-tree d8492f56 9697d5f2 reproduces the content conflict: this PR adds the diagnostic wrapper with a 600-second deadline, while master extends that maintenance step to 900 seconds for fixture cleanup and JUnit output. Please reconcile the wrapper with the updated maintenance timeout and rerun the affected acceptance checks on the resolved commit. The current successful CI run does not validate that integration.

[Zxf-xufeng]

Addressed the merge-readiness review in 08a34940.

Merged the latest fetched master (ed8902b2) and resolved the maintenance-step conflict by keeping the diagnostic wrapper with master's 900-second timeout. The five maintenance cases and the no-skips/JUnit checks remain intact.

Also fixed the diagnostic lifecycle: the native snapshot is collected before replay, and failed OceanBase replay collects a separate bounded snapshot before compose down. Diagnostic failures preserve the original replay exit status and do not prevent teardown.

Local validation passed: make check, actionlint for the affected workflows, and the focused CI/launcher regression suites (289 passed, 15 skipped for platform/external-runtime conditions). The launcher regression uses a Docker substitute to verify evidence preservation and cleanup; it does not establish real OceanBase acceptance. The pushed merge commit triggers a fresh CI run, which must validate the integrated native acceptance before merge.

[AlexStocks]

[P2] 分片 3→2 之后，这一步的 90 分钟上限不再覆盖逐用例超时之和

同一个 diff 里 --test-timeout 300 保持不变，但每片用例数从 16 增到 24，最坏路径从 16 × (300 + 10s kill-after) = 82.7 分钟（小于 90 分钟上限）变成 24 × 310s = 124 分钟（大于 90 分钟上限）。新注释把推导从「16 个用例 × 300s」改成「实测约 21m per worker」，等于用均值替换了原来的最坏情况保证。

数字来自本 PR 新增的 .github/oceanbase-migration-durations.json：SUITES 三个文件共 47 个用例、实测合计 2543.7s（42.4 分钟），二分片后每片 24 个、平均 21.2 分钟——注释里的「21m」是准的，但它是最坏情况之外的另一种口径。

影响：本 PR 要治的失败模式正是 OceanBase 启动/迁移阶段的挂起。同片内多个用例挂起时，scripts/oceanbase_ci.py run 的 wait(timeout - min(30, timeout/2)) → collect(budget=20) → execution.json / before-timeout/ 这条取证路径还没轮到，步骤就先被 90 分钟上限杀掉，失败归因退化成「步骤超时」而不是带诊断的逐用例超时。同一个作业里其它三步仍然自洽（background-metrics：1 个用例 / 10 分钟；startup-snapshot：最多 2 个 × 310s + 180s = 13.3 分钟 / 15 分钟；long-query：4 个 × 310s + 180s = 23.7 分钟 / 30 分钟），只有这一处被打破。

建议二选一：

• 提高该步骤上限到 ≥ 130 分钟，与 24 × (300 + 10s) 对齐；或
• 把 --test-timeout 降到 200s（24 × 210s = 84 分钟 ≤ 90 分钟），并同步改写注释里的推导。

如果确实有意只按实测均值设上限，请在注释里写明这一点，否则后续维护者会继续按「上限 ≥ 逐用例超时之和」读这组数字。

[Zxf-xufeng]

Fixed in ad6b9b5b, with cold-start headroom completed in da11dcd4. The duration-based assignment produces 22 and 25 clean-migration cases. The larger shard needs up to 25 × (300 + 10) + (180 + 10) = 7,940s (132m 20s), including collection and termination grace. The step now allows 140 minutes. The job allows 250 minutes: 210 minutes of test-step budgets plus 40 minutes for a separately bounded cold image pull, startup, diagnostics and upload. The per-case deadline remains 300 seconds.

The regression check fails with the original 90-minute step and passes with the corrected budget. The runner still stops at the first failed case; the budget also covers slow passing cases before that failure. This changes ceilings, not the number of executions or retries. A separate cold-pull regression reproduces the previous 180-second startup failure and verifies warm-cache reuse and failure propagation.

The same changes are included in #2526. Both preceding runs passed all Python 3.11–3.14 unit/E2E jobs, but full backend CI is not yet green. Native logs from the separate background-metrics case show successive full-text index DDL still progressing at 277 seconds, before listener checks. That full-Server lifecycle now has a bounded 600-second deadline and a 15-minute step (including collection/grace); ordinary clean-migration cases retain 300 seconds. Lifecycle checkpoints are preserved, assertions remain intact, and no retry is added. This addresses premature termination of observed progress; the underlying engine latency and final pass still require CI verification. Final native acceptance is pending the new CI runs.

[Zxf-xufeng]

The budget fix and latest master conflict resolution are pushed in 9636ae2. The larger clean-migration shard has 25 cases and a 360-second per-case ceiling. Including termination and collection, 25 × (360 + 10) + 190 = 9,440s fits the 160-minute step. The 13 long-query cases are assigned 6/7 and fit the 40-minute step. The 280-minute job covers 240 minutes of test-step ceilings plus 40 minutes for bounded setup, diagnostics and upload. No test retry or duplicate execution is added.

Both branches include master 7fef43d, pass make check, and combine locally without conflicts. Python jobs retain the 60-minute ceiling and cleanup reserve; master's website runner change is preserved. The resource producer has a checksum-preserving official Release API fallback for the observed HTTP 504 download failures.

The previous head passed all Python 3.11–3.14 unit/E2E jobs. Its downstream validation was incomplete when the conflict-resolution push superseded it. Run 42573624133 validates this new head; real-backend acceptance remains pending.

The same bounded, checksum-verified downloader now covers skill-up and pre-staged Windows NSIS tools. The plugin’s ordinary release URL reproduced HTTP 504 locally, while the official asset API returned bytes matching both the pinned SHA256 and Tauri’s upstream SHA1. This verifies the alternate artifact route; the native Windows installer and backend suites still require the linked CI run.

[AlexStocks]

[P2] long-query 步骤的 40 分钟上限同样只对当前基线成立，而该基线缺 5 个已收集的用例

clean-migration 这一条已核实（exact head c63f9b42，e2e-harness.yml / durations.json / test_oceanbase_ci.py 与 1742c5fb 逐 blob 相同）：--test-timeout 360、步骤 160 分钟、作业 280 分钟；-k oceanbase --collect-only 收集 47 个节点，assigned_nodes 打包为 22/25，25 × (360+10) + 190 = 9,440s ≤ 9,600s；tests/test_oceanbase_ci.py 的预算用例在本机实跑 15 passed，把该步骤改回 90 分钟即失败（assert 5400 >= 9440）。这条视为闭合。

同一作业里还有一个步骤靠的是同一种等式，但没有对应的断言：

• 位置：.github/workflows/e2e-harness.yml 的 Test long-query retrieval and admission without skips，注释写「Seven cases in the larger shard: 7 * (300s + 10s grace) + 190s collection < 40m」，timeout-minutes: 40，--test-timeout 300。
• 该步骤收集到 13 个 oceanbase 节点，但 .github/oceanbase-migration-durations.json 的 source_commit 是 4b6cf42a，早于引入这 5 个用例的提交，13 个里只有 8 个有时长：在册的是 27–163s（test_supported_long_query_preserves_all_terms_in_both_http_searches 四条为 132/136/136/163s）；缺失的 5 个全在 tests/e2e/test_atomic_memory_fts_coverage.py——test_fts_coverage_counts_analyzer_terms_omitted_by_space_parser[oceanbase-{com,the,www}-legacy]、test_fts_coverage_keeps_distinct_terms_and_filters_before_limits[oceanbase]、test_hybrid_coverage_keeps_vectors_and_validates_rows_outside_lexical_matches[oceanbase]。
• assigned_nodes 对缺失节点用全体中位数（29s）估算，于是它们被排到最后、按「负载」而不是「条数」分摊。回复里的 13 个用例、6/7 正是这个估算下的结果。把缺失节点的耗时按同组在册用例的量级代入（≥120s）重跑同一算法，分配变成 8/5，最坏路径 8 × (300+10) + 190 = 2,670s = 44.5 分钟 > 40 分钟；翻转阈值落在 100s 与 120s 之间。
• 后果与原问题同形：同片多个用例挂起时，步骤上限先杀掉进程，scripts/oceanbase_ci.py run 的 wait(timeout - min(30, timeout/2)) → collect(budget=20) → execution.json / before-timeout/ 这条取证路径还没轮到，失败退化成「步骤超时」。

差别只在覆盖面：tests/test_oceanbase_ci.py 里 largest_shard × (case_timeout + 10) + 190 的断言只对 clean-migration 步骤生效（该用例名即 test_clean_migration_shard_budget_covers_slow_cases_and_keeps_cleanup_time），long-query、startup-snapshot、background-metrics 都没有同类断言，所以 master 再加用例时只有注释在维持等式。

建议：

1. 用包含这 5 个节点的运行刷新 .github/oceanbase-migration-durations.json，并把 source_commit 前移到包含它们的提交；
2. 把该断言按步骤表驱动到所有运行 run_atomic_memory_clean_migration.py 的步骤（从 --test-timeout 取逐用例上限、从该组 SUITES 的收集数取条数），而不是只钉住 clean-migration 一个步骤。

验证范围说明：13 个节点是用 -k oceanbase --collect-only 在本机 WSL 收集的（head 1742c5fb，上述 6 个文件在 c63f9b42 上 blob 相同）；5 个缺失节点的真实耗时无法在无 OceanBase 的环境测得，因此这一条的依据是「基线缺测 + 打包会翻转」的算术，不是实测耗时。

[hireview001]

• CI failed: CI result (failure).
• CI failed: harness-validation / Atomic Memory clean migration (OceanBase) (failure).
• CI failed: harness-validation / Acceptance (oceanbase) (failure).
• CI failed: harness-validation / Recall effort (OceanBase) (failure).
• CI failed: harness-validation / Atomic Memory clean migration (OceanBase shard 1/2) (failure).
• CI failed: harness-validation / Atomic Memory clean migration (OceanBase shard 0/2) (failure).
• CI failed: desktop-validation / linux (failure).
• CI failed: desktop-validation / macos (macos-15) (failure).
• CI failed: desktop-validation / windows (failure).
• CI failed: desktop-validation / macos (macos-15-intel) (failure).
• Additional CI issues: PR checks.

[Zxf-xufeng]

The conflicts with master 7fef43d are resolved and pushed in 9636ae2. I merged master independently into both PR branches, retained the Python test/cache budgets and cached license checker, and preserved master’s website runner change. GitHub now reports both #2515 and #2526 as MERGEABLE; the branches also combine locally without conflicts. Both pass make check. The latest CI run is still running, so this confirms conflict resolution rather than a full CI pass.