Integrate EQ3 thermal tooling with GPU-verified hybrid waits - #6
Merged
Merged
Conversation
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
It combines production build/link changes with multiple new resource-governed experiment launchers and integrity-sensitive campaign tooling.
Review effort: Lite
Findings: 3
Open (8)
What changed in this PR
Integrates isolated EQ3 thermal, maintenance, workload, and GPU-helper tooling with current main while preserving production ABI/PTX/MQSim behavior.
Changes:
- Adds thermal, maintenance, rate-workload, and campaign validation tooling.
- Extends resource-gated launch and native MQSim observation paths.
- Adds GPU/PTX, provenance, compatibility, and regression-test coverage.
| File | Description |
|---|---|
tools/test_eq3_sensor_time_canonicalize.py |
Tests timestamp canonicalization. |
tools/test_eq3_layered_launch.py |
Tests scoped resource limits. |
tools/test_eq3_domain_v2_input.py |
Tests domain input derivation. |
tools/test_eq3_cpu_load_matrix.py |
Tests CPU load matrix generation and comparison. |
tools/test_eq3_campaign_stream.py |
Tests stream resource policies. |
tools/test_eq3_campaign_gate.py |
Tests campaign resource gating. |
tools/test_eq3_campaign_compare.py |
Tests comparison probes and crossings. |
tools/test_eq3_all_source_cap.py |
Tests source-cap conservation. |
tools/eq3_sensor_time_canonicalize.py |
Canonicalizes sensor timestamps. |
tools/eq3_plot_local_field.py |
Plots local-field diagnostics. |
tools/eq3_p2_common_window.py |
Compares common sensor windows. |
tools/eq3_layered_launch.py |
Propagates scoped launch resources. |
tools/eq3_experiment_gate.py |
Returns validated stage resource bindings. |
tools/eq3_cpu_load_matrix.py |
Generates sustained-load cases. |
tools/eq3_cpu_load_compare.py |
Aggregates CPU load arms. |
tools/eq3_campaign_stream.py |
Applies configurable process/output limits. |
tests/integration/test_vmem_tuning.py |
Separates external provenance validation. |
tests/integration/test_timing_future_device_ptx.py |
Expands PTX wait-loop coverage. |
src/mqsim_adapter/mqsim_online.cpp |
Carries request identity into MQSim observations. |
scripts/eval/test_mqsim_service_client.py |
Tests explicit artifact roots. |
scripts/eval/mqsim_service.py |
Adds gated submission and native observation support. |
scripts/build/prepare_cuda_glibc_overlay.py |
Creates a local CUDA/glibc header overlay. |
experiments/eq3_system_thermal/topology_model_map.json |
Maps topology layouts to thermal models. |
experiments/eq3_system_thermal/test_tiny_cpu_trace.py |
Tests deterministic CPU trace structure. |
experiments/eq3_system_thermal/test_run_system_point.py |
Tests topology energy execution. |
experiments/eq3_system_thermal/test_rate_workload.py |
Tests rate workload distributions. |
experiments/eq3_system_thermal/test_prepare_sensitivity.py |
Tests sensitivity preparation. |
experiments/eq3_system_thermal/test_prepare_rate_diagnostics.py |
Tests rate diagnostic preparation. |
experiments/eq3_system_thermal/test_energy.py |
Tests topology energy mapping. |
experiments/eq3_system_thermal/test_endpoint_policy.py |
Tests endpoint guard behavior. |
experiments/eq3_system_thermal/test_ecc_service_adapter.py |
Tests ECC service integration. |
experiments/eq3_system_thermal/test_ecc_cost_proxy.py |
Tests conditional ECC cost proxy behavior. |
experiments/eq3_system_thermal/test_causal_maintenance_age.py |
Tests causal maintenance aging. |
experiments/eq3_system_thermal/test_analyze_uncontrolled_first_constraint.py |
Tests thermal constraint analysis. |
experiments/eq3_system_thermal/test_aggregate_maintenance_campaign.py |
Tests unknown-value aggregation. |
experiments/eq3_system_thermal/stage/traces/tiny_qwen2_cpu_trace_v1/RUN_RECEIPT.json |
Records CPU trace execution evidence. |
experiments/eq3_system_thermal/stage/traces/tiny_qwen2_cpu_trace_v1/README.md |
Documents the tiny CPU trace. |
experiments/eq3_system_thermal/stage/traces/tiny_qwen2_cpu_trace_v1/CONSUMER_EVIDENCE.json |
Records trace consumer evidence. |
experiments/eq3_system_thermal/stage/traces/tiny_qwen2_cpu_trace_v1/config.json |
Defines trace configuration. |
experiments/eq3_system_thermal/run_endpoint_guard_point.py |
Provides isolated endpoint-policy execution. |
experiments/eq3_system_thermal/rate_workload.py |
Generates controlled offered traffic. |
experiments/eq3_system_thermal/prepare_maintenance_rate_repair.py |
Repairs pilot input bandwidth. |
experiments/eq3_system_thermal/prepare_ecc_pilots.py |
Prepares ECC proxy pilots. |
experiments/eq3_system_thermal/maintenance_point.schema.json |
Defines maintenance-point validation. |
experiments/eq3_system_thermal/maintenance_pilot_inputs_v2/SOFTWARE_COST_PROJECTION.json |
Records software cost projection evidence. |
experiments/eq3_system_thermal/maintenance_pilot_inputs_v2/MAIN_PROPOSAL.json |
Defines maintenance candidate arms. |
experiments/eq3_system_thermal/maintenance_pilot_inputs_v2/INDEX.json |
Indexes prepared maintenance pilots. |
experiments/eq3_system_thermal/MAINTENANCE_MAIN_CANDIDATE.md |
Documents maintenance campaign gates. |
experiments/eq3_system_thermal/maintenance_inputs.py |
Adds maintenance scenario inputs. |
experiments/eq3_system_thermal/freeze_extension.py |
Freezes extension inputs and identities. |
experiments/eq3_system_thermal/endpoint_policy.py |
Implements the HBM endpoint adapter. |
experiments/eq3_system_thermal/ecc_proxy/profile_v1.json |
Defines the ECC cost proxy. |
experiments/eq3_system_thermal/__init__.py |
Exposes experiment helpers. |
experiments/eq3_rate_thermal/plot_rate_thermal.py |
Plots rate and thermal results. |
experiments/eq3_rate_thermal/MODEL_WORKLOADS.md |
Documents model-size workloads. |
experiments/eq3_rate_thermal/model_workload_config.schema.json |
Defines workload configuration. |
experiments/eq3_rate_thermal/controlled_scenario.schema.json |
Defines controlled scenarios. |
experiments/eq3_rate_thermal/CONTROLLED_ANALYSIS.md |
Documents campaign analysis. |
experiments/eq3_maintenance/WORKLOAD_AND_ENERGY_BASIS.md |
Records workload and energy assumptions. |
experiments/eq3_maintenance/thermal/README.md |
Documents persistent thermal service usage. |
experiments/eq3_maintenance/thermal/CMakeLists.txt |
Builds the isolated thermal service. |
experiments/eq3_maintenance/thermal/client_schema.json |
Defines thermal client responses. |
experiments/eq3_maintenance/test_thermal_client.py |
Tests thermal guard behavior. |
experiments/eq3_maintenance/test_run_point_frozen.py |
Tests frozen maintenance inputs. |
experiments/eq3_maintenance/test_repro_maintenance_energy_source_label.py |
Tests maintenance source labeling. |
experiments/eq3_maintenance/test_incremental_fabric.py |
Tests incremental fabric observations. |
experiments/eq3_maintenance/test_energy_ledger.py |
Tests activity energy accounting. |
experiments/eq3_maintenance/steady_feasibility.py |
Runs bounded feasibility checks. |
experiments/eq3_maintenance/startup/test_analyze_concurrency.py |
Tests startup concurrency analysis. |
experiments/eq3_maintenance/startup/README.md |
Documents startup diagnostics. |
experiments/eq3_maintenance/startup/CMakeLists.txt |
Builds the startup probe. |
experiments/eq3_maintenance/startup/.gitignore |
Excludes isolated startup builds. |
experiments/eq3_maintenance/sources/WEIGHT_MODEL_SOURCES.md |
Documents model metadata sources. |
experiments/eq3_maintenance/sources/qwen2_5_weight_models.json |
Records model metadata and provenance. |
experiments/eq3_maintenance/source_domain_replay.py |
Replays isolated thermal source domains. |
experiments/eq3_maintenance/run_frozen_queue.py |
Executes frozen experiment queues. |
experiments/eq3_maintenance/run_campaign.py |
Runs the maintenance campaign. |
experiments/eq3_maintenance/repro_maintenance_energy_source_label.py |
Reproduces source-label classification. |
experiments/eq3_maintenance/launch_point.py |
Supervises bounded point execution. |
experiments/eq3_maintenance/incremental_fabric.py |
Adds incremental fabric cursors. |
experiments/eq3_maintenance/backend/service/replay_json.hpp |
Adds JSON integer parsing helpers. |
experiments/eq3_maintenance/backend/patches/series.json |
Records isolated MQSim patches. |
experiments/eq3_maintenance/backend/include/hbfsim/eq3_thermal/observer.hpp |
Declares isolated activity observation types. |
experiments/eq3_maintenance/backend/docs/REBUILD_AND_VALIDATE.md |
Documents isolated backend validation. |
experiments/eq3_maintenance/backend/client/maintenance_service.py |
Adds maintenance lifecycle client support. |
experiments/eq3_maintenance/analyze_source_domain_replay.py |
Analyzes source-domain replays. |
experiments/eq3_maintenance/.gitignore |
Excludes isolated experiment builds. |
docs/eq3_thermal/SYSTEM_THERMAL_INTERFACE_COVERAGE.md |
Records system interface coverage. |
docs/eq3_thermal/SOURCE_AND_GAP_LEDGER.md |
Adds ECC proxy provenance. |
docs/eq3_thermal/READ_BANDWIDTH_ALIGNMENT_STATUS.md |
Records bandwidth alignment limits. |
docs/eq3_thermal/Q4_8HBF_THERMAL_MODEL.md |
Documents the Q4 thermal model. |
docs/eq3_thermal/PARAMETER_GAPS.md |
Updates parameter-gap status. |
docs/eq3_thermal/HBF_ECC_REFRESH_IDENTITY_PROPOSAL.md |
Documents remaining ECC/refresh identity work. |
docs/eq3_thermal/GPU_GUARD_SOURCE_AUDIT.md |
Records GPU guard assumptions. |
docs/eq3_thermal/FOUR_TOPOLOGY_V2_STATUS.md |
Summarizes topology evidence. |
docs/eq3_thermal/ENDPOINT_POLICY_ADAPTER.md |
Documents endpoint policy behavior. |
docs/eq3_thermal/D3_ACCEPTANCE_V3_RETROSPECTIVE.md |
Records acceptance retrospective results. |
docs/eq3_thermal/BASIC_SYSTEM_CPU_CHAIN.md |
Documents the CPU topology chain. |
docs/eq3_thermal/BASIC_HBM_SCENARIO.md |
Documents the parameterized HBM scenario. |
docs/eq3_thermal/BASIC_FABRIC_DESIGN.md |
Documents the basic transfer fabric. |
docs/eq3_thermal/ADDITIONAL_RATE_DIAGNOSTICS_FREEZE.md |
Records prepared rate diagnostics. |
docs/eq3_thermal/A2_IDEAL_MAINTENANCE_REPLAY_DESIGN.md |
Documents ideal maintenance replay. |
docs/52-main-gpu-thermal-integration/evidence/review.json |
Records integration review evidence. |
docs/52-main-gpu-thermal-integration/evidence/live-main.jsonl |
Records live GPU helper cases. |
docs/52-main-gpu-thermal-integration/evidence/integration.json |
Records integration validation totals. |
CMakeLists.txt |
Links the MQSim service with the patched engine. |
cmake/MQSimPatchedBuild.cmake |
Applies the command-observer patch. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| import time | ||
|
|
||
| p=argparse.ArgumentParser();p.add_argument('--receipt',type=Path,required=True);p.add_argument('--wall-s',type=float,default=600);p.add_argument('--memory-reserve-gib',type=float,default=0);p.add_argument('--disk-reserve-gib',type=float,default=0);p.add_argument('command',nargs=argparse.REMAINDER);a=p.parse_args() | ||
| if not a.command or a.wall_s<=0:p.error('positive finite wall and explicit command required') |
Comment on lines
+48
to
+49
| if stack_map is not None: | ||
| self.argv+=['--stack-map',str(Path(stack_map).resolve(strict=True))] |
| if min(layers, frames, nx, ny) < 1 or not 0 < watchdog <= 600: | ||
| max_bytes=4 * 1024**3, policy='gzip', codec_library=None, | ||
| process_ram_gib=12): | ||
| if min(layers, frames, nx, ny) < 1 or watchdog <= 0 or process_ram_gib <= 0: |
| import argparse,concurrent.futures,datetime,json,os,pathlib,shutil,subprocess,time | ||
| p=argparse.ArgumentParser();p.add_argument('--stage',type=pathlib.Path,required=True);p.add_argument('--artifact-root',type=pathlib.Path,required=True);p.add_argument('--binary',type=pathlib.Path,required=True);p.add_argument('--thermal-binary',type=pathlib.Path,required=True);a=p.parse_args() | ||
| a.stage=a.stage.resolve();root=pathlib.Path(__file__).resolve().parents[2];here=pathlib.Path(__file__).resolve().parent | ||
| lock_path=a.stage/'MAIN_CAMPAIGN_LOCK_v1.json';lock=json.loads(lock_path.read_text());cpus=sorted(os.sched_getaffinity(0))[:3] |
| # Each wave has same workload/topology/scenario,3policies, independent singletons. | ||
| for start in range(0,len(queue),3): | ||
| wave=queue[start:start+3] | ||
| with concurrent.futures.ThreadPoolExecutor(max_workers=3) as pool: |
Comment on lines
+7
to
+8
| keyed={(x['rate_per_stack_rps'],x['policy']):x for x in rows} | ||
| if set(keyed)!={(r,p) for r in RATES for p in POLICIES}:raise ValueError('exact approved nine-point matrix required') |
| receipts.append({'file':path.name,'sha256':hashlib.sha256(path.read_bytes()).hexdigest(), | ||
| 'rate_per_stack_rps':rate,'policy':policy,'requests':len(case['requests']), | ||
| 'workload_identity_sha256':case['workload_identity_sha256']}) | ||
| assert len(identities)==1 |
| max_adjustment = 0.0 | ||
| non_time = [] | ||
| decimal_quantum = Decimal(str(quantum_s)) | ||
| with source_path.open(newline="") as source, output_path.open("x", newline="") as output: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Integrates the isolated EQ3 thermal simulation and maintenance tooling with current main (
78125a5) after verifying the consume hybrid-wait changes on an RTX 5090. The wait implementation, production CUDA/PTX runtime, future ABI, and original MQSim submodule are preserved; no timing rollback was needed.Adds a live production-helper regression for ready/timeout, mid-wait shutdown and generation invalidation, native bypass, and mixed warp groups. Repairs the raw-versus-filtered PTX coverage test, keeps its assertions active in Release, explicitly separates unavailable external calibration provenance from software checks, and adds a reproducible build-local CUDA13.1/glibc compatibility workaround.
Validation:
The existing cache-frame retirement/eviction race remains a documented protocol limitation. These are bounded software integration checks, not an application throughput claim or physical HBF calibration. Original P2 failures,400K domain and blind-test status remain unchanged; ECC/refresh identity coupling and all research ablations are not claimed complete.
Reproduction details and compact receipts:
docs/52-main-gpu-thermal-integration/README.md. Large campaign raw, build products and machine-specific credentials are not included.