apollo_batcher: read class definitions through the class manager in view calls - #14967
Conversation
51d9b58 to
c7d7144
Compare
This stack of pull requests is managed by Graphite. Learn more about stacking. |
yoavGrs
left a comment
There was a problem hiding this comment.
@yoavGrs reviewed 1 file and made 3 comments.
Reviewable status: 1 of 6 files reviewed, 3 unresolved discussions (waiting on asaf-sw and matanl-starkware).
crates/apollo_batcher/src/batcher.rs line 1572 at r1 (raw file):
/// Maximal wall time a view entry point call may hold the batcher's request slot, which block /// production shares. Execution is bounded by the view call's resource bounds; the budget is for
Consider skipping the resources and the budget in the comment.
crates/apollo_batcher/src/batcher.rs line 1576 at r1 (raw file):
/// retries a 30 second per-attempt timeout up to 150 times in the distributed deployment, and the /// local client the other deployments use has neither timeout nor retries. pub(crate) const VIEW_CALL_TIMEOUT: Duration = Duration::from_secs(30);
- Move both consts to the top of the file.
- 30 secs sounds a bit long for batch production, no?
- Consider controlling it by conig.
- From where come the 150 times? It means we let 150 tasks work concurrently, each task takes 75 minutes.
crates/apollo_dashboard/src/panels/blockifier.rs line 30 at r1 (raw file):
class_cache_metrics: &CacheMetrics, ) -> Panel { let scope = class_cache_metrics.get_scope();
- Consider adding the scope validation of
get_scope. - Is ' get_scope ' dead code now?
Code quote:
get_scope()
yoavGrs
left a comment
There was a problem hiding this comment.
@yoavGrs reviewed 4 files and all commit messages.
Reviewable status: 5 of 6 files reviewed, 3 unresolved discussions (waiting on asaf-sw and matanl-starkware).
matanl-starkware
left a comment
There was a problem hiding this comment.
@matanl-starkware reviewed 6 files and all commit messages, and made 1 comment.
Reviewable status: all files reviewed, 3 unresolved discussions (waiting on asaf-sw and yoavGrs).
crates/apollo_batcher/src/batcher.rs line 1576 at r1 (raw file):
Previously, yoavGrs wrote…
- Move both consts to the top of the file.
- 30 secs sounds a bit long for batch production, no?
- Consider controlling it by conig.
- From where come the 150 times? It means we let 150 tasks work concurrently, each task takes 75 minutes.
Agree. The timeout should be 5 seconds.
c7d7144 to
61e8aa1
Compare
|
Artifacts upload workflows: |
asaf-sw
left a comment
There was a problem hiding this comment.
@asaf-sw+AGNT made 3 comments.
Reviewable status: 3 of 11 files reviewed, 3 unresolved discussions (waiting on matanl-starkware and yoavGrs).
crates/apollo_batcher/src/batcher.rs line 1572 at r1 (raw file):
Previously, yoavGrs wrote…
Consider skipping the resources and the budget in the comment.
Done. VIEW_CALL_TIMEOUT is now a config param and its description says only what it bounds: "Maximal wall time (in milliseconds) a view entry point call may hold the batcher's request slot, which block production shares." Dropped the resource bounds sentence from MAX_VIEW_CALL_RETDATA_LENGTH's doc too.
crates/apollo_batcher/src/batcher.rs line 1576 at r1 (raw file):
Previously, matanl-starkware (Matan Lior) wrote…
Agree. The timeout should be 5 seconds.
- Done.
MAX_VIEW_CALL_RETDATA_LENGTHnow sits right after the type aliases at the top of the file, andVIEW_CALL_TIMEOUTis gone (see 3). -
-
- Both done: it is now
batcher_config.dynamic_config.view_call_timeout_millis, default 5000, added tobatcher_config.jsonandreplacer_batcher_config.jsonalongside the schema. Dynamic rather than static so it can be retuned without a redeploy once the endpoint has a caller.
- Both done: it is now
-
- From
crates/apollo_deployments/resources/services/distributed/batcher.json:
"components.class_manager.remote_client_config.retries": 150over the default request_timeout_ms of 30000, so a single class read against an unreachable class manager can retry for 75 minutes. Those are sequential retries inside one read, not 150 concurrent tasks. Each view call reads its classes one after another, and the batcher serves requests one at a time, so the batcher's own request slot is what the timeout protects: timing out frees the slot, though the blocking task keeps running on its own thread until the reads give up. #14972 on top of this bounds how many such orphaned tasks can pile up.
crates/apollo_dashboard/src/panels/blockifier.rs line 30 at r1 (raw file):
Previously, yoavGrs wrote…
- Consider adding the scope validation of
get_scope.- Is ' get_scope ' dead code now?
- Restored.
CacheMetrics::get_scopeis nowCacheMetrics::validate_scope, which keeps theassert_eq!that misses and hits agree on their scope, and the panel builder calls it first. The dashboard generator runs on everyapollo_dashboardtest, so a mismatch still fails CI. - It was: after this change no caller needed the returned scope. Renaming rather than deleting keeps the assertion.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 61e8aa1. Configure here.
…lass manager in view calls Batcher::call_contract built its state reader with ApolloReader::new, which leaves class_reader as None. On that path classes are read straight from the batcher's storage, but the batcher only records that a class was declared, via append_state_diff, and never writes the definition: on the node path only apollo_central_sync calls append_classes. For a Cairo 1 class is_declared is true while get_casm_and_sierra returns (None, None), couple_casm_and_sierra maps that to Ok(None), and the expect in read_casm_and_sierra panics inside the blocking task. The node installs set_exit_process_on_panic, a global hook that calls std::process::exit(1), so the panic killed the node. Cairo 0 was broken too: ThinStateDiff::deprecated_declared_classes carries class hashes only, and the definitions reach storage through that same append_classes path, so the read failed with UndeclaredClassHash. Every view call therefore failed. Nothing caught it because call_contract is not wired into any deployment yet, and the existing tests inject a DictStateReader, which never reaches ApolloReader. Build the reader with new_with_class_reader instead, passing the class manager the execution path already uses in block_builder.rs, and thread the runtime handle the class reader needs to block on it from the blocking task, the way create_block_builder already takes one. Wrap that reader in StateReaderAndContractManager the way the execution path does, so both share one ContractClassManager. Without it every view call re-fetches and re-compiles every class it touches, pulling the full Sierra just to read get_sierra_version. The native classes whitelist comes from the same dynamic config block production reads, so a view call executes on the backend a block would: the deployments set it to the empty list, which the implicit NativeClassesWhitelist::All of StateReaderAndContractManager::new would have ignored. That part holds by construction and not by test: in the default build TrivialClassManager::get_runnable discards the whitelist and BatcherDynamicConfig::default() is All, so reverting to the implicit constructor fails nothing in CI. Sharing the manager shares its costs too. A view call cache miss enqueues on the same single-threaded native compilation worker block production uses, and the classes a view call brings in occupy the same 2000 entry LRU. Both are latent while the deployed whitelist is empty and nothing compiles natively. View call cache hits and misses are counted under their own metrics, so the block production miss ratio keeps measuring block production alone. Both sets live under the batcher scope, so the dashboard panel names its source explicitly instead of deriving the name from the scope; CacheMetrics::get_scope becomes CacheMetrics::validate_scope, keeping the assertion that misses and hits agree on their scope. The class reads have no overall deadline across retries. The distributed deployment sets components.class_manager.remote_client_config.retries to 150 in apollo_deployments/resources/services/distributed/batcher.json, over the default 30 second request_timeout_ms, and the local client the other deployments use has neither timeout nor retries. Bound the call with batcher_config.dynamic_config.view_call_timeout_millis, five seconds. It holds the batcher's serialized request slot, which block production shares; timing out frees the slot but does not cancel the blocking task, which runs to completion on its own thread. The new tests read classes back through the factory at the height call_contract uses, one past the last written state diff, covering the Cairo 1 route, the deprecated route, and a class the class manager does not have. Four more drive call_contract itself: over the real factory against a deployed Cairo 1 contract and a deployed Cairo 0 one, over a second call that must serve the class from the cache, and against a reader that never answers, for the timeout. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
61e8aa1 to
dffc899
Compare
asaf-sw
left a comment
There was a problem hiding this comment.
@asaf-sw+AGNT made 1 comment and resolved 1 discussion.
Reviewable status: 2 of 11 files reviewed, 3 unresolved discussions (waiting on matanl-starkware and yoavGrs).
yoavGrs
left a comment
There was a problem hiding this comment.
@yoavGrs resolved 3 discussions.
Reviewable status: 2 of 11 files reviewed, all discussions resolved (waiting on matanl-starkware).
asaf-sw
left a comment
There was a problem hiding this comment.
@asaf-sw reviewed 11 files and all commit messages.
Reviewable status: 2 of 11 files reviewed, all discussions resolved (waiting on matanl-starkware).
Resolved all comments but is sick and can't continue reviewing.
matanl-starkware
left a comment
There was a problem hiding this comment.
@matanl-starkware reviewed 9 files and all commit messages.
Reviewable status:complete! all files reviewed, all discussions resolved (waiting on asaf-sw).


Batcher::call_contract built its state reader with ApolloReader::new, which leaves
class_reader as None. On that path a Cairo 1 class is read straight from the batcher's
storage, and the batcher records that a class was declared, via append_state_diff, but
never writes the definition: only apollo_central_sync calls append_classes. So
is_declared is true while get_casm_and_sierra returns (None, None), couple_casm_and_sierra
maps that to Ok(None), and the expect in read_casm_and_sierra panics inside the blocking
task, surfacing as an InternalError.
Every view call against a Cairo 1 contract therefore fails. Nothing caught it because
call_contract has no production caller yet and the existing tests inject a DictStateReader,
which never reaches ApolloReader. A Cairo 0 callee is unaffected: its definition travels
inline in the state diff and is served by read_deprecated_casm.
Build the reader with new_with_class_reader instead, passing the class manager the
execution path already uses in block_builder.rs, and thread the runtime handle the
class reader needs to block on it from the blocking task, the way create_block_builder
already takes one.
The new test writes a declaration marker to real storage with no definition behind it and
reads the class back through the factory. It panics at apollo_state.rs:151 without the fix
and passes with it.
Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com