Rewrite/dag runner - #333
Open
Tanker2020 wants to merge 10 commits into
Open
Conversation
- Node: name, optional NodeFunc, directed edges to upstream deps - Edge: carries optional EdgeFunc to gate dependent start - ResourceNode: embeds Node, adds Manifest/VerifyFunc/DeployMethod - Graph: root-anchored container; AddNode, AddDependency, Topology - Cycle detection on every AddChild call (DFS reachability) - Topology() returns DFS post-order (dependency-first deploy order) Files: rewrite/dag/node.go, rewrite/dag/graph.go Signed-off-by: Nishanth Kolakalapudi <Nishanth.Kol@ibm.com>
- CompletionState: Verified/Unverified/Failed/Unstarted node buckets
- DeployCompleted(), VerifyCompleted(), AnyFailed() predicates
- HaltError{Fatal bool}: returned by NodeFunc to signal runner halt
Fatal=true → node lands in Failed, downstreams become Unstarted
Fatal=false → node lands in Unverified (deployed, not yet ready)
Files: rewrite/dag/completion_state.go
Signed-off-by: Nishanth Kolakalapudi <Nishanth.Kol@ibm.com>
Python used ThreadPoolExecutor + time.sleep(0.05) busy-poll loop. Go port uses goroutines + buffered results channel; scheduler blocks on select — zero CPU busy-polling. - NewRunner(graph, opts...) with functional options - WithConcurrency(0): serial topology walk, no goroutines (dry-run/test) - WithConcurrency(n): semaphore-capped parallel execution - WithVerifyUpstream(bool): gate dependent start on EdgeFunc result - context.Context cancellation: drains in-flight, marks rest Unstarted - Independent graph branches continue executing after sibling failure (matches Python oper8 intended behaviour; Python had a bug where the serial loop broke early on fatalErr) - stateMap protected by sync.Mutex; scheduler is single writer Files: rewrite/dag/runner.go Signed-off-by: Nishanth Kolakalapudi <Nishanth.Kol@ibm.com>
23 test cases covering:
Graph/Node: empty graph, duplicate node, empty name, cycle detection,
self-loop, topology order, String()
Runner serial: all succeed, empty graph, fatal halt (independent branch
still runs), unverified halt, disabled node, execution order
Runner concurrent: all succeed, fatal halt, independent nodes verified
parallel via start-time spread, race detector stress test
(20 nodes, atomic counter), context cancellation
EdgeFunc: blocks dependent when returns false, allows when true
CompletionState: all predicate combinations
ResourceNode: construction and field access
Concurrency test uses start-time recording rather than wall-clock
total elapsed — CI-safe on slow runners.
Files: rewrite/dag/runner_test.go
Signed-off-by: Nishanth Kolakalapudi <Nishanth.Kol@ibm.com>
- Matrix: Go 1.22 and 1.23 - go test -race -count=1 -timeout=60s ./dag/... - go build ./... and go vet ./dag/... - golangci-lint on dag/ package - Triggered on push to rewrite/DAG_Runner and PRs targeting main - working-directory: rewrite (module root) Files: ./.github/workflows/pr1-dag-runner.yml Signed-off-by: Nishanth Kolakalapudi <Nishanth.Kol@ibm.com>
…load config Signed-off-by: Nishanth Kolakalapudi <Nishanth.Kol@ibm.com>
Signed-off-by: Nishanth Kolakalapudi <Nishanth.Kol@ibm.com>
… and extra go test run Signed-off-by: Nishanth Kolakalapudi <Nishanth.Kol@ibm.com>
Tanker2020
marked this pull request as ready for review
August 11, 2026 08:17
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ports the Python
oper8DAG execution engine to Go as the first package in thegithub.com/example/oper8-gomodule (rewrite/). This is a standalone package with no dependencies on other rewrite PRs — it can be reviewed and merged independently.Files changed:
rewrite/dag/node.go—Node,ResourceNoderewrite/dag/graph.go—Graphrewrite/dag/completion_state.go—CompletionState,HaltErrorrewrite/dag/runner.go—Runner(serial + concurrent modes)rewrite/dag/runner_test.go— 23 table-driven testsrewrite/go.mod— module root,go 1.23.github/workflows/pr1-dag-runner.yml— CIWhat was ported
dag/node.py—Node,ResourceNodedag/node.godag/graph.py—Graphdag/graph.godag/completion_state.py—CompletionStatedag/completion_state.godag/runner.py—Runner,DagHaltError,NonThreadPoolExecutordag/runner.goDesign decisions
ThreadPoolExecutor+ sleep-poll → goroutines + buffered channelPython's Runner submits nodes to a
ThreadPoolExecutorand busy-polls withtime.sleep(poll_time)to check completion. Go replaces this entirely with a scheduler loop that reads from a bufferedchan nodeResult. Nodes post their result when done; the scheduler wakes only when there is actual work — zero busy-polling.DagHaltError→HaltErrorstructPython used
DagHaltError(failure=True/False, exception=...). Go uses*HaltError{Fatal bool, Cause error}. TheFatalfield maps directly to Python'sfailurefield. Any non-HaltErrorreturned from aNodeFuncis automatically wrapped as a fatal halt (matches Python's bareexcept Exceptionhandler).(bool, bool)returns →(changed bool, err error)Python returned
(success, changed)tuples throughout. Go uses the idiomatic(result, error)pattern. Callers useif err != nilinstead of checking two booleans.Serial mode (
concurrency=0) bug fixPython's
NonThreadPoolExecutorran nodes synchronously but the loopbreaked on a fatal error, meaning independent branches of the graph would never run. The Go serial runner usescontinueinstead — a failed node marks its dependentsUnstartedbut independent nodes still execute. This matches the intended concurrent behaviour.CompletionState.all_nodesremovedPython computed
all_nodesas the union of all four sets in__init__. Go omits this field —all_nodesis always derivable aslen(Verified) + len(Unverified) + len(Failed) + len(Unstarted)and the component-status formatter computes it inline. Removing it eliminates a redundant field that could become inconsistent.EdgeFunc/verify_upstreamPython's
_dependency_satisfiedcheckedverify_fn()per-edge to gate dependent execution. Go preserves this exactly viaEdgeFuncon each edge. TheWithVerifyUpstream(false)option disables all edge checks, matching Python'sverify_upstream=False.Test coverage (23 tests)
All tests run with
-race.