Skip to content

fix(worker): omit empty topology config - #262

Closed
crliu3227 wants to merge 1 commit into
SlinkyProject:mainfrom
crliu3227:fix-non-topology
Closed

fix(worker): omit empty topology config#262
crliu3227 wants to merge 1 commit into
SlinkyProject:mainfrom
crliu3227:fix-non-topology

Conversation

@crliu3227

Copy link
Copy Markdown

Summary

Fixes Worker registration failures when no Slurm topology is configured.

The operator previously emitted an empty Topology= entry in slurmd --conf.
When the container entrypoint appended CoreSpecCount and MemSpecLimit, Slurm
parsed CoreSpecCount as the topology value and marked the node as
INVALID_REG.

Changes

  • Do not emit Topology= unless topology is explicitly configured through
    NodeSet.spec.extraConf.
  • Remove the unused SLINKY_TOPOLOGY environment variable.
  • Preserve explicit Topology=<value> configuration through extraConf.
  • Keep topology synchronization through the Slurm API after node registration.
  • Add regression coverage for the default --conf output.
  • Update the architecture documentation.

Result

Workers without topology now register with resource configuration similar to:

--conf Features=slinky CoreSpecCount=36 MemSpecLimit=175607
This allows Slurm to correctly report the configured effective CPU and memory
resources.
Testing
go test -mod=mod ./internal/builder/...
go test -mod=mod ./internal/builder/workerbuilder

Fixes SlinkyProject/containers#20

@vivian-hafener

Copy link
Copy Markdown
Contributor

Good morning @crliu3227,

I am looking into this. I believe that the issue reported in SlinkyProject/containers#20 could be resolved more simply via a change to the slurmd-entrypoint file logic in the containers repo, instead of a change to operator. I am pursuing that option via an internal MR. Once I have confirmed that my resolution resolves your issue, I will close this MR.

Best,
Vivian Hafener

@vivian-hafener

Copy link
Copy Markdown
Contributor

I have an internal MR to https://github.com/SlinkyProject/containers that resolves this. I will update SlinkyProject/containers#20 with the commit ID once that is merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: Empty Topology= causes Worker CPU and memory registration failure

2 participants