Add slurm node name labeling to k8s nodes for StatefulSet-mode NodeSets for slurm-bridge interoperability

[arsdragonfly]

Summary

Adds slurm node name labeling to k8s nodes for StatefulSet-mode NodeSets for slurm-bridge interoperability. Applicable when pinToNode is true and oversubscription is turned off.

Checklist

• I have read the
  CONTRIBUTING.md
  and the
  Code of Conduct.
• New or existing tests cover these changes (where applicable).
• Documentation is updated if user-visible behavior changes.

Additional Context

An important nuance regarding why StatefulSet-mode NodeSets could be more appealing (even in a slurm-bridge setting), compared to DaemonSet-mode NodeSets, emerges when one considers the scenario of workload-aware scale-in of the k8s cluster itself (think of saving costs on the cloud).
With Karpenter or Cluster Autoscaler, which work via eliminating idle nodes without running pods, StatefulSet-mode NodeSet scale-in is already workload-aware in slurm-operator, after which scale-in would then trivially work in a workload-aware fashion as well. A working example can be found here.
With DaemonSet-mode NodeSet (which, unlike real DaemonSet, is actually also workload-bearing), the decision of "which node to evict during scale-in" no longer has a nice workload-aware k8s-native answer, and one would find oneself needing awkward custom solutions.

[killianmuldoon]

Thanks for this @arsdragonfly - I'll pull it internally and incorporate the changes.

[killianmuldoon]

I'm not totally comfortable with including this change. What we want here is to encode a contract slurm-bridge uses for hybrid nodes. Today this has to be done manually by cluster admins, I'm not sure encoding it inside slurm-operator is the right choice here.

The current implementation creates a very contract for the StatefulSet NodeSet - leaking slurm-bridge implementation details into the way the operator works.

I do understand the pain point here and would love to find a solution - I wonder if this be done more suitably with an admission controller or some other custom piece of orchestration?

[arsdragonfly]

I'm not totally comfortable with including this change. What we want here is to encode a contract slurm-bridge uses for hybrid nodes. Today this has to be done manually by cluster admins, I'm not sure encoding it inside slurm-operator is the right choice here.

The current implementation creates a very contract for the StatefulSet NodeSet - leaking slurm-bridge implementation details into the way the operator works.

I do understand the pain point here and would love to find a solution - I wonder if this be done more suitably with an admission controller or some other custom piece of orchestration?

Not sure why the cluster-admin-tunable label isn't a proper contract already.
slurm-operator owns the lifecycle of its pods and hence should manage the lifecycle of mappings.

[killianmuldoon]

slurm-operator owns the lifecycle of its pods and hence should manage the lifecycle of mappings

Agreed - but this is a slurm-bridge mapping rather than something related to operator. I'm trying to understand what would be the right approach here - we don't want tight coupling between the two projects without a strong motivation.

[killianmuldoon]

@arsdragonfly - can I ask what kind of use case you're trying to unlock here?

• Should the slurm-operator slurmd nodes scheduled by slurm-bridge?
• Are you looking to have a single slurmd pod per Kubernetes node?
• Is the main focus having a slurm-bridge hybrid cluster but with the right metadata to allow autoscaling?

[arsdragonfly]

Should the slurm-operator slurmd nodes scheduled by slurm-bridge?

Yes, that should be a valid scenario.

Are you looking to have a single slurmd pod per Kubernetes node?

That's what oversubscribe set to false already supports, yes.

Is the main focus having a slurm-bridge hybrid cluster but with the right metadata to allow autoscaling?

Yes, with particular focus on sane workload-aware scale-down behavior, as stated in the additional context in the beginning. (Scale-up-wise DaemonSet is also doable if you have something like KEDA -> CAPI but its scale-down isn't trivially graceful and workload-aware)

we don't want tight coupling between the two projects without a strong motivation

Not sure if that had been a goal at all from other maintainers' perspective. DaemonSet relied on an even more implicit pod hostname - k8s node name matching.