Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
148 changes: 148 additions & 0 deletions docs/usage/network-policies.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,148 @@
# Network Policies

## Table of Contents

<!-- mdformat-toc start --slug=github --no-anchors --maxlevel=6 --minlevel=1 -->

- [Network Policies](#network-policies)
- [Table of Contents](#table-of-contents)
- [Overview](#overview)
- [Component Traffic Diagram](#component-traffic-diagram)
- [Enabling Network Policies](#enabling-network-policies)
- [Per-Component and Per-Instance Toggles](#per-component-and-per-instance-toggles)
- [Extra Rules](#extra-rules)
- [srun and Ephemeral Ports](#srun-and-ephemeral-ports)
- [Caveats](#caveats)

<!-- mdformat-toc end -->

## Overview

Both the `slurm-operator` and `slurm` Helm charts can render opt-in Kubernetes
[NetworkPolicy] resources that isolate each Slurm component at the network
layer. Policies are **disabled by default** and are enabled through the
`networkPolicy.enabled` value in each chart.

When enabled, every component only accepts the ingress it needs and is only
allowed to reach the peers it must talk to (plus DNS). The diagram below
captures the intended communication boundaries between all components.

## Component Traffic Diagram

```mermaid
flowchart LR
subgraph kube [Kubernetes Control Plane]
kubeapi["Kube API Server (443)"]
end
subgraph operator [Slurm Operator]
op["Operator (metrics:8080)"]
wh["Webhook (server:9443)"]
end
subgraph slurm [Slurm Cluster]
ctrl["Controller (slurmctld:6817)"]
worker["NodeSet (slurmd:6818, srun:*, ssh:22)"]
acct["Accounting (slurmdbd:6819)"]
rest["RestApi (slurmrestd:6820)"]
login["LoginSet (ssh:22)"]
end
db["External DB (3306)"]
users["Users / Clients"]

worker <-->|"6817 / 6818"| ctrl
worker <-->|"all TCP (srun)"| worker
acct <-->|"6817 / 6819"| ctrl
rest -->|"6817"| ctrl
login -->|"6817"| ctrl
login -->|"6819 (sacct)"| acct
login -->|"all TCP (srun/ssh)"| worker
op -->|"6820"| rest
op -->|"443"| kubeapi
wh -->|"443"| kubeapi
kubeapi -->|"9443"| wh
acct -->|"3306"| db
users -->|"6820"| rest
users -->|"22"| login
```

## Enabling Network Policies

Enable the policies per chart with the global toggle:

```sh
# slurm chart (controller, nodeset, accounting, restapi, loginset)
helm upgrade slurm oci://ghcr.io/slinkyproject/charts/slurm \
--namespace slurm --set networkPolicy.enabled=true

# slurm-operator chart (operator, webhook)
helm upgrade slurm-operator oci://ghcr.io/slinkyproject/charts/slurm-operator \
--namespace slinky --set networkPolicy.enabled=true
```

DNS resolution (UDP/TCP 53) is always permitted as egress so the components can
resolve in-cluster service names.

## Per-Component and Per-Instance Toggles

Once the global toggle is on, each component can be disabled individually:

- Singleton components (`controller`, `restapi`, `accounting`, `operator`,
`webhook`) expose `<component>.networkPolicy.enabled`.
- Map components (`nodesets`, `loginsets`) expose a per-instance
`networkPolicy.enabled` flag inside each map entry, defaulting to the value in
`nodesetDefaults` / `loginsetDefaults`. One NetworkPolicy is generated per
enabled instance, scoped via the `app.kubernetes.io/instance` label.

```yaml
networkPolicy:
enabled: true
nodesets:
gpu:
networkPolicy:
enabled: false # disable just this NodeSet's policy
```

## Extra Rules

Additional ingress/egress rules can be appended at three levels:

- Global: `networkPolicy.extraIngress` / `networkPolicy.extraEgress` (applied to
every policy in the chart).
- Per-component: `<component>.networkPolicy.extraIngress` /
`extraEgress`.
- Per-instance: inside each `nodesets` / `loginsets` map entry under
`networkPolicy.extraIngress` / `extraEgress`.

```yaml
networkPolicy:
enabled: true
extraEgress:
- to:
- ipBlock:
cidr: 10.0.0.0/8
ports:
- protocol: TCP
port: 443
```

## srun and Ephemeral Ports

`srun` opens ephemeral ports for interactive job I/O. To keep these flows
working, the policies allow **all TCP** between `slurmd` <-> `slurmd` and from
`login` -> `slurmd`. In hardened environments, constrain this range with
[`SrunPortRange`] in `slurm.conf` and tighten the corresponding `extraIngress`
/ `extraEgress` rules accordingly.

## Caveats

- Policies use the `app.kubernetes.io/name` and `app.kubernetes.io/instance`
labels applied by the operator; a CNI that enforces NetworkPolicy is required.
- Non-default ports (Slurm, ssh, mariadb) must be reflected in your values; the
policies follow the ports configured for each component.
- The operator-to-`slurmrestd` egress uses an empty `namespaceSelector` to
support deployments where the operator and Slurm cluster live in different
namespaces.

<!-- links -->

[NetworkPolicy]: https://kubernetes.io/docs/concepts/services-networking/network-policies/
[`SrunPortRange`]: https://slurm.schedmd.com/slurm.conf.html#OPT_SrunPortRange
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
{{- /*
SPDX-FileCopyrightText: Copyright (C) SchedMD LLC.
SPDX-License-Identifier: Apache-2.0
*/}}

{{- if and .Values.networkPolicy.enabled .Values.operator.enabled .Values.operator.networkPolicy.enabled }}
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: {{ include "slurm-operator.name" . }}
namespace: {{ include "slurm-operator.namespace" . }}
labels:
{{- include "slurm-operator.operator.labels" . | nindent 4 }}
spec:
podSelector:
matchLabels:
{{- include "slurm-operator.operator.selectorLabels" . | nindent 6 }}
policyTypes:
- Ingress
- Egress
ingress:
{{- if ne (int .Values.operator.metricsPort) 0 }}
- ports:
- protocol: TCP
port: {{ .Values.operator.metricsPort | default 8080 }}
{{- end }}
{{- with .Values.networkPolicy.extraIngress }}
{{- toYaml . | nindent 4 }}
{{- end }}
{{- with .Values.operator.networkPolicy.extraIngress }}
{{- toYaml . | nindent 4 }}
{{- end }}
egress:
- ports:
- protocol: TCP
port: 443
- ports:
- protocol: TCP
port: 6820
to:
- namespaceSelector: {}
podSelector:
matchLabels:
app.kubernetes.io/name: slurmrestd
- ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
{{- with .Values.networkPolicy.extraEgress }}
{{- toYaml . | nindent 4 }}
{{- end }}
{{- with .Values.operator.networkPolicy.extraEgress }}
{{- toYaml . | nindent 4 }}
{{- end }}
{{- end }}
51 changes: 51 additions & 0 deletions helm/slurm-operator/templates/networkpolicy/webhook-netpol.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
{{- /*
SPDX-FileCopyrightText: Copyright (C) SchedMD LLC.
SPDX-License-Identifier: Apache-2.0
*/}}

{{- if and .Values.networkPolicy.enabled .Values.webhook.enabled .Values.webhook.networkPolicy.enabled }}
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: {{ include "slurm-operator.webhook.name" . }}
namespace: {{ include "slurm-operator.namespace" . }}
labels:
{{- include "slurm-operator.webhook.labels" . | nindent 4 }}
spec:
podSelector:
matchLabels:
{{- include "slurm-operator.webhook.selectorLabels" . | nindent 6 }}
policyTypes:
- Ingress
- Egress
ingress:
- ports:
- protocol: TCP
port: {{ .Values.webhook.serverPort | default 9443 }}
{{- if ne (int .Values.webhook.metricsPort) 0 }}
- ports:
- protocol: TCP
port: {{ .Values.webhook.metricsPort }}
{{- end }}
{{- with .Values.networkPolicy.extraIngress }}
{{- toYaml . | nindent 4 }}
{{- end }}
{{- with .Values.webhook.networkPolicy.extraIngress }}
{{- toYaml . | nindent 4 }}
{{- end }}
egress:
- ports:
- protocol: TCP
port: 443
- ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
{{- with .Values.networkPolicy.extraEgress }}
{{- toYaml . | nindent 4 }}
{{- end }}
{{- with .Values.webhook.networkPolicy.extraEgress }}
{{- toYaml . | nindent 4 }}
{{- end }}
{{- end }}
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
operator manifest should match snapshot:
1: |
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
labels:
app.kubernetes.io/instance: test-release
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: slurm-operator
app.kubernetes.io/part-of: slurm-operator
app.kubernetes.io/version: 1.2.3
helm.sh/chart: slurm-operator-1.2.3
name: slurm-operator
namespace: test-namespace
spec:
egress:
- ports:
- port: 443
protocol: TCP
- ports:
- port: 6820
protocol: TCP
to:
- namespaceSelector: {}
podSelector:
matchLabels:
app.kubernetes.io/name: slurmrestd
- ports:
- port: 53
protocol: UDP
- port: 53
protocol: TCP
ingress:
- ports:
- port: 8080
protocol: TCP
podSelector:
matchLabels:
app.kubernetes.io/instance: test-release
app.kubernetes.io/name: slurm-operator
policyTypes:
- Ingress
- Egress
webhook manifest should match snapshot:
1: |
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
labels:
app.kubernetes.io/instance: test-release
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: slurm-operator-webhook
app.kubernetes.io/part-of: slurm-operator
app.kubernetes.io/version: 1.2.3
helm.sh/chart: slurm-operator-1.2.3
name: slurm-operator-webhook
namespace: test-namespace
spec:
egress:
- ports:
- port: 443
protocol: TCP
- ports:
- port: 53
protocol: UDP
- port: 53
protocol: TCP
ingress:
- ports:
- port: 9443
protocol: TCP
podSelector:
matchLabels:
app.kubernetes.io/instance: test-release
app.kubernetes.io/name: slurm-operator-webhook
policyTypes:
- Ingress
- Egress
Loading