Skip to content

Skaram/detect softnpu review - #11302

Open
sion42x wants to merge 10 commits into
skaram/detect-softnpufrom
skaram/detect-softnpu-review
Open

sion42x wants to merge 10 commits into
skaram/detect-softnpufrom
skaram/detect-softnpu-review

Conversation

@sion42x

@sion42x sion42x commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Series of smallish commits to address @jgallagher's comments on #11237

Tested on voxel (with and without SP/RoT emu) and will be tested on a racklette after CI.

`production` and `custom: tofino_asic` deployments resolve to `Auto`
without a startup probe since the monitor takes care of the ASIC.
Without anything for the physical ASIC table to find, removing that
and `SwitchProbe` as well. At this point only `detect_switch_hardware`
only looks for SoftNPU devices.

Worth noting this walks back the "identical shape" part of @jmpesp's
comment on #11286. So there's still asymmetry but only one function
for startup detection. Hopefully a good compromise.
If a device stays busy, can't be opened, or answers with something
other than Rversion, it can't be ruled out as that could let a scrimlet
come up as a plain sled. Only a different 9P version string is a
negative answer.

Also added better comments on vio9p reads, and split the I/O error
by operation.
Start pumpkind when a physical ASIC is present and the host is an
Oxide sled.
The machine flag selects a sled-agent config that names its switch.
The switch flag selects the zone package. Without this check a
mismatch would boot a zone the config can't drive.
@sion42x
sion42x requested a review from jgallagher September 16, 2026 14:56
@sion42x

sion42x commented Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

@jmpesp tagging you here primarily for 4f181ec which walks back some of the symmetry in detection. Still one function for startup detection though, so hopefully this works? The change was made based on @jgallagher's comment here: #11237 (comment)

Comment thread sled-hardware/src/illumos/mod.rs Outdated
Comment thread sled-hardware/src/illumos/softnpu.rs Outdated
Comment thread sled-hardware/src/illumos/softnpu.rs
Comment thread package/src/target.rs Outdated
Comment thread sled-agent/src/config.rs Outdated
Comment thread sled-agent/src/config.rs Outdated
Deployment becomes a role and a switch, built from the config table
by `try_from`. This will reject combinations sled-agent couldn't
resolve (e.g. scrimlet with a detected backend, auto with a fixed one).
Each method is a match on the switch.
`detect_switch_hardware` could only ever find a SoftNPU once
Tofino went back to hardware monitor detection. Log before each
open of the 9p device to bracket slow handshakes.
The machine's sled-agent config names its switch, which made the
switch flag redundant and capable of breaking a built. Presets
and target files that still spell out a switch are detected against
the machine now, and the switch flag is dropped.
@sion42x

sion42x commented Sep 18, 2026

Copy link
Copy Markdown
Contributor Author

I merged main in due to the CI issues and the time this PR and its parent have been open.

Tests ran successfully on voxel and omicron-simulated, exercising both SoftNPU paths. Specifically, standalone used kind = "standalone" and came online, pumpkind was skipped, the 9p probe never ran, and the scrimlet resolved. Good to see the new pathing in action there. I'll test on a racklette as well to exercise the prod path one more time while waiting for review.

@sion42x

sion42x commented Sep 19, 2026

Copy link
Copy Markdown
Contributor Author

@jgallagher Tested on dublin, came up exactly as expected:

[dublin]root@oxz_switch0:~# pilot host exec -c 'svcs -H -o state oxide/sled-agent pumpkind; grep -c "kind = .production" /opt/oxide/sled-agent/pkg/config.toml' 14-17
14  2F8JEXDK           ok: online
online
1
15  BRM27230037        ok: online
online
1
16  BRM23230018        ok: online
online
1
17  BRM23230010        ok: online
online
1
[dublin]root@oxz_switch0:~# pilot host exec -c 'grep -c "probing virtio 9p" /var/svc/log/oxide-sled-agent:default.log; grep -h -E "Failed to detect|SledModeConfig|unknown field|pumpkind manifest" /var/svc/log/oxide-sled-agent:default.log | cut -c1-140' 14-17
14  2F8JEXDK           ok: 0
15  BRM27230037        ok: 0
16  BRM23230018        ok: 0
17  BRM23230010        ok: 0
[dublin]root@oxz_switch0:~# pilot host exec -c 'grep -h -E "Found tofino node|Updated tofino|tofino device found" /var/svc/log/oxide-sled-agent:default.log /var/svc/log/oxide-tofino-monitor:default.log 2>/dev/null | cut -c1-120 | head -2; zoneadm list | grep -c oxz_switch' 14-17
14  2F8JEXDK           ok: {"msg":"Updated tofino","v":0,"name":"SledAgent","level":30,"time":"1986-12-28T00:00:49.448956867Z","hostname":"2F8JEXDK
Dec 28 00:00:49.531 INFO tofino device found, unit: tofino-monitor
1
15  BRM27230037        failure: exit code 1: 0
16  BRM23230018        ok: {"msg":"Updated tofino","v":0,"name":"SledAgent","level":30,"time":"1986-12-28T00:00:45.47954221Z","hostname":"BRM232300
Dec 28 00:00:45.531 INFO tofino device found, unit: tofino-monitor
1
17  BRM23230010        failure: exit code 1: 0
Error: some operations failed
[dublin]root@oxz_switch0:~# 
[dublin]root@oxz_switch0:~# 
[dublin]root@oxz_switch0:~# 
[dublin]root@oxz_switch0:~# svcs -H -o state dendrite; svcprop -p config/board_rev dendrite; ls -l /dev/tofino/1
online
b
crw-------   1 root     sys      143,  1 Dec 28  1986 /dev/tofino/1
[dublin]root@oxz_switch0:~# exit
logout
stevekaram@castle ~/2bc84e9346c70f9f7d0b4553e8858380b56705b8 $ pilot tp login 1  
The illumos Project     helios-3.0.24263        September 2026
[dublin]root@oxz_switch1:~# [dublin]root@oxz_switch0:~# svcs -H -o state dendrite; svcprop -p config/board_rev dendrite; ls -l /dev/tofino/1
online
b
crw-------   1 root     sys      143,  1 Dec 28  1986 /dev/tofino/1
-bash: [dublin]root@oxz_switch0:~#: command not found
b
crw-------   1 root     sys      143,  1 Dec 28  1986 /dev/tofino/1
-bash: online: command not found
-bash: b: command not found
-bash: crw-------: command not found
[dublin]root@oxz_switch1:~# svcs -H -o state dendrite; svcprop -p config/board_rev dendrite; ls -l /dev/tofino/1
online
b
crw-------   1 root     sys      143,  1 Dec 28  1986 /dev/tofino/1

So the gist path worked, and no startup probe had to happen. Good stuff!

Opens the 9p device non-blocking, and fails with Timeout when
Tversion goes unanswered. Since vio9p doesn't have chpoll, the
reply is polled by read. Also now checks the Rversion size field,
drops an unused error, sources Pumpkind detect error, simplifies
a config test, and describes deployment kinds better in docs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants