A late actuator exits its caller, which is worst in the loops most likely to hit it

[jimsynz]

Falls out of #235, where BB.Actuator.set_position/4 became synchronous. Filing it separately because it's a deliberate change to that function's contract rather than part of the change that exposed it.

The behaviour

set_position/4 under the default delivery: :pubsub delivers by GenServer.call, so a :timeout doesn't return — it exits the caller, per GenServer.call/3 semantics. BB.Motion inherits this, and so does anything wrapping it.

Before #235 the same call sites were fire-and-forget publishes. A wedged actuator was invisible; now it's fatal to whoever asked.

Why it matters more than it looks

Porting the ecosystem to #235 turned up the same concern independently in three packages, each a long-lived process that drives a robot in a loop:

• bb_tui — set_actuator/4 runs inline in the TUI's event loop. A 5 s stall per jog keypress (doubled over :rpc, once per key autorepeat), and a wedged actuator exits the process that owns the terminal. Its PR chose delivery: :direct specifically to avoid killing the dashboard.
• bb_policy — ActuatorCommand.apply/2 runs inside a Runner/Controller tick. A slow actuator can now take the runner down where previously it published and moved on. Its PR declined to invent a timeout, on the grounds that the budget is robot-specific, and documented the knobs instead.
• BB.Motion — the fan-out uses Task.async, which links, so one late actuator exits the task and the exit propagates to whoever called move_to/4.

The pattern is consistent: the callers most likely to meet a slow actuator are the ones least able to afford dying, and the only escape offered is delivery: :direct, which gives up refusal reporting altogether. That's the same all-or-nothing the issue behind #235 complained about, one layer out.

The question

Should a timeout come back as {:error, :timeout} instead of exiting, so a control loop can skip a late step and send the next target?

Arguments for: it makes :timeout an outcome like a refusal, which is what a loop wants; it removes the "either block and risk death, or cast and learn nothing" fork; and it matches what the caller can actually do about it — a late step is stale, so dropping it is usually correct.

Arguments against: GenServer.call/3 semantics are what an Elixir developer expects from a blocking call, and hiding an exit behind an error tuple hides a genuinely broken actuator; a caller that ignores {:error, :timeout} in a loop will spin happily against dead hardware; and catching an exit to synthesise a return value is the kind of thing that looks tidy and reads badly.

Worth noting either way: a timeout means "I stopped waiting", not "it didn't happen". The command is already in the actuator's mailbox and will be handled. Whatever shape the answer takes has to say that, or callers will treat a timeout as a rollback.

Not urgent

Nothing is blocked. The downstream PRs each made a local choice and documented it. This wants deciding before the pattern sets, not before the release.

[jimsynz]

More evidence from finishing the downstream ports, and it sharpens the question: the packages that chose the synchronous path are now writing the workaround by hand.

bb_kino and bb_liveview both took set_position/4 rather than delivery: :direct, and to make it survivable in a UI each of them had to:

• catch the exit, so a dead or wedged actuator reports instead of killing the widget
• drop the timeout to 250 ms, on the grounds that only a drag's latest target matters
• throttle the event rate (33 ms in bb_kino; bb_liveview already had phx-debounce="16")

That's three packages independently reinventing "turn the exit into a value, and don't wait 5 seconds about it" — which is what this issue proposes doing once, in the framework.

Worth recording the case that made them choose synchronous at all, because it isn't the one I'd assumed. It wasn't NotArmed — both widgets already subscribe to [:state_machine] and disable their sliders when disarmed. It was UnsupportedCommand: against a driver that narrows c:BB.Actuator.command_payloads/1, you drag the slider, nothing moves, and nothing anywhere ever says why. A cast cannot report that, and no amount of UI state can infer it. So the refusal genuinely has to come back — which means the exit has to be survivable.

Also worth noting the two UI packages in the same ecosystem reached opposite conclusions for the same kind of caller: bb_tui chose delivery: :direct to protect its event loop, bb_kino and bb_liveview chose synchronous plus a catch. Both reasoned it carefully and wrote the reasoning down. When equally careful readers of the same docs split like that, the default is doing too little to help them.

Two smaller notes for whoever picks this up:

• set_position/4's docs state the return contract but mention the exit only in passing, under :timeout. Every agent that ported a package flagged it as the thing the docs undersell. Even without changing the contract, that paragraph should lead with it.
• If a timeout does become {:error, :timeout}, :noproc needs deciding at the same time — a command to an actuator that isn't running is the other exit, and it's arguably a different answer (a missing actuator is a wiring bug; a slow one is a runtime condition).

[jimsynz]

Researched this across the whole workspace plus bb_tui. Three things turned up that aren't in the issue, and one of them changes the shape of the question, so I'll lead with that rather than restate the arguments.

The framing is wrong on one point: a timeout does not drop the command

The issue's own caveat — "a timeout means I stopped waiting, not it didn't happen" — is stated as a documentation problem. It isn't. It invalidates the main argument for the change.

Arguments for: [...] it matches what the caller can actually do about it — a late step is stale, so dropping it is usually correct.

Nothing gets dropped. The caller drops waiting; the actuator still executes. Demonstrated with a GenServer that takes 300 ms per command, called three times with a 100 ms timeout, exit caught each time:

caller: command 1 -> :timed_out
caller: command 2 -> :timed_out
  [actuator] HANDLED command 1 — hardware driven
caller: command 3 -> :timed_out
  [actuator] HANDLED command 2 — hardware driven
  [actuator] HANDLED command 3 — hardware driven
caller mailbox after all late replies: {:messages, []}

Every command the caller gave up on was applied to the hardware, in order, late. (Stale replies don't leak back — OTP aliases discard them — so that part is clean.)

So {:error, :timeout} plus "skip the late step and send the next target", which is precisely what the issue proposes a control loop would do, builds an unbounded backlog of stale targets that the robot then sweeps through. At bb_policy's default 20 Hz against an actuator taking 1 s per command, that's roughly three commands per second accumulating forever, and a joint chasing positions the operator abandoned seconds ago. On real hardware that's a worse failure than dying.

There is no coalescing, stale-drop or queue-depth awareness anywhere in BB.Actuator.Server or BB.Actuator — I grepped for all of it. And delivery: :direct, the currently-recommended escape, has exactly the same unbounded queue and always has. So the framework already ships the failure mode the exit is supposedly protecting us from; the exit only prevents it on the one path where you also lose the caller.

This is also, I think, the real reason all three UI packages throttle. bb_kino says so out loud, in JavaScript:

// The server waits for the actuator to accept each command, so an
// unthrottled drag would queue events faster than the robot can answer.
// The trailing send makes sure the position you let go on is the one the
// joint ends up at.

That is latest-wins trailing-edge coalescing, implemented in a browser, because bb doesn't offer it. bb_liveview does the same with phx-debounce="16"; bb_tui leans on keypress rate. Which means the duplicated workaround is three separate things, and answering this issue only deletes one of them:

workaround	deleted by {:error, _}?
catch the exit	yes
drop the timeout to 250 ms	no — still a per-caller policy call
rate-limit the command stream	no — becomes more necessary

bb_mcp's workaround does not work, and cannot be fixed at the call site

Fourth site the issue doesn't name. bb_mcp/lib/bb/mcp/tools/send_joint_positions.ex:253:

defp send_positions(robot, positions, opts) do
  Motion.send_positions(robot, positions, opts)
rescue
  exception -> {:error, Error.execution("send_joint_positions failed: #{Exception.message(exception)}")}
catch
  kind, reason -> {:error, Error.execution("send_joint_positions failed: #{kind}: #{inspect(reason)}")}
end

That catch never fires. BB.Motion.send_positions/3's :pubsub branch is Task.async + Task.await_many(:infinity) (lib/bb/motion.ex:503-511), and Task.async links — a task exit arrives as a link signal, which kills the process outside any try. Verified both shapes side by side:

CASE 1 (direct call in try/catch):        :caught_timeout
CASE 2 (Task.async inside try/catch):     CALLER DIED, catch did NOT fire.
                                          reason={:timeout, {GenServer, :call, [Slow, :slow, 100]}}

So the "each caller can handle this locally" position is false for anything going through BB.Motion. It is not locally handleable without Task.Supervisor.async_nolink or trapping exits, and only bb can change that. Worse, when it doesn't die (a refusal rather than a timeout) the same wrapper reports send_joint_positions failed for commands that in the timeout case will have been applied — it tells an LLM the move failed when the arm is moving.

Add a fifth instance, in bb_tui/lib/bb/tui/robot.ex:369, same class and about bb's own partial exit-catching:

BB.Command.cancel/1 is GenServer.stop(pid, {:shutdown, :cancelled}), which exits when the command terminates for any other reason. A handler that stops cleanly on cancellation exits :normal, so the stop call raises even though the cancel did exactly what was asked. bb only catches {:noproc, _}, so that exit escapes — and since this is dispatched inline from BB.TUI.App, an escaping exit would take the whole dashboard down with it.

bb_tui reversed course, so the "opposite conclusions" observation is out of date

mcass19/bb_tui#21 ("feat: surface actuator refusals"), merged 25 Aug — five days after this issue was filed — moved jog keypresses off delivery: :direct and onto the synchronous default, citing bb_liveview's reasoning by name. @jog_timeout_ms is 250. So it isn't "two careful readers split"; it's all three UI packages independently converging on the same shape — synchronous, 250 ms, exit-safety, rate limiting. That's stronger evidence for the fix than the split was.

But it also supplies the best structural counter-argument: bb_tui did not solve it with a catch. It moved the call off the event loop into ExRatatui.Command.async/2:

# Off the event loop because `set_position/4`'s `:pubsub` delivery is a
# `GenServer.call`: inline it would stall the terminal-owning process on
# every jog keypress and exit it if the actuator is dead. The async runner
# traps that exit and delivers it as `{:error, {:exit, reason}}`.

Note the two reasons, and that only the second is about dying. {:error, :timeout} removes the death but not the 250 ms stall in the process that owns the terminal — so bb_tui keeps that async hop either way. For the callers this issue is most worried about, the problem is blocking, not exiting. An error tuple fixes the smaller half.

What it would fix for bb_tui is the shape of the answer: it currently logs an opaque {:exit, reason} and cannot tell "wedged, probably still moving" from "dead, definitely didn't".

:noproc

Different answer from :timeout, but not for the reason the issue gives. "A missing actuator is a wiring bug" doesn't survive contact with the code:

• Wiring bugs never reach :noproc. An unknown atom name raises ArgumentError in actuator_path!/2 before anything is sent, listing the known actuators. In BB.Motion the path comes from the joint's own actuator list, so it can't miss. The one exception is a bogus list path — actuator_path!(_robot, path) when is_list(path), do: path passes lists through unvalidated (lib/bb/actuator.ex:969), so set_position(Robot, [:nope], 1.0) publishes to a dead topic and then exits :noproc. That asymmetry looks like its own small bug, worth closing whatever else happens here.
• A declared actuator is always started. I chased the simulation: :omit hypothesis and it doesn't apply: simulation: is only on the controller and bridge entities (lib/bb/dsl.ex:952, :1223), not actuators. Under simulation BB.Process swaps the callback module for BB.Sim.Actuator, but BB.Actuator.Server still starts and still registers under the same name. So no, simulation: :omit doesn't make a missing actuator normal — though it does mean a DSL-declared controller isn't running to be the caller, which is a separate trap already documented.
• So :noproc on a declared actuator means one of: the driver's init/1 returned :ignore — which c:BB.Actuator.init/1 documents as supported (":ignore to skip this actuator"), and the joint supervisor is :one_for_one, so it stays absent for the robot's lifetime; or the process is inside a supervisor restart window; or the tree isn't up yet. All runtime conditions, and the first is a deliberate, supported configuration.

So both should become values. But they are semantic opposites on the axis that matters, which is the argument against a single {:error, :timeout} spelling:

• :noproc — definitely did not happen. Nothing received it. Note send_command/4 publishes before it calls, so on :noproc the command appears on [:actuator | path], every observer and log records it, and no actuator ever executes it. That's a worse lie than the exit.
• :timeout — probably did happen, or is about to. Per the experiment above.

A caller that can't tell those apart can't do anything sensible with either.

What shape says that honestly

Not {:error, :timeout}. A bare atom named after the waiting reads as failure, and AGENTS.md:172 says so directly: "Prefer structured errors over tuples — use {:error, %BB.Error.State.NotAllowed{}} instead of {:error, {:not_allowed, reason}}."

bb has also already answered this exact question once, at its other main entry point, and the spelling is the part that went wrong. BB.Command.await/2 (lib/bb/command.ex:247) catches :noproc, :timeout and arbitrary exits and returns {:error, {:command_failed, reason}}. Two consequences:

1. "GenServer.call/3 semantics are what an Elixir developer expects" is already not what bb does when a caller awaits a command. The precedent for turning these exits into values is in-tree.
2. Its bare-atom shape is actively costing downstream work. From bb_jido/lib/bb/jido/action/command.ex:115: "BB.Command.await/2 pre-wraps infrastructure failures (crash, :timeout, :noproc) as {:command_failed, reason}; unwrap before mapping so those…". bb_reactor hand-rolls its own monitor-based equivalent rather than use it.

So: two BB.Error types in class :state, distinguished by whether the command was received, and using the severity protocol to carry the distinction — which is what it's for, and there's precedent in Estimator.StaleInput / SyncMiss for :warning meaning "unusual, not a failure":

# severity :warning — the operation has not failed
defmodule BB.Error.State.CommandPending do
  use BB.Error, class: :state, fields: [:actuator, :command, :timeout_ms]
  # "servo has not answered Position within 250ms; the command is queued and
  #  will most likely still be applied — this is not a rollback"
end

# severity :error — nothing received it
defmodule BB.Error.State.ActuatorNotRunning do
  use BB.Error, class: :state, fields: [:actuator, :path]
end

CommandPending is deliberately not called Timeout (and there are already two unrelated Timeout types, Hardware.Timeout and State.Timeout, so the name is taken twice over). It's named after the state the command is in, not the state the caller got bored in, because the whole risk of this change is callers reading it as a rollback.

Recommendation

Three things, only the first urgent, and I'd do it even if the answer to the headline question is no.

1. Fix BB.Motion's fan-out. This is a live bug, not a contract question. Task.async makes the exit un-catchable by the caller, bb_mcp already ships a workaround that silently doesn't work, and the docstring on set_position/4 recommends the pattern to users ("Commanding several joints") while correctly warning that Task.async links. Collect with monitors, or Task.Supervisor.async_nolink. Independent of everything below.

2. Then yes — convert both exits to values, spelled as above. The two structured errors, caught in send_command/4. And separately (breaking, its own change) retro-fit BB.Command.await/2 off {:command_failed, atom} onto the same types, since it's the same question already answered badly in a way two packages unwrap.

3. File the coalescing gap separately; don't let it ride along here. "Latest target wins per actuator" is what three packages are approximating in client code, and it's the actual fix for the backlog. Conflating it with this would sink both.

Docs-only is not enough, but the docs point in the issue is right and independent: set_position/4's exit paragraph is one sentence buried under ## Returns after the :timeout option, while the :direct-returns-:ok gotcha gets a full .warning admonition. The thing that kills your process deserves at least the styling of the thing that misleads it. Same for BB.Motion's three copies.

The strongest argument against my own recommendation

The one I found rather than the ones in the issue: converting the timeout to a value makes the unbounded-backlog failure mode reachable, where the exit currently prevents it. Today a loop that meets a slow actuator dies. With an error tuple it survives, keeps sending, and the actuator sweeps through a growing queue of obsolete targets. Dying is loud and safe; surviving-and-lagging is quiet and dangerous, and on a robot arm that difference is physical. If I'm honest, that argument alone would justify closing this issue as "working as intended, document it harder."

I still land on the change, for three reasons:

• The exit doesn't actually prevent it. A supervised loop restarts. At a 5 s timeout, one crash per 5 s never trips a default 3 restarts in 5 s intensity, so the loop comes back and resumes sending — you get the backlog anyway, plus lost loop state, plus a crash report per cycle. bb_policy's Runner is exactly this: handle_info(:tick, …) → Step.run → Effect.apply → set_position, no catch anywhere, 20 Hz default, 5 s default timeout. Under start_link/1 it dies and restarts forever; under run/4 (unlinked + monitor) the caller gets {:error, {:timeout, {GenServer, :call, [Actuator, {:command, %Message{...}}, 5000]}}} — a raw exit tuple with the whole message struct in it — and finish/2 never runs, so the telemetry and {:episode_result, …} paths are bypassed entirely.
• :direct already has the identical unbounded queue. The framework isn't currently protecting anyone from this; it's protecting one path and shipping the other as the recommended escape.
• A named :warning-severity error makes the backlog visible, which is the precondition for handling it. An exit tells the caller nothing it can act on. The fix for stale accumulation is coalescing, not death — and death isn't buying the safety it appears to.

Churn

Smaller than it looks, because the return type doesn't change. @spec set_position(...) :: :ok | {:error, term()} already admits both structs, so this is additive, not breaking.

• bb: two error modules, one catch in send_command/4, the BB.Motion fan-out fix, docs across set_position/4 + BB.Motion's three copies + usage-rules/actuators.md. Call it ~40 lines of code and a bigger docs diff. No ! needed.
• bb_kino, bb_liveview: work unchanged and better — both already do {:error, reason} when is_exception(reason) -> Exception.message(reason), so they'd start rendering a real message where they currently render a hand-built string. Their catch blocks become dead code, deletable whenever.
• bb_policy: works unchanged. Step.applied/4 already finds the first {:error, _} and returns {:error, {:actuator, reason}}. It stops dying, which is the whole point, and it never has to invent the timeout it declined to invent.
• bb_mcp: its broken catch becomes unnecessary, and it stops reporting failure for commands that ran.
• bb_tui: keeps the async hop (needs it for the stall, not the death), swaps {:error, {:exit, reason}} for a real error in the event log.
• bb_reactor / bb_jido: untouched unless the BB.Command.await/2 retro-fit rides along — which is why that should be its own change. Both match on {:command_failed, reason} today.

So: zero downstream changes required to keep working, four packages able to delete code at their leisure, one genuinely breaking follow-up that can wait.