timely-util: benchmark async operator scheduling against a tokio task

[petrosagg]

Async operators built with builder_async run their logic future on the timely worker, so every wake of the future is a timely activation: unpark, step, schedule, poll. The premise behind a prototype tokio driven operator builder was that this is far more expensive than a tokio task wake, so a future woken at a high rate would be cheaper to run as a tokio task that ships its output back to the worker through channels. This benchmark tests that premise.

The logic future awaits items from a bounded tokio channel and does nothing else while holding its capability, so only scheduling is measured. The timely variant is a builder_async operator. The tokio variant spawns the future as a tokio task next to an operator with a pending logic, which is all the worker half of such a builder would do for a logic that sends nothing. Items come from a plain OS thread or from a tokio task, which decides whether tokio's same thread wake path applies. A parked worker sleeps between activations and must be woken through the OS, as an idle production worker does, while a busy worker steps continuously, as a production worker with other dataflows to run does. Per item, medians of 5 runs of 200k items, channel capacity 1, on an Apple Silicon laptop:

producer      worker    channel     timely    tokio
OS thread     parked    1 slot      5.5 µs    7.9 µs
OS thread     busy      1 slot      2.5 µs    7.9 µs
tokio task    parked    1 slot      8.3 µs    77 ns
tokio task    parked    64 slots    142 ns    149 ns

Findings:

• Timely is not the expensive side of a remote wake. Both paths are dominated by parking and unparking a thread, timely's plain thread park is the cheaper one, and a busy worker is 3x cheaper still. Worker count does not matter, and generating the six introspection events per wake is free.
• Tokio wins only for wakes born on the runtime, where it keeps producer and consumer on one thread and never parks, by 100x per wake. A timely driven future crosses threads twice per wake however cheap each crossing is. Timely's idle spin, added after 0.31, removes the park and brings the gap to 40x.
• Buffering closes the gap. With a channel capacity of 48 or more the timely variant is equal or cheaper per item for both producers, because the park is amortized over the items each wake drains.
• An earlier revision of the harness measured the prototype's data path at 4x the wall time and 9x the CPU of a direct container push per record, and the consumption of introspection events at about 1.5 µs per wake on the timely side.

Verdict: We should hold off from building the tokio driven builder. For wakes from outside tokio the timely worker is the cheaper side in every configuration. For wakes born on tokio the saving is a few microseconds per wake, which only adds up at tens of thousands of uncoalesced wakes per second. That rate needs a producer that is gated on the consumer for every single item, which is not how the network or any other resource naturally presents itself: a socket delivers a segment per readiness event, persist delivers a batch per listen, and a channel drains everything queued per poll, so wakes coalesce on their own as soon as the rate is high enough to matter. An operator that is nonetheless woken per item has a producer that fails to batch, and a 64 slot channel in front of it fixes that in isolation. What would reopen this is a production operator with a high scheduling rate and tiny elapsed time per scheduling, which mz_scheduling_elapsed and the operator duration histogram show directly.

Closes CPU-240