fctl-agent's lock test can fail when another test starts a process #415

What is wrong

protected::tests::finds_who_holds_a_lock in fctl-agent (added in #409) can fail now and then for the reason #413 found in fctl-host's image store lock test, so cargo test -p fctl-agent (and CI's Rust tests) can fail without any change to it. Nobody has seen it fail yet, but the window is there: in a throwaway stress test, where each round took link::lock on a new path, dropped it and asked held again while two threads ran std::process::Command::new("true").status() in a loop, 168 and 179 of 10,000 rounds found the lock still held right after drop(lock), where the test asserts !held(&path).

How it works now

link::lock (crates/fctl-agent/src/link.rs) takes an flock with File::try_lock and returns the File, which lets go of the lock only by closing. An flock belongs to the open file description, and a process another test forks at that moment (fctl-agent's tests start git, sh and scripts) shares it until it execs, close-on-exec descriptors included. When the fork lands between link::lock and drop(lock), the lock outlives the drop until that process execs, and held finds it held. link::tests::a_second_supervisor_leaves_the_socket_alone retries its claim for this reason (#84); this test doesn't.

What should happen

• The test passes whatever other tests start (fix(agent): let go of the supervisor's lock even while a command starts #417). fctl-host's image store lock test fails now and then #413's fix for fctl-host's image store unlocks before the file closes (a guard whose Drop calls File::unlock), which releases the lock whatever copies of the file exist, and its test keeps a copy of the holder's file open across the drop, so that it fails every time without the unlock rather than now and then. The same would do here.

Compatibility

Tests, and how the supervisor lets go of its lock: no protocol or config change, nothing to release.

Session: https://app.fctl.sh/s/ses_01M6RQNKSZVECTTV7F8RA0FCYF