dig-peer-selector

Self-optimizing peer-selection middleware for the DIG Node. It tracks live network topology, learns each peer's real latency/bandwidth from actual download outcomes, and autonomously chooses the best peer subset for every multi-peer RPC/download request — minimizing P99 latency and avoiding thundering-herd on high-capacity peers, with no user-facing configuration knobs.

Status: implemented to SPEC.md (API version 1). The crate is the pure decision + learning layer; SPEC.md is the normative contract. The dig-node integration (wiring the selector between dig-dht and dig-download in digstore) is the remaining follow-up. Ships as a modules/crates/ submodule of the dig_ecosystem superproject.

---

What it is

A Rust crate, dig-peer-selector, that sits between the DIG Node's download path and the peer network. Given a content request it returns the best peer subset to fetch from; then it absorbs the actual throughput outcome of that fetch and gets smarter — a fully autonomous feedback loop that needs no manual tuning. It opens no socket, runs no discovery, and exposes no user-facing configuration knobs: every tradeoff (saturation point, relayed penalty, decay) is self-tuned from observed data.

Public API (frozen surface, SPEC.md §5/§11)

use dig_peer_selector::{PeerSelector, SelectorConfig, ContentRequest, Candidate,
                        Selection, TransferOutcome, RangePlanDelta};

let selector = PeerSelector::new(SelectorConfig::default());       // no behavior knobs
let selection: Selection = selector.select(&request, &candidates); // ranked subset + per-peer max_concurrency
selector.record_outcome(&outcome);                                 // measured result, folded in real time
let replacement = selector.rebalance(&request, &active, &need);    // mid-transfer re-query
// registry feeds: on_pool_event, on_connection_class, upsert_candidate, remove_peer
// read-only observability: peer_snapshot, snapshot, registry_size

What it does (all learned, measured-only, non-gameable)

1. Dynamic peer registry — fed by the discovery layers (gossip churn + DHT candidates), keyed by peer_id = SHA-256(TLS SPKI DER), bounded with lowest-value eviction; a reconnecting peer keeps its learned history.
2. Continuous per-peer learning — throughput / RTT / reliability learned from the REAL measured outcome of every transfer; observed capacity always overrides advertised (there is no input path by which a peer raises its own score).
3. Autonomous scoring — a learned per-class saturation point (anti-thundering-herd), an adaptive relayed penalty, and a decay derived from each peer's observed volatility (no baked constant).
4. Objective — minimize P99 request latency while avoiding thundering-herd collapse on a fast peer.
5. Closed loop — select → execute (dig-download) → record_outcome → rebalance, updating the models mid-transfer so an in-flight download re-balances.

Boundaries

• It does NOT do transport, NAT traversal, discovery, the DHT lookup, or the byte transfer — it only selects + learns. The sibling crates own those (see Integration).
• No user-facing tuning surface (read-only observability is fine; behavior knobs are not).
• Deterministic-testable: the learning + scoring are proven over a synthetic-topology + throughput-trace harness (no real network) in tests/conformance.rs — convergence, load-spread, degradation adaptation, P99 orientation, bounded exploration, seed-determinism, anti-gaming, and rebalance.

---

Integration with the existing DIG Node P2P stack

The selector is the "brain"; these crates are the "senses + hands". Wire it as middleware between dig-download and the peer layer.

• dig-dht (find_providers) — the CANDIDATE SOURCE. When the node wants content, dig-dht returns the set of provider peers (peer_id + addresses) holding it. Those candidates are the input population the selector scores/ranks. The selector filters + orders them; it does not query the DHT itself (the node does that and hands the providers in).
• dig-pex / dig-gossip — the LIVE TOPOLOGY FEED. The gossip peer pool + PEX peer-exchange keep a continuously-updated view of which peers exist + are connected (join/leave churn, connection class, via provenance). The selector's dynamic registry is fed from this pool (subscribe to pool churn events — dig-gossip already exposes subscribe_pool_events / connected_pool_peers / pool_stats). PEX-learned peers arrive as hints; the selector treats a peer's quality as unknown until it has real outcome data.
• dig-nat — the CONNECTION-CLASS + TRANSPORT signal. dig-nat established each peer connection via its ladder (direct → UPnP → NAT-PMP → PCP → hole-punch → relayed-TURN) and knows which method won + exposes it. The selector reads that connection class per peer (esp. "relayed" vs "direct") to seed the relayed-penalty learning, and the actual transfers ride dig-nat's mTLS mux streams.
• dig-download — the EXECUTOR + FEEDBACK PRODUCER. This is the tight loop: dig-download's multi-source range scheduler asks the selector for the best peer subset per download (and per re-balance when a source drops / a range needs relocating), executes the byte-range fan-out with pause/resume + per-chunk merkle verification, and streams every range's measured outcome (throughput, latency, success/fail, which peer, which range size) back into the selector. dig-download already re-queues a bad range to another provider — the selector should DRIVE that choice + LEARN from the failure.
• dig-node (in digstore) — the HOST that owns the instances and wires them together: it constructs the selector, feeds it the gossip pool + dig-nat connection classes, passes dig-dht providers in on each content-want, and hands the selector↔dig-download loop the content requests. The selector is a dependency of the node's content-fetch path (the same path #164/#165 build).

The end-to-end flow (what to implement to)

content want (store/capsule/root/.dig)
  → dig-node: dig-dht.find_providers(content_id)          → candidate peers
  → dig-peer-selector.select(content_req, candidates)     → ranked best peer subset
  → dig-download.download(req, subset, sink)               → multi-peer byte-range fan-out (pause/resume,
                                                             per-chunk merkle verify) over dig-nat mTLS mux
  → per-range/per-request outcomes stream back
  → dig-peer-selector.record_outcome(...)                 → update capacity models + scores in real time
  → next select() (and mid-transfer re-balance) is smarter — autonomous, no user tuning

---

Status of the deliverables

1. SPEC.md — DONE (normative, API version 1): the registry model, the per-peer capacity/quality model, the learned scoring quantities (saturation point, relayed penalty, volatility-driven decay), the min-P99 + anti-thundering-herd objective, the public API, and the feedback-loop contract with dig-download.
2. Implementation — DONE, TDD, over the synthetic-topology harness (tests/conformance.rs, no real network); the learning converges + adapts in the conformance tests (SEL-01..SEL-11).
3. CI — DONE, mirroring the sibling crates: ci.yml (fmt + clippy -D warnings + test + docs) + coverage gate (cargo llvm-cov --fail-under-lines 80) + tag-driven publish.yml (crates.io, CARGO_REGISTRY_TOKEN). Full crates.io Cargo.toml metadata; DIG-crate deps git-pinned like the others.
4. Integration — FOLLOW-UP: wire the selector into the dig-node content-fetch path (digstore crates/dig-node) as the source-selection seam of the selector↔dig-download loop, and document the selection layer in docs.dig.net (peer-network / download flow) + SYSTEM.md. See the implementers' note in src/lib.rs for exactly how dig-node embeds the selector.

Ecosystem conventions (followed)

• Peer identity is peer_id = SHA-256(TLS SubjectPublicKeyInfo DER) everywhere (re-used from dig-nat; same as dig-gossip / dig-dht / dig-pex).
• All node-to-node traffic is mTLS via dig-nat; the selector never opens its own transport.
• Backwards-compatible, additive; every change updates SPEC.md in the same unit of work.
• No commit footers / no co-authoring lines.