Background
PR #597 proposes using GZip-kNN as a local request classifier. It compares a request with labeled examples using compression distance, then returns either a target score or an ambiguous result for a later classifier.
This may avoid some separate judge-model calls, but we do not yet have enough evaluation to know where it is a good fit for Switchyard.
There is also some product overlap with the prefill-router work. The implementations and cost profiles are different, but both use the request itself to choose a target without making a separate judge API call:
- #506 added Transformers feature extraction.
- #539 added batched prefill inference.
- #593 adds the libsy
Algorithm wrapper and is currently under review.
Questions
- How well does GZip-kNN route held-out and out-of-domain requests?
- How often does it avoid a judge call at different confidence thresholds?
- What happens to end-to-end task quality when it selects the less capable target?
- What are its latency and throughput as requests and training sets grow?
- How does it compare with the prefill router using the same requests, targets, and scoring?
The integration shape also needs to be clear. #597 exposes a Rust Classifier, but not a runnable libsy Algorithm, server/TOML route, or Python entry point. The FallThrough composition shown in the PR is not available through libsy's public API, and the adapter does not plug directly into the existing judge classifier as written.
@urirosenberg, please add any evaluation or benchmarking you have already run. Would you be willing to compare it with the prefill router once the integration in #593 settles?
Tagging @nachiketb-nvidia for context on the prefill work.
This issue is for collecting the evidence first. We can decide whether to take the implementation further after that.
Background
PR #597 proposes using GZip-kNN as a local request classifier. It compares a request with labeled examples using compression distance, then returns either a target score or an ambiguous result for a later classifier.
This may avoid some separate judge-model calls, but we do not yet have enough evaluation to know where it is a good fit for Switchyard.
There is also some product overlap with the prefill-router work. The implementations and cost profiles are different, but both use the request itself to choose a target without making a separate judge API call:
Algorithmwrapper and is currently under review.Questions
The integration shape also needs to be clear. #597 exposes a Rust
Classifier, but not a runnable libsyAlgorithm, server/TOML route, or Python entry point. TheFallThroughcomposition shown in the PR is not available through libsy's public API, and the adapter does not plug directly into the existing judge classifier as written.@urirosenberg, please add any evaluation or benchmarking you have already run. Would you be willing to compare it with the prefill router once the integration in #593 settles?
Tagging @nachiketb-nvidia for context on the prefill work.
This issue is for collecting the evidence first. We can decide whether to take the implementation further after that.