Skip to content

feat: server side streaming rather than a single response - #10

Merged
abelfx merged 1 commit into
mainfrom
feat/server-side-stream
Jul 31, 2026
Merged

abelfx merged 1 commit into
mainfrom
feat/server-side-stream

Conversation

@abelfx

@abelfx abelfx commented Jul 31, 2026

Copy link
Copy Markdown
Owner

This pull request introduces support for streaming inference responses, allowing clients to receive results incrementally from the server. It updates the gRPC API, client, worker, and proxy to handle streaming requests and responses, while maintaining backward compatibility with the existing unary Predict endpoint.

Streaming Inference Support

  • proto/inference.proto: Added a new PredictStream RPC to InferenceService and defined the StreamChunk message for streaming tensor data and signaling the end of the stream.
  • src/bin/client.rs: Updated the client to use the new predict_stream method, processing and printing streamed chunks as they arrive.
  • src/bin/worker.rs: Implemented the predict_stream handler in GpuWorker, which splits input data into chunks, processes them, and streams results back to the client with an is_final flag.

Proxy and Load Balancing Enhancements

  • src/proxy/server.rs, src/proxy/mod.rs, src/main.rs: Updated the proxy to forward streaming requests directly to backend workers, using round-robin load balancing across available worker clients. The proxy now supports both unary and streaming inference endpoints.
    Dependency Updates

  • Cargo.toml: Added the tokio-stream dependency to support streaming response handling.

@abelfx
abelfx merged commit ebd6f48 into main Jul 31, 2026
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant