fix: detect rate limits on streaming responses for proper failover - #60
Open
wagner-sousa wants to merge 3 commits into
Open
fix: detect rate limits on streaming responses for proper failover#60wagner-sousa wants to merge 3 commits into
wagner-sousa wants to merge 3 commits into
Conversation
Remove the !isStream guard from processProxyResponse so rate limits are detected regardless of whether the response is streaming or not. Add a fallback check for HTTP 429 status code to handle cases where parseRateLimit may not return isRateLimited for SSE error responses. This fixes the issue where Claude Code streaming requests that receive 429 rate limit errors from Anthropic were not triggering account failover, causing the error to be forwarded directly to the client.
- Add response.body?.cancel() before failover in proxy-operations.ts - Add response.body?.cancel() before failover in compat/handler.ts - Fix type error in response-processor.test.ts (remove null values) - Update build script to compile worker binary separately This fixes the memory leak that caused Bun to crash with 34GB RSS when streaming responses were not consumed during failover.
Anthropic OAuth rate limits typically last ~30 minutes, not 5. The previous fallback was too short and caused accounts to be unblocked prematurely in ccflare's view while still rejected by the API.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When Claude Code makes streaming requests to Anthropic through ccflare and receives a 429 rate limit error, ccflare fails to:
Detect the rate limit - The
processProxyResponsefunction only checked for rate limits on non-streaming responses (if (!isStream && rateLimitInfo.isRateLimited)), so streaming 429 responses were silently ignored.Trigger failover - Since the rate limit wasn't detected, the account was never marked as rate-limited, and the failover loop never moved to the next account.
This caused the 429 error to be forwarded directly to the user instead of automatically retrying with another available account.
Solution
Removed the
!isStreamguard fromprocessProxyResponseso rate limits are detected regardless of response type.Added a fallback check for HTTP 429 status code to handle cases where
parseRateLimitmay not returnisRateLimitedfor SSE error responses.Changes
packages/proxy/src/handlers/response-processor.ts: Fixed rate limit detectionpackages/proxy/src/handlers/response-processor.test.ts: Added test for streaming 429 fallbackTesting
All existing tests pass. Added new test case that verifies 429 detection via status code fallback for streaming responses.