-
Notifications
You must be signed in to change notification settings - Fork 1.5k
fix(client): keep request timeout armed across body-read phase #1826
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -243,6 +243,62 @@ import { isEmptyObj } from './internal/utils/values'; | |
|
|
||
| const WORKLOAD_IDENTITY_API_KEY_PLACEHOLDER = 'workload-identity-auth'; | ||
|
|
||
| /** | ||
| * Wrap a `Response` so that the given `clearRequestTimeout` callback runs | ||
| * exactly once, when the body is either fully read, cancelled, or errors. | ||
| * Responses without a body have the timeout cleared synchronously. | ||
| * | ||
| * This lets the request-timeout timer stay armed across the body-read phase | ||
| * (e.g. `await response.json()`), which native `await fetch()` does not cover. | ||
| */ | ||
| function wrapResponseForRequestTimeout(response: Response, clearRequestTimeout: () => void): Response { | ||
| if (!response.body) { | ||
| clearRequestTimeout(); | ||
| return response; | ||
| } | ||
|
|
||
| let cleared = false; | ||
| const clearOnce = () => { | ||
| if (cleared) return; | ||
| cleared = true; | ||
| clearRequestTimeout(); | ||
| }; | ||
|
|
||
| const originalBody = response.body; | ||
| const wrappedBody = new ReadableStream({ | ||
| async start(controller) { | ||
| const reader = originalBody.getReader(); | ||
| try { | ||
| for (;;) { | ||
| const { done, value } = await reader.read(); | ||
| if (done) break; | ||
| controller.enqueue(value); | ||
| } | ||
| controller.close(); | ||
| } catch (err) { | ||
| controller.error(err); | ||
| } finally { | ||
| clearOnce(); | ||
| try { | ||
| reader.releaseLock(); | ||
| } catch { | ||
| // reader may already be released | ||
| } | ||
| } | ||
| }, | ||
| cancel(reason) { | ||
| clearOnce(); | ||
| return originalBody.cancel(reason); | ||
|
Comment on lines
+289
to
+291
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
In Useful? React with 👍 / 👎. |
||
| }, | ||
| }); | ||
|
|
||
| return new Response(wrappedBody, { | ||
| status: response.status, | ||
| statusText: response.statusText, | ||
| headers: response.headers, | ||
| }); | ||
| } | ||
|
|
||
| export type ApiKeySetter = () => Promise<string>; | ||
|
|
||
| export interface ClientOptions { | ||
|
|
@@ -887,12 +943,22 @@ export class OpenAI { | |
| fetchOptions.method = method.toUpperCase(); | ||
| } | ||
|
|
||
| let response: Response; | ||
| try { | ||
| // use undefined this binding; fetch errors if bound to something else in browser/cloudflare | ||
| return await this.fetch.call(undefined, url, fetchOptions); | ||
| } finally { | ||
| response = await this.fetch.call(undefined, url, fetchOptions); | ||
| } catch (err) { | ||
| clearTimeout(timeout); | ||
| throw err; | ||
| } | ||
|
|
||
| // Keep the timer armed until the body is fully read, cancelled, or errored. | ||
| // The `await fetch()` above resolves as soon as response *headers* arrive, so | ||
| // clearing the timeout here would leave subsequent body readers — e.g. | ||
| // `await response.json()` in `internal/parse.ts` — without any timeout. | ||
| // Servers that flush 200 headers fast and then stall mid-body would cause | ||
| // the SDK to hang indefinitely. See openai/openai-node#1825. | ||
| return wrapResponseForRequestTimeout(response, () => clearTimeout(timeout)); | ||
| } | ||
|
|
||
| private async shouldRetry(response: Response): Promise<boolean> { | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The wrapper reads the upstream
response.bodyto completion insidestart(), which runs eagerly whennew ReadableStream(...)is constructed, not when the caller pulls. Because this loop never checks downstream demand, slow consumers (especiallystream: trueSSE users that process events incrementally) can accumulate an unbounded in-memory queue and lose the original backpressure behavior offetchbodies, leading to large memory growth or OOM on long streams.Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.