GenAI: clarify what 'chunk' means in time-to-chunk metrics

[lmolkova]

This PR talks about "chunks". What is the definition of "chunk"? Is that only about assistant content? Does it include reasoning if the model makes a distinction between reasoning and response text? Does it include any notification from the service, like a function call request (or a part of one)? Etc. My assumption is it's any packet of data from the llm, such that each update produced as part of a streaming implementation, regardless of what that update contains, counts as a "chunk".

Originally posted by @stephentoub in #3377 (comment)

[castlenthesky]

Any response would be considered a "chunk". The objective of the metric is to approximate the amount of time a request is sitting in a queue or waiting for the LLM to be provisioned for the response.

Some systems stream tokens while others stream chunks. I aligned at the term "chunk" to apply more broadly as a chunk is more coarse than a token.

Finally, yes, reasoning, tool calls, text, all would be considered a chunk. Service notifications would not, only generated text from the service should count (regardless of the type of text).

Your final assessment: My assumption is it's any packet of data from the llm, such that each update produced as part of a streaming implementation, regardless of what that update contains, counts as a "chunk". is correct and aligned with my intent for these metrics.

[lmolkova]

@castlenthesky would you be interested in adding a note on the definitions? I think it's important to capture in the convention itself

[stephentoub]

Service notifications would not, only generated text from the service should count (regardless of the type of text).

There are "packets of data" though that aren't text (unless you mean text as in the JSON a service might be sending if they're using a JSON-based protocol). Could be image data or audio data. Could be a packet of usage details (e.g. token counts). OpenAI Responses sends updates for things like "new output item added" before it sends any content for that output item. Services send updates indicating that various server-side operations are in progress, like it performing a search of some kind. I would expect all of those to count. I guess I'm wondering what do you mean by "service notifications" that wouldn't count?

Basically I'd expect that, for example, every server-sent events packet that I receive from a service using SSE would count as a chunk. Is that your expectation?

Thanks.