Skip to content

Add a batch-asynchronous version of suffix cache updateΒ #205

@sfc-gh-aqiao

Description

@sfc-gh-aqiao

Suffix decoding overheads start becoming an issue at high concurrencies. For the update operation, we should add a batch version that can run asynchronously to it can be overlapped with the main decoding loop.

Discussion: vllm-project/vllm#25784

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions