I've implemented streaming inference, is there interest in a PR?

[CorentinJ]

Hello,

I would I have liked to make this an email or a Discussion tab, but neither options were available. Sorry for creating an issue instead.

As part of a recent project I have created, I have implemented streaming inference for Kokoro TTS. You can view the implementation in this web demo.

On CPU, 90% of the inference time is spent on streamable ops, so the Time To First Sound can be decreased heavily. This without any major refactor, retraining or by using sentence splitting.

I'd be happy to build a PR to incorporate the streaming into this repo, but this is quite a bit of additional work given it would be a new API. Therefore I would only do it if there is genuine interest in the PR being merged.

[andrewfenn]

I would be interested in this. Your live demo doesn't seem to be online.

[CorentinJ]

Do you mean that the website was down at the time you checked or that the demo does not show streaming playback? If it's the latter, I didn't include it because it was quite a bit of work for showing something that is technically redundant - but I get that it misses on a lot of the wow factor to not be able to hear the lower streaming latency.