Bottleneck in AI-powered upscale

[naxci1]

There's been a bottleneck in AI-powered upscale tools for the last 4-5 months. None of them are receiving any new features or updates. It seems like they're waiting for new methods or Nvidia updates. However, the full potential of Nvidia's 50xx GPUs isn't being utilized yet. SeedVR 2.5, FlashVSR, Topaz, and others are all stalled. Nvidia will probably release their 60xx GPUs in another two years, meaning there won't be any new features for another two years?

What are your thoughts and opinions?

[Fenixdg3duy]

@naxci1 I feel the same way, and it's concerning—for those who want to stay on the cutting edge, there's nothing new under the sun.

[amrshank92]

You tried this ?
https://sparkvsr.github.io/

[naxci1]

Hi @amrshank92

taco-group/SparkVSR#2

I actually wrote a question here before, but I haven't fully tested it yet. I downloaded it and then deleted it; it requires a lot of installation and the model files are excessively large. Yesterday I checked and found that there's a VRAM issue. It's been tested on A100 GPUs, but there aren't any tests for end-user GPUs. I plan to try it at the first opportunity.

For example, in the simplest tests:

360p (480x360) -> 720p (960x720) [2x upscaling, 192 frames]: ~32.52 GB VRAM

I don't have that much VRAM. So I gave up on trying it, but then a reply said it's possible to try it optionally. I haven't tried it yet. If anyone tries it and shares their experiences, we'd appreciate it.

Actually, it would be great if @IceClear examined this repository; I think it would be useful for SeedVR3 development.

[naxci1]

naxci1/SparkVSR#1

I'm currently optimizing SparkVSR for 50xx GPUs, I've separated the training section, made it easier to use, added conversion to the GGUF model, and included a SeedVR2-style Comfyui interface. I'll let you know when the tests are complete.

[naxci1]

This application doesn't work for end-user graphics cards. I ran tests, and the model files are so large they occupy 70GB of RAM, and 16GB of VRAM is insufficient, so it's overflowing RAM. A 10-second video took over an hour to complete, so I stopped it. Currently, it's useless on our GPUs; it requires at least 80GB of VRAM. It's only possible to test it on cloud servers, which is outside my area of ​​interest. Since it failed in real-world tests, I can't comment on the quality because there are no results. As for slowness, even SLM is faster than this; its speed is 0fps.

Maybe it will be developed for PCs in the future, then we'll have a chance to test it. For now, I'm continuing with SeedVR2.

[naxci1]

https://x.com/AIWarper/status/2035030733144957360

A real test was conducted here, read the reviews.

[naxci1]

When Topaz Starlight was released, they made such exaggerated claims in their advertisements and showed unrealistic results in the demos, but in real tests we saw that none of this reflected the truth.

We experienced the same thing when FlashVSR came out, and now I'm seeing the same thing with SparkVSR.

[amrshank92]

When Topaz Starlight was released, they made such exaggerated claims in their advertisements and showed unrealistic results in the demos, but in real tests we saw that none of this reflected the truth.

We experienced the same thing when FlashVSR came out, and now I'm seeing the same thing with SparkVSR.

True. SeedVR2 was the only one where the results actually matched 90% of their demo claims.

[naxci1]

https://github.com/smthemex/ComfyUI_SparkVSR_SM

@smthemex has now shared the one for Comfyui; I haven't tried it yet, but I'll try it at the first opportunity. Those who have used it can also share their results.

By the way, thank you @smthemex for preparing it so quickly.

[sowbran]

HI! What about this, do you have try it?

https://github.com/smthemex/ComfyUI_LucidNFT

[naxci1]

Hi @sowbran

I checked the repository, and it only provides improvements for images, not videos.

[zelenooki87]

Hey @naxci1 , did you end up getting SeedVR2 running successfully in FP4 mode? How far did you get with it?

Also, any word on whether @adrientoupet will be back soon? Any updates on that?

Thanks!

[naxci1]

I ran it, converted the model, but gave up because the speed difference was too small. It's currently in my repository; I'm waiting for SeedVR3, so I'm not waiting for it.

I haven't heard from @adrientoupet; he said it would be coming between March and April, I'm waiting.

@IceClear said he would be working on SeedVR3. We're waiting; none of the new ones are as high-quality as SeedVR2.5. The SeedVR2 model is really good; it would be great if they could just speed up the VAE.

[adrientoupet]

Hi team - I'm tied up for another week or so - I hope I can help clear some of the backlog after. Thanks for your patience

[HKunz1969]

Thank goodness for a sign of life from you!

[zelenooki87]

@naxci1 I just tested SparkVSR on a few older clips, and I still think SeedVR2 is convincingly the best overall. SparkVSR is interesting from a scientific research perspective, but it depends heavily on the quality of the reference images. Using the API with Nano Banana would likely improve the results, as the authors mention in their paper. When my reference images were blurry, practically the entire clip suffered and the output came out poor and blurry. For a 40-second clip at 320×240 input, upscaled 4× to 1280×960, it took about 8 and a half minutes.

SeedVR2 seems to me like the best option out there right now — better than FlashVSR, better than all the commercial solutions that run locally, and it even looks better to me than Topaz AstraHQ. I think the batch CLI script for FFmpeg could use some improvements though — things like preserving audio, handling variable frame rates, supporting more codecs, and so on. I'm confident Adrien will tighten that up and get it sorted.

Do you happen to have any info or any hints on when we might expect SeedVR3? Could it come by mid-year, or is it more likely toward the end?

By the way, you can keep track of other related work on this page
https://github.com/yjsunnn/Awesome-video-super-resolution-diffusion
— they regularly update the list.

[amrshank92]

Any chance for speed improvement on 3090ti. Currently getting only .65 fps. Batch size is 57(Maximum 3090ti Supports)
Currently using naxci1's 2.5 template.

[naxci1]

Hi @zelenooki87

The situation is very bad; it uses such large model files. I was still hopeful, but according to real tests, it's not good at all. As I said, they only show good results in demos, but we don't see that in real tests. FlashVSR and Topaz Video are also basically deceiving people with fake demos. I've used almost all the paid and free applications so far, and SeedVR2.5 is unrivaled in this regard.

I also tried Topaz's new Starlight models in the new 1.3 model; they're terrible. The Fast 2 gives completely corrupted output, and the other one is even lower quality than SLM at 0.6fps. They're doing pointless work; they can't develop anything new, they're just stringing along old customers. The AI ​​sector has been stagnant for a long time; there's no innovation.

I use Video Combine as output in SeedVR2.5; it already allows for more codecs and bitrates, and the merging is quite high quality. I don't merge the audio because I edit and clean it up, adding the audio at the end with the video editor during color correction.

We're waiting for SeedVR3, let's see when it will be released. The only issue with SeedVR2.5 is VAE acceleration; it would be great if that were fixed.

[cd1188]

SeedVR2.5 is the best model so far, but I'm waiting for a fix for the ghosting problem. I'm upscaling 160p clips with 15fps and its a big problem for me. My workaround is to convert it to 60fps with 4 identical frames in a row and remove 3/4 of the frames afterwards. its a waste of time to render that way. Hope SeedVR3 does fix this.

[naxci1]

Hi @cd1188

I also take 15fps videos and then use FrameGUI to upscale them to 30 or 60fps; it does it with AI, and it works very well.

[cd1188]

Hi @naxci1
I'm using RIFE after the upscale, because If I interpolate my 160p15 mpeg1 clips before SeedVR, the upscaled video has a small image wobble in the details, because of the low resolution. The ghosting problem is a bit better but still visible..
SeedVR2.5 runs really well on 160p, but the ghosting gives me headache.

I did some tests to find a workaround and wrote here some things and ideas:
#187 (comment)

[naxci1]

Hi @cd1188

I've switched to a different method now:
SeedVR2 - Topaz Iris 3 - FrameGUI

This way, motion issues are also better resolved.

What is your batch_size? If it's too small, it causes ghosting. The larger it is, the higher the output quality.

[cd1188]

batch-size was 45 ... you can see it at my comment here #187 (comment)
With splitting the input video to tiles, I have now a batch-size of 121 so I don't lose that much temporal consistency with the 60fps input.

Info: 60fps so I have 4 identical frames in a row because SeedVR is using a 4-frame-window to analyse the video?
This avoids ghosting for me... Maybe its possible to deactivate this 4-frame behavior from SeedVR as a fix/option?

I'm experimenting with this:
160p15 -> convert to 60fps without interpolation -> SeedVR2 960p60 (batchsize 45 - temporal overlap 1) -> reduce framerate to 15
-> using RIFE to 30fps -> TOPAZ-Reha

[naxci1]

Here's what you're doing wrong:

At low FPS, upscale with SeedVR2, then sharpen with Topaz Iris 3, and finally increase FPS by 2x or 3x with FrameGUI.

[cd1188]

This workflow does not work for me ... then I have ghosting...
I can make a simple test animation with a red circle moving on a white backgroud and then you can see the ghosting.. if you convert the fps 4x with duplicating frames, SeedVR does not produce ghosting... but it has also other downsides.

I put a screenshot on my comment here so you can see my problem.. #187 (comment)