Any workflow for Shot-by-Shot Video Processing to Avoid Ghosting

[amrshank92]

Is there any way to process a full video—for example, a 5-minute music video—shot by shot? I tried processing it by splitting the video into 1-minute clips, but when it transitions from one shot to another, I get ghosting. Then I tried manually splitting the shots and running them individually. When processed shot by shot, the ghosting disappears, and the temporal consistency also seems much better.

[amrshank92]

naxci1

can you help me

[naxci1]

Hi @amrshank92

I didn't quite understand your question, which is why I didn't reply. What do you mean by processing frame by frame? Is it like batch_size=1? This actually reduces the video quality. It needs to be at least 33, or 45 for it to process at normal quality. I process videos by splitting them into 3-minute segments. I've shared workflow examples and video splitting/joining applications on this forum before; you should check them out.

[naxci1]

naxci1 Not frame-by-frame — it is shot-by-shot processing. For example, in a music video there will be multiple shots. So the idea is to process each individual shot one by one. One shot may be only 2 seconds long, another may be 10, 20, or even 30 seconds.

I tried this by splitting the clip shot-by-shot, but I had to manually load each file every time for processing. If I could simply input all the files in a folder into SeedVR and have SeedVR process them one by one automatically, it would be much better.

One more thing I want to mention: if FlashAttention is not installed, SeedVR processing with attention mode = SDPA runs without any OOM issues on a 3090 Ti, even with a batch size of 61. However, if FlashAttention is installed and I use attention mode = flash_attn, the processing becomes much slower compared to SDPA. But the strange part is that when FlashAttention is installed, even if I switch back to attention mode = SDPA, SeedVR starts giving OOM errors. So there’s some weird behaviour happening.

You mention Scene Detection, and they added it to Topaz. I tried it, but it doesn't work properly. There's no need for that. A higher batch size will ensure high-quality output. Regarding the other issue, you should enable debug mode and file the issue in the "Issues" section along with the debug log; technical issues are resolved there.

[amrshank92]

but on my 3090 ti, batch size more than 61 gives me OOM.

[naxci1]

but on my 3090 ti, batch size more than 61 gives me OOM.

You use 61, I set it to 57 when I upscale it by 4x.

[amrshank92]

can you please share your node setup

[naxci1]

can you please share your node setup

I'm sorry I didn't respond to this. I don't receive a notification when a reply is sent. Of course, the settings change when the intro video is different, so you have to adjust it yourself. I usually run 100-frame tests to find the optimal setting and then start the process. I've already shared the workflow.

[naxci1]

HB2k-SeedVR2.5-Auto-288p.json

Workflow templates (for SeedVR2.5)
#294

[pany860-cu]

A scene detection node is what he needs i think,i dont know there is such a node or not

[urixl]

But if you have a scene larger that 1 minute, your processing speed will be slower.

[Fenixdg3duy]

I ran different tests with an NVIDIA RTX PRO 6000 Blackwell 96GB card, 640x360 with a batch size of 125 and multiplied the resolution by 3 and 4. I still don't realize what causes the ghosting effect, but it happens. It's probably a very deep-rooted issue and won't be easy to eliminate.