Take output of neural network as reference along with neighbour pixels

[gordinmitya]

Hello @ZQPei and @zvezdochiot Thank you for your work!
I wonder if is it possible to improve results of deep learning approach with patch-match algorithm. It seems to me neural network should better understand the content of a gap in image. Unfortunately methods that we have tried output quite blurred texture. In contrast patch match is good at giving sharp textures.
How do you think will patchmatch work on neural network output? And where should I start in order to modify the code to take not only neighbour pixels but also pixels under the mask?

Here are some examples where neural network could improve in-painting results:
left – original; center – neural network; right – patchmatch (this repo);

notice border of column

deformation of road

smth brown in the water and creepy copy of the guy's hand

Thank you in advance for any help and thoughts!

[zvezdochiot]

I’m not sure that the neural network makes sense (it will give a result on what it was trained on). There are more standard (but no less expensive) solutions: https://github.com/zvezdochiot/inpaint-cimg

[ZQPei]

Hi, @gordinmitya,
Deep learning based methods are now the mainstream in image inpainting. The first paper introduces CNN to do inpainting is Context Encoder. Then more and more researchers joined in and proposed better approaches, such as context attention, partialconv, edgeconnect, biattention map, etc.
PatchMatch and its countparts, which are exemplar based methods, are really out of date and way slow.
Here is a repo for a quick view on the current popular deep learning based works on image inpainting, and it is worth to take a watch.

[gordinmitya]

Thank you for your ultra fast response!
@zvezdochiot as long as we have very large datasets like places2 (10+ million photos) and CelebA (200+ thousands of faces) we can expect to cover many situations.
But as I mentioned above – on our test images it produces blurred results. We have tested a lot of works from list @ZQPei refer (including most recent works like PEPSI++ 2019 and Region Normalization 2020).
So I want to combine best of two worlds: understanding of content from DL and sharp textures from patch-matching.
Please, point me where should I modify your code in order to find patches similar to network output. And I will share results with you!

[zvezdochiot]

Not so simple. The main code (https://github.com/ZQPei/patchmatch_inpainting/blob/master/src/inpaint.cpp) is borrowed (link bottom README).

I fixed main.cpp and around it, without getting into the main code. Perhaps @ZQPei climbed deeper.

[ofirkris]

@ZQPei How can this code be modified to support post-processing on DL approach and sharp textures from patch-matching?

[ZQPei]

@ofirkris
To insert patchmatch into post-processing on DL approach, I think we should convert this code to python version first or provide a python api first.

[ZQPei]

@gordinmitya
I think there is a conflict between patchmatch and the DL method, don't they?
It would make any sense to do patchmatch from the deep feature maps of CNN methods learned from large dataset.
If you want sharp edges for CNN methods, better add l1_loss during training.

[ZQPei]

@gordinmitya
Combining exemplar based and neural based methods is a good idea, and I think this paper did some research on it. What do you think?
Shift-Net: Image Inpainting via Deep Feature Rearrangement

[zengxianyu]

Since the first step of this code uses the extremely downscaled input as both target and source, I thought that we may achieve this by passing the output of neural network and mask to patchmatch with a larger initial scale and remove the random search. However, the results I got are not good. I'm not sure was I doing that right, I just set the radius in main.cpp to 10 and commented out the random search in nearestneighborfield.cpp.