Implement masking

[JorisGoosen]

The code in #220 actually looked like a good base to implement 2d masking, which I wanted opensplat to support for a while.
Ive fixed some loose ends in it and changed the application of the mask.

Implements #35

Ive tested this with some datasets of mine, any suggestion on other things to test would be welcome.
So far it seems to be working fine.

Any suggestions for possible datasets to test things on?

[pierotofy]

Thanks @JorisGoosen. Can you test with https://drive.google.com/file/d/1XzRI7kKKx3hvql7KKhvaIWGRwebltm9N/view?usp=sharing and post some screenshots?

[JorisGoosen]

After 9000 or so iterations:


During:

Ill let it run a bit longer now.
I should see about how close the masks fit the shapes.
Maybe I can show it in the visualizer somehow.

[JorisGoosen]

After 40000:
bananasLonger.ply.zip

[pierotofy]

Thanks Joris. You can see the rendering is no good. Compare the result with a run without masks and you'll see the difference.

But even just looking at the renderer should give a clue that this is not the right approach:

[JorisGoosen]

Well, the thing is, the part outside of the mask is not being updated (properly) now.

Maybe adding a very small loss to parts that are consistently masked might help?

Ive been running with a less tight mask and I get better results then.

Edit: definitely need to tweak this, but just have to find what to tweak.

[JorisGoosen]

So with a bit of a wide mask its already almost usable for me.
But I agree that something needs to be done to also make the banana one come out well (because I am also planning to use such masks soon anyway)

[pierotofy]

Right, one cannot simply zero out the loss around the mask.

I would start by looking at how other implementations do it for possible approaches (e.g. https://github.com/MrNeRF/LichtFeld-Studio/blob/master/src/training/trainer.cpp#L246)

I think depending on the approach you'll need to track the outAlpha in the backward step (https://github.com/nerfstudio-project/gsplat/pull/70/changes) as well, as I currently always initialize it to zero:

tensor_list RasterizeGaussians::backward(AutogradContext *ctx, tensor_list grad_outputs){
//...
torch::Tensor v_outAlpha = torch::zeros_like(v_outImg.index({"...", 0}));

[JorisGoosen]

So last week I did look into the code at lichtfeld and seem to have a better idea on how to get this used.
Just need to find some time to work on it.

[JorisGoosen]

(Haven't forgotten about this, its just that im in the middle of moving houses)