[Kept for Feedback] Multi-GPU & New models

[TiankaiHang]

Thanks for your nice work and congratulations on your good results!

I have several questions.

• Will your model extended to Parallel (distributed data-parallel) in the future.
• Why don't you try to use deeplabv3+, will it lead to a better result?

Best.

[voldemortX]

@TiankaiHang Hi! Thanks for your attention in our work!

• Will your model extended to Parallel (distributed data-parallel) in the future.

I'd love to do that, but I have never had access to a machine with more than 1 GPU in my entire life... So if you or anyone else can make a pull request to support this would be really nice...

• Why don't you try to use deeplabv3+, will it lead to a better result?

I believe a better model would lead to better results. But training V3/V3+ requires at least double the compute budget, that's why I did not do them. Also because the V2 results are still important for comparison against prior arts, that would lead to at least 3x compute budget if I chose to use V3/V3+ back then. I just do not have the cards.

Some additional info: On ResNet backbones, my experience tells me that V3+ could be worse than V3. For background: pytorch/vision#2689

[TiankaiHang]

Thanks for your kind reply~ :-)

Best.

[TiankaiHang]

Thanks for your kind reply~ :-)

Best.

[voldemortX]

You're welcome.
I'll pin this issue as a call for help to:

• Support multi-GPU training [RFC] Upgrade to torch 1.6 & ddp #9
• Support newer models such as DeepLabV3+

[jinhuan-hit]

I will update it for multi-gpus after I reproduce it. Maybe next week, now the time is not enough.

[voldemortX]

I will update it for multi-gpus after I reproduce it. Maybe next week, now the time is not enough.

That's great to hear! Go for it!

[lorenmt]

I would suggest checking out: https://github.com/huggingface/accelerate which should be relatively easy to deploy any model in a distributed setting.

[voldemortX]

@lorenmt Good point! Thanks a lot!
@jinhuan-hit If you're still working on this, Accelerate seems a good place to start. And it's perfectly ok if you don't want to send a PR just now. I'll update for multi-GPU myself when I got more bandwidth and cards for testing, it should be soon (when I get my internship).

[jinhuan-hit]

@lorenmt Good point! Thanks a lot!
@jinhuan-hit If you're still working on this, Accelerate seems a good place to start. And it's perfectly ok if you don't want to send a PR just now. I'll update for multi-GPU myself when I got more bandwidth and cards for testing, it should be soon (when I get my internship).

Yeah, thank you for your sharing and I am still working on this project. I'am sorry to say that I don't update it for multi-GPU until now. Something changed, I reproduce this project on another job. So the multi-GPU code is not match now. I'm trying to add multi-GPU on this project following Accelerate, today. Unfortunately, I have not solved this bug below.

RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by (1) passing the keyword argument find_unused_parameters=True to torch.nn.parallel.DistributedDataParallel; (2) making sure all forward function outputs participate in calculating loss. If you already have done the above two steps, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's forward function. Please include the loss function and the structure of the return value of forward of your module when reporting this issue (e.g. list, dict, iterable).

[voldemortX]

@jinhuan-hit Thanks a lot! I still don't have the hardware to debug multi-GPU for now. But hopefully I'll be able to debug this month/the next.
The problem seems related to network design, I don't remember I have additional (unused) parameters though, I'll check that later tonight and get back to you.