Unfreezing layers during training?

[s-rog]

Freezing layers at the beginning of training works, however unfreezing in on_epoch_start() during training causes the gradient to explode. Without the unfreezing part (or without freezing at all), the model trains fine with no gradient issues.

I'm using DDP + Apex O2 and the loss scaling will keep going down to 0 where it would encounter 0 division and crash.

Is unfreezing during training not possible in pytorch/lightning? or am I missing snippet?

[williamFalcon]

you can unfreeze whenever. if gradients explode it's for another reason