Implement Optimizers

[bhavnicksm]

This issue is to keep track of all the optimizers to implement in the repository.

To Do

• SGD (.SGD)

• Momentum (.Momentum)

• RMSProp (.RMSProp)

• AdaGrad (.Adagrad)

• Adam (.Adam)

• AdaMax (.Adamax)

• AdaDelta (.Adadelta)

• Implement AdamW (.AdamW) #27

• AMSGrad (AMSGrad)

• RAdam (.Radam)

• LION (.Lion)

• Nesterov Accelerated Gradient (.Nesterov) #31

• NAdam (.NAdam)

• AdaBelief (.Adabelief)

• Adan (.Adan)

• AdaFactor(.Adafactor)

• AccSGD (.AccSGD)

• AdaBound (.Adabound)

• AdaMod (.AdaMod)

• Demon SGD/Adam (.DemonSGD & .DemonAdam)

• AggMo (.AggMo)

• QHM (.QHM)

• QHAdam (.QHAdam)

• LAMB (.Lamb)

• SWATS (.Swats)

• Shampoo (.Shampoo)

• SAM (.SAM)

• AdamP (.AdamP)

• Sophia (.Sophia)

• GaLore (.GaLore)

Resources

• Blog on writing custom optimisers by Daniel Mcneela
• Source code of Torch.Optim.SGD
• Source code of Torch.Optim.Adam
• Blog on difference of PyTorch Implementation and Normal SGD
• Ways to visualize a optimizer
• LabLab's implementation of the Adam and other Optimisers