Nadam (Nesterov-accelerated Adaptive Moment Estimation)

Published on: August 4, 2021

Nadam (Nesterov-accelerated Adaptive Moment Estimation)

Table of Content

Nadam (Nesterov-accelerated Adaptive Moment Estimation) combines NAG (Nesterov accelerated gradient) and Adam. To do so, the momentum term needs to be updated. For more information, check out the paper or the Nadam section of 'An overview of gradient descent optimization algorithms'.

The final update rule looks as follows:

Code

Resources

More stories

  • Activation Functions

  • RAdam - Rectified Adam

  • QHM (Quasi-Hyperbolic Momentum)