QHAdam (Quasi-Hyperbolic Adam)

Quasi-Hyperbolic Momentum Algorithm (QHM) is a simple alteration of SGD with momentum, averaging a plain SGD step with a momentum step. QHAdam (Quasi-Hyperbolic Adam) is a QH augmented version of Adam that replaces both of Adam's moment estimators with quasi-hyperbolic terms. Namely, QHAdam decouples the momentum term from the current gradient when updating the weights, and decouples the mean squared gradients term from the current squared gradient when updating the weights. [1, 2, 3]

Essentially, it's a weighted average of the momentum and plain SGD, weighting the current gradient with an immediate discount factor divided by a weighted average of the mean squared gradients and the current squared gradient, weighting the current squared gradient with an immediate discount factor . [2]

[1] Ma, J. and Yarats, D. Quasi-hyperbolic momentum and Adam for deep learning. arXiv preprint arXiv:1810.06801, 2018

[2] QHAdam Papers With Code

[3] John Chen. An updated overview of recent gradient descent algorithms

Code

• QHAdam Numpy Implementation