This is a study based on the paper On The Convergence of ADAM and Beyond. In this report we tried to present a short summery explaining what this paper is about and reproduce the results of the experiments that was shown in the paper.
There are 3 experiments presented in the paper and the implementation of these experiments can be found in:
In addition to these experiment, we also designed an experiment of our own, experiment 4, for which the implementation can be found here.
- Note: The first section of the report provides a short summery of the paper, the motivation behind its study, and the solution/proposed method to tackle the problem it is addressing. The goal of this section is to target any reader who is seeking to get a general idea of what the paper is about. Hence, this section does not contain any mathematical analysis of the problem. For readers who seek a deeper understanding of the paper and the mathematical concepts related to it, we refer them to the paper itself.