Performance analysis of optimization algorithms for convolutional neural network-based handwritten digit recognition

Abstract

Handwritten digit recognition has been widely researched by the recognition society during the last decades. Deep convolutional neural networks (CNN) have been exploited to propose efficient handwritten digit recognition approaches. However, the CNN model may need an optimization algorithm to achieve satisfactory performance. In this work, a performance evaluation of seven optimization methods applied in a straightforward CNN architecture is presented. The inspected algorithms are stochastic gradient descent (SGD), adaptive gradient (AdaGrad), adaptive delta (AdaDelta), adaptive moment estimation (ADAM), maximum adaptive moment estimation (AdaMax), nesterov-accelerated adaptive moment estimation (Nadam), and root mean square propagation (RMSprop). Experiments have been carried out on two standard digit datasets, namely Modified National Institute of Standards and Technology (MNIST) and Extended MNIST (EMNIST). The results have shown the superior performance of RMSprop and Adam algorithms over the peer methods, respectively.