Enhancing machine learning optimization algorithms
by leveraging memory caching
Imen Chakroun, Tom Vander Aa and Thomas Ashby
IMEC/ ExaScience Life Lab, Kapeldreef 75, Leuven, Belgium
Introduction
Stochastic Gradient Descent (SGD) is
probably the most popular family of
optimisation algorithms used in
machine learning
It iteratively searches for the best
model parameters by taking small
steps in the direction of the negative
gradient of the cost (loss) function
until it reaches its minimum
Data access in SGD has extremely
low temporal locality (randomness)
Minimal use of the small/fast layers of
memory in an HPC memory hierarchy
Algorithm that uses temporal
locality of data access while
combining the advantages of SGD
with mini-batch SGD by leveraging
memory hierarchy
Sliding window SGD
SW-SGD is computed using new
training points from the large memory
number of training points that are in
cache
Extra training points in the cache are
“free” due to:
- Cache effect
- Accessing them uses dead time
whilst waiting for new points from
the main memory
Compared to one point SGD and Mini
batch SGD:
extra points in a gradient calculation
lower noise
smoothness and easy identification of
convergence
Experiments
A classification problem of the MNIST
dataset + a regression problem using
the ChEMBL dataset.
Tested using several gradient descent
optimization algorithms: Momentum,
Adam, Adagrad...
We used a neural network with 3 layers
and 100 hidden units each. Averaged
results from 5-fold cross-validation.
Results
The fundamental idea of the SW-GD is
valid for many GD variants without any
change to the algorithm.
For Adam using 256 new + 128 cached
has less fluctuation and converges
slightly earlier.
However using a batch of size 256 and
512 is less efficient than using 128
(preliminary experiments)
The added value here is brought by
using visited points not because of a
bigger batch size.
SW-SGD incurs extra compute time
(more points).
The bigger the size of the problem
and the batch size are the more SW-
SGD reduces the cache hits misses.
Conclusion
SW-SGD adapts SGD to the access
characteristics of modern HPC memory
to gain extra gradient noise smoothing.
It combines the epoch efficiency of SGD
with the lower noise and easier to spot
convergence of mini batch SGD.
SW-SGD is applicable for different
variants of SGD.
Acknowledgment
This work is funded by the European
project ExCAPE Grant Agreement no.
671555.