Memory Consolidation with Orthogonal Gradients for avoiding Catastrophic Forgetting
The memory consolidation process enables the accumulation of recent and remote memories in the long-term memory store. In general, the deep network models of memory suffer from forgetting old information while learning new information, called catastrophic forgetting/interference. The human brain overcomes this problem quite effectively, a problem that continues to challenge current deep neural network models. We propose a regularization-based model to solve the problem of catastrophic forgetting. According to the proposed training mechanism, the network parameters are constrained to vary in a direction orthogonal to the average of the error gradients corresponding to the previous tasks. We also ensure that the constraint used in parameter updating satisfies the locality principle. The proposed models performance is compared with Elastic Weight Consolidation on standard datasets such as permuted MNIST and split MNIST on classification tasks using fully connected networks, and Convolution-based networks. The model performance is also compared to an autoencoder on split MNIST dataset, and to complex core50 dataset on two types of classification tasks with EWC. The proposed model gives a new view on plasticity at the neuronal level. In the proposed model, the parameter updating is controlled by the neuronal level plasticity rather than synapse level plasticity as in other standard models. The biological plausibility of the proposed model is discussed by linking the extra parameters to synaptic tagging, which represents the state of the synapse involved in Long Term Potentiation (LTP).