Cortical microcircuits as gated-recurrent neural networks

Rui Ponte Costa∗
Centre for Neural Circuits and Behaviour
Dept. of Physiology, Anatomy and Genetics
University of Oxford, Oxford, UK
[email protected]
Yannis M. Assael∗
Dept. of Computer Science
University of Oxford, Oxford, UK
and DeepMind, London, UK
[email protected]
Brendan Shillingford∗
Dept. of Computer Science
University of Oxford, Oxford, UK
and DeepMind, London, UK
[email protected]
Nando de Freitas
DeepMind
London, UK
[email protected]
Tim P. Vogels
Centre for Neural Circuits and Behaviour
Dept. of Physiology, Anatomy and Genetics
University of Oxford, Oxford, UK
[email protected]
Abstract
Cortical circuits exhibit intricate recurrent architectures that are remarkably similar
across different brain areas. Such stereotyped structure suggests the existence of
common computational principles. However, such principles have remained largely
elusive. Inspired by gated-memory networks, namely long short-term memory
networks (LSTMs), we introduce a recurrent neural network in which information
is gated through inhibitory cells that are subtractive (subLSTM). We propose a
natural mapping of subLSTMs onto known canonical excitatory-inhibitory cortical
microcircuits. Our empirical evaluation across sequential image classification
and language modelling tasks shows that subLSTM units can achieve similar
performance to LSTM units. These results suggest that cortical circuits can be
optimised to solve complex contextual problems and proposes a novel view on
their computational function. Overall our work provides a step towards unifying
recurrent networks as used in machine learning with their biological counterparts.