Temporal dynamics and deep learning in a recurrent neural network with time delays

Besides their computational applications like classification and prediction, recurrent neural networks (RNNs) are themselves a model for many phenomena in neuroscience, social dynamics, and other areas, and the diversity of architectural variants and learning algorithms has grown very rapidly in the last 20 years. In many ways, however, the spatially-recurrent architecture —where every node in the network can be simultaneously connected to many others— is considered the most challenging of all, specifically due to the presence of excitatory loops. If the excitatory connections are sufficiently strong, chaotic neural activity patterns arise and the network becomes exponentially sensitive to small changes in initial conditions, causing most training algorithms to fail. If we also consider that the recurrent network model can have time delays, the richness and complexity of possible solutions grows further, as well as the difficulties for numeric simulation, training, and analysis of solutions. In this work we take a deep learning algorithm that can handle the chaotic regime (InnateTraining or Dynamical Attractor) and apply it to the training of chaotic RNNs that also have time delays in the connections between neurons. We show that the algorithm can successfully train a network despite the existence of delays of appreciable size compared to the characteristic time of each neuron.