Deep learning as dynamical systems

I am Davide Murari, and I am a first-year PhD student in the group of Differential Equations and Numerical Analysis (DNA) at NTNU.

Both my Bachelor degree and my Master degree were in Applied Mathematics, at the University of Verona. During the master, I had the opportunity of attending the Modelling week at Grenoble and spending five months in Nice (at the University of Cote d’Azur) for the Erasmus program. My bachelor thesis was on dynamical billiards, while the master’s one on the theory of integrability of non-Hamiltonian dynamical systems.

Why I moved to Trondheim to pursue my PhD and what I am going to work on

During university, I developed a great interest in dynamical systems and geometric mechanics. In the DNA group here at NTNU, I found the opportunity of merging these two interests with the one of numerical implementation. My PhD thesis will focus on

• analysing deep neural networks from the perspective of dynamical systems and
• data-driven modelling for mechanical systems.

The current draft of the thesis’ title is “From dynamical systems to deep learning and back: network architectures based on vector fields and data-driven modelling”.

By dynamical systems’ approach to deep learning, I refer to their possible interpretation as non-autonomous parametric ODEs. Indeed, this comes thinking to time as a measure of layers’ depth, and when the number of layers ideally tends to infinity.  Therefore, for example, we can think of the challenge of binary classification of points of the plane as “learning a vector field whose flow moves the points so that a hyperplane can separate the two labelled groups”.

Thanks to this construction, many relevant questions and techniques typical of ODEs and Numerical analysis arise in this research area and make me interested in these problems. Therefore, in this research project, I want to use dynamical systems to study deep learning architectures and attempt to explain their training and reliability as predictive tools.

Furthermore, there are many applications where the data are manifold-valued, and hence the forward propagation in the neural network should remain on that manifold. Neural networks on manifolds are still an area with not too many results, but with clear potential, so the plan is to investigate this problem throughout the four years of the PhD.

For the data-driven modelling part, I have the opportunity of collaborating with the ETN THREAD project “Numerical Modelling of Highly Flexible Structures”. In this context, we will work first modelling real-world mechanical systems and then studying their time evolution thanks to the instruments provided by neural networks.

Thank you for reading,

Davide