Reading movement from posture: Machine learning approach

With the advent of high-speed cameras and the widespread use of artificial intelligence, software tools can now extract trajectories and body parts from videos, changing how behavior and neuroethology are studied. A key idea is that an animal’s posture or sequence of postures contains information about its behavior and intentions. Knowing an animal’s body movements over a given period, one might expect to predict its future movement using equations. However, the high dimensionality and many interactions involved make a general analytical solution difficult. However, many animals can do this and take advantage of it in their own behavior, without needing to formulate or solve equations. Machine learning offers an alternative by identifying patterns across different situations and using them to make predictions. We investigate whether an animal’s displacement and rotation can be predicted from its previous postures. Using freely swimming zebrafish larvae, we extracted body-part trajectories with DeepLabCut and trained machine learning algorithms to predict displacement and rotation from a preceding time window. Our results show good predictive performance, allowing the animal’s trajectory to be predicted even a few seconds into the future. However, performance is strongly affected by errors in posture estimation. This work provides a starting point for developing more general approaches and exploring further improvements and applications of the algorithm.