Short-term trajectory prediction for autonomous vehicles

Today we caught up with Sushil Sharma a PhD student at the Science Foundation Ireland Centre for Research Training (CRT) in Foundations of Data Science who carried out his placement with Valeo and to find out more about his poster presentation at the 30^th Irish Conference on Artificial Intelligence and Cognitive Science (AICS).

Sushil holds a Bachelor of Engineering degree in electronics and instrumentation stream from India and a Master’s degree focused on Automatic control and robotics with a specialisation in smart aerospace and autonomous systems from Poland and France. He spent one year in manufacturing industries in the process department as an Engineer. After that, he spent over two years at Laboratoire IBISC-CNRS in France. He is interested in working with smart manufacturing with machine learning. His primary research interests include autonomous systems, robotics and machine vision. He also likes to work on smart sensors.

Sushil Sharma, CRT Student

Placement: Valeo

Last year I presented the work I carried out as part of my 12-week summer placement with the Deep Learning team at Valeo, Tuam. My project was focused on the short-term trajectory prediction for autonomous vehicles. I presented this work as part of the AICS 2022 conference with Dr. Ciaran Eising, Mark Halton, Ganesh Sistu and Jonathan Horgen.

Autonomous short-term trajectory forecasting is important for applications like lane merge and change, speed control, and exiting motorways. At present, the majority of commercially automated vehicles use state-based machine-learning algorithms to predict the short-term trajectory of the vehicle. This involves two stages, a deep learning-based perception system to detect the other agents in the traffic like vehicles, pedestrians, traffic lights, stop signs, etc. followed by state machine-based probabilistic models to forecast the trajectory of the ego vehicle by minimizing the risk of accidents. Though recent methods focused on end-to-end differentiable data-driven systems, most of the model designs are limited by the available datasets where the data is limited to generic scenarios with limited corner cases. Hence there is a need for substantial datasets that correspond to complex scenarios that are often hard to find in the real world. This inspired us to create a CARLA-based synthetic dataset for short-term trajectory prediction tasks. The dataset consists of 6K perspective and orthographic view images and corresponding IMU, and odometry information for each frame. Dataset involves use cases like pedestrians crossing the road, vehicles overtaking-and-stop and starts scenarios. We have also built an end-to-end short-term trajectory prediction model consisting of CNN and LSTM that removes the need for explicit encoding of the surroundings’ knowledge into the model by inducing safety constraints into the loss function.

We demonstrated that complex stochastic tasks like trajectory forecasting can be implemented in a supervised fashion with safety as an objective during the learning process. We show that a complete data-driven implicit system understands corner case scenarios like slowing down near the zebra crossings and stopping when pedestrians cross the road. We are releasing the dataset and our model to help the research community accelerate their research in short-term vehicle trajectory forecasting. The research involves two considerable contributions.

1. Prediction of vehicle’s short-term trajectory from only orthographic images with no explicit knowledge encoding

2. Dataset to encourage the research community to pursue the direction of end-to-end implicit trajectory prediction learning methods.

The dataset consists of two subsets divided into 1000 and 5000 frames. The first 1K images focus on pedestrians and stationary vehicles. where the second submission focuses on dense traffic conditions. All the instances consist of both perspective and orthographic images. We used orthographic images to keep the focus on the learning trajectory prediction as the extrinsic deviation in the simulations is minimal. This also helped us to focus more on learning forecasting. The network is made up primarily of two components: · The Convolutional Neural Network (CNN), which receives a sequence of n orthographic images as input and is responsible for extracting essential features from these images · The Long-Short Term Memory Network (LSTM), which forecasts the trajectories based on the CNN’s deep features. LSTM are very well studied in the vision for temporal prediction tasks with sequenced input data. Because each position indicates the ego vehicle’s location at a future moment in time, the goal is for our LSTM network to learn to infer the future positions. In this study, we separated it into two groups, such as the CNN-LSTM model and the CARLA dataset. We utilised a Python script to build our simulation for the CARLA dataset and connect the sensors, such as the camera and IMU on vehicles, which involved specifying the camera location and angle. Generally, our network was trained using python programming.

Key learnings from my placement with Valeo:

1. My project is incredibly cool and fascinating. The step-by-step explanation from the industrial supervisor kept me motivated each week.
2. I found it interesting when the industrial supervisor discussed their conference and workshop attendance experiences.
3. Real-world problem-solving: Experienced professionals can bring practical experience and a deep understanding of real-world problems to the team. This can help the team to tackle complex problems that may arise during the development process.

About the SFI Centre for Research Training (CRT) in Foundations of Data Science

The Science Foundation Ireland (SFI) Centre for Research Training (CRT) in Foundations of Data Science is a large-scale collaborative initiative between the University of Limerick (UL), Maynooth University (MU), University College Dublin (UCD), a rich ecosystem of multinational and indigenous companies and an international network of collaborators; it is coordinated by Skillnet Ireland and Technology Ireland ICT Skillnet. The programme is led by co-Directors Professor James Gleeson (UL) , Professor Claire Gormley (UCD) and Professor David Malone (MU). The academic team includes vice-Directors Professor Norma Bargary (UL), Professor Nial Friel (UCD) and Associate Professor Catherine Hurley (MU), and a wide network of over 120 academic supervisors. .

Currently the CRT hosts 112 full time doctoral students distributed essentially equally across the three Higher Education Institutions (HEIs): UL, MU and UCD. From the four recruitment intakes to date 48 of the cohort self-identify as female and 49 as non-EU, (19 nationalities represented) demonstrating the CRT’s strong commitment to equality, diversity, and inclusion.

The CRT programme offers students a unique career pathway that will see them develop cutting-edge tools and technologies to provide insights to help shape industry, academia, and policy across the country. It not only helps mobilise the power of data nationally for industry across multiple sectors but will also assist in establishing Ireland as a leading data science research nation with a sustainable flow of expertise. All CRT students undertake a mandatory 12-week training placement with one of the CRT Enterprise Alliance (EA), a 16-strong body of industry and enterprise organisations, where they train with EA partners to solve an industry-relevant problem.