AI-powered detection and tracking of passengers and their luggage
Each year, numerous pieces of luggage are left behind on trains, leading to delays, security risks, and inconvenience for both passengers and operators. Manual monitoring of passengers and their belongings is not only labor-intensive but also prone to human error, highlighting the need for automated solutions.
To tackle this challenge, the Belgian railway technology company Televic GSP has partnered with Sirris, an industry-driven research centre, to develop an AI-powered luggage tracking system in the scope of the ITEA project TAPCOP. This innovative system uses AI to automatically detect and track both passengers and their luggage, raising an alert whenever someone disembarks without their belongings. By swiftly identifying and managing left-behind items, the solution enhances both safety and efficiency in railway operations.
To achieve these objectives, several key challenges must be addressed:
- Luggage detection: while there are reliable models for detecting people, identifying luggage remains a challenge. Items like backpacks and purses often change their visual appearance depending on their position, making detection and tracking more difficult.
- Associating luggage with owners: luggage ownership can be ambiguous, especially for groups like families. For example, the person who initially brings a backpack may not be the one leaving with it. The system must correctly associate luggage with its rightful owner or group of owners to avoid false alerts.
- Tracking in low-quality footage: trains are often equipped with low-resolution fisheye cameras at the ends of carriages, creating additional challenges such as obstructions from other passengers and seats, as well as sudden lighting variations when passing through tunnels.
The proposed solution has shown promising results in real-world tests using data from a variety of train types and locations. It uses multiple AI models, including an object detection model to identify people and luggage, and a segmentation model to refine these detections with precise boundaries. These two models form the system’s core. Building on this, a tracking and re-identification system is under development to maintain continuity, even when individuals are temporarily obscured or move between train carriages, enabling train-wide monitoring. Additionally, a relational linking model associates luggage with its rightful owner. Currently, this solution achieves 80% accuracy in tracking and linking passengers with their luggage on moderately full trains. Edge cases - such as groups of passengers with shared luggage and overcrowded trains with significant obstructions in narrow hallways - provide opportunities for further improvement of the system.
Beyond luggage tracking, the TAPCOP project focuses on achieving situational awareness and data-driven visitor flow management by employing AIbased sensors. The system aggregates multiple data sources using AI, offering a more reliable, complete view of the environment and predicting overcrowding. In addition to the Televic use-case, the project also explores other innovative applications aimed at improving traffic and mobility conditions during rare, crowded events, such as festivals, road accidents and sports events.
