Redefining emergency call systems with multimodal AI

Every second counts in emergency response. Although traditional emergency call systems have been saving lives for years, they face serious operational and structural problems due to fake calls, non-emergency calls, a high false alarm rate, a lack of situational awareness, and technological limitations. Today, emergency call operations rely heavily on text-based data sources, which limits the ability to fully leverage all available context during an emergency.

Multimodal AI can transform emergency call systems from merely a communication tool into proactive protection systems with situational awareness. By processing audio, video, text, images, geographic, and environmental data together, and using the outputs of the ITEA project CLEAR (Comprehensive Learning for Advanced AI Responsiveness), much faster, more accurate, and life-saving decisions can be made.

In the Turkish consortium, which consists of Orion Innovation, Defne, and Codebase, the emergency call system use case is currently in the proof-of-concept phase. Initial experiments have focused on recognising language, accent, emotion, and background sounds using various multimodal fusion strategies as well as speech, audio and text models. These activities form the initial technical foundation for the development of AI-powered emergency call analysis. Further integration and validation are required to assess its impact on location verification, incident assessment, and response team selection, and quantitative results will be analysed.

The recommended operational workflow consists of four main stages:

1. Data collection and initial assessment – First 15 seconds

As soon as a call comes in, the system collects and analyses available multimodal data. While data such as Transaction Detail Records (TDR), Call Detail Records (CDR), GPS logs, and Base Station Almanac (BSA) are transmitted to the system, voice analysis can identify the language, accent, speech characteristics, background noises and relevant acoustic cues. As textual information is extracted from the conversation, video or image data can provide extra visual evidence. Geographic information and environmental sensor data provide additional context about the caller and the surrounding conditions.

2. Location verification and context analysis – 15–45 seconds

The system combines information from different modalities to verify the estimated location. The caller’s verbal description can be matched with geographic features; images, videos, ambient sounds, weather conditions, and sensor data can provide additional evidence. This cross-modal verification helps reduce uncertainty in situations where individual data sources are incomplete or unreliable.

3. Team selection and route optimisation – 45–60 seconds

The incident is classified based on its characteristics and urgency. The system then supports the selection of the most appropriate response team by taking into account location, incident type, terrain, environmental conditions, and available resources. Traffic and geographic constraints are also considered when determining primary and alternative routes.

4. Dispatch and continuous monitoring – 60 seconds later

While incoming multimodal data continues to be analysed, relevant information is shared with field teams. New information creates a continuous feedback loop between the emergency call center and response teams by updating the estimated location, incident context, or recommended route.

This enables emergency dispatchers to make more informed decisions in situations where every second counts.

More information: https://itea4.org/project/clear.html