Automated, intelligent surveillance and rescue framework adapted to the mobile environment

SPY Innovation report

Situational awareness is critical for law enforcement in complex, highly dynamic environments. Richer media resources can equip national and local authorities – particularly the police and fire brigades – with the kind of trustworthy, reliable information to help them in their day-to-day tasks. A new generation of intelligent surveillance – the automated wide-area system – is set to significantly improve the way that control rooms share information and supply it to operators in the field. To facilitate smart situational awareness requires new multi-camera surveillance techniques and data-integration methods as well as timely communication of the information together with suitable mobile and multi-sensor platforms. The ITEA 2 project SPY (Surveillance imProved sYstem) set out to address this security and safety challenge by creating an automated, intelligent surveillance and rescue framework adapted to the mobile environment.

Embedded intelligent solutions

The new framework developed by SPY is adapted to the mobile environment that requires optimised transmission and distributed intelligence. Key to the mobile aspect was the definition of embedded intelligent solutions to minimise the amount of information to be transmitted over the wireless networks, offering discrimination in field services depending on the detected situation – such as stolen objects, abnormal behaviour, suspicious event tracking and identification. SPY addressed these concerns by adapting and extending state-of-the-art algorithms already existing on personal computers to mobile and unpredictable contexts. The addition of a rules engine and an ontology to these algorithms enabled the creation of a standalone software core able to detect, understand and act during events. In addition, SPY defined and implemented a new protocol to ensure integrity, adaptive compression and encapsulation of metadata to provide the trusted and interoperable wireless media required in such environments. A secondary objective of the project was to define a new adaptive interaction solution to provide users with a comprehensive and intuitive picture of the incident adapted to their environment.

Common surveillance framework

The development of a common surveillance framework adapted to mobile environments must use advanced video-analysis techniques designed for mobile and unpredictable situations. While context sources were intentionally restricted to video sensors embedded in a vehicle, the SPY framework allows extension to others kinds of sensors. The use of advanced adaptive-coding, compression, error-resilience and error-concealment techniques improved transmission and saved on bandwidth over wireless networks. Distributed intelligence and decision-making capabilities, including data fusion, analysis and ranking of alarms were also important developments in the course of the project and video/image transmission security was enhanced through a combination of authentication, integrity and encryption mechanisms.

Among the major innovations generated within this project , we can emphasys on: the way in which different technologies have been integrated to produce a full surveillance system, the event detection algorithms, adaptive video coding that provides optimum bandwidth use and smart mobile cameras. Beyond these innovative features in fields commonly associated with video surveillance, SPY also brings a technique that is a true first in this domain: the watermarking technologies were adapted for mobile surveillance purposes. Compressed-domain watermarking is essential to being able to ensure video stream certification while copin g with real-time and reduced complexity requirements. SPY managed to specify the first hybrid watermarking-compression authentication scheme and exploited these schemes to design and deploy the first on-the-fly MPEG-4 AVC / H.264 authentication system.

In terms of standardisation, SPY had to jointly deal with technical domains in which no or few standards exist and others in which too many different standards co-exist. Hence, in-depth knowledge on a large variety of standardisation approaches was vital at all stages and levels of the project. Ultimately, SPY was responsible for three new contributions to ISO MPEG standards (collaborative BiFS, MPEG user description, compact descriptors for video search) and also helped to bridge existing standards (MPEG and W3C-HTML5).

Results tested by end users

This new framework improved the use of wireless infrastructures and mobile sensors to help professional end-users in the field as well as the possibility to exploit existing applications and advanced customised algorithms to provide the most comprehensive aggregated picture and automated decision support for faster and more effective monitoring, intervention and control of unpredictable situations. A central feature of SPY was the involvement of real end-users to demonstrate this mobile wireless surveillance solution in an urban environment. French and Turkish police and cities tested an integrated smart-camera – software prototype, embedded on a vehicle (e.g. a police car) and featuring:

• advanced video analysis, such as, object recognition, movement recognition and understanding;
• new certification techniques that enable surveillance information to be used for evidence;
• advanced adaptive video-coding, error-resilience, error-concealment and transport techniques;
• versatile and virtually error-free video stream transfer to the control room, using either non real-time video streaming or a reliable transmission method such as file transfer;
• video transmission optimisation over WiMAX (Worldwide Interoperability for Microwave Access), long-term evolution (LTE) and other wireless IP networks;
• the incorporation in the vehicle of abnormal-situation detection in mobile and unpredictable contexts such as aggression, fire and crowd formation, something that can also be used to trigger alarms or live streaming to save network bandwidth when nothing important is happening;
• remote pan/tilt/zoom control for mobile cameras and to trigger and program specific behaviour or events – enabling users in the vehicle or control room to have full access to all camera data; and
• provision of a relevant operational picture to users by offering an adapted man-machine interface in the mobile environment with rich and intuitive display at control-room level thanks to a smart integration with the SCADA (Supervisory Control and Data Acquisition) system.

Secure and safe environment

The results of the project are already evident in the creation of one Austrian spin-off company to exploit the results and the submission by consortium member Aselsan of three international patent applications and one national patent application. Moreover, a spin-off partially exploiting the SPY results is currently under creation in the Institut Mines-Télécom ecosystem. An exploitation plan was presented for the key industrial partners and two of these, Cassidian and Aselsan, are already in discussion with potential customers.

With SPY systems becoming mandatory for deploying on-the-move video solutions using wireless communication in the future, SPY is among the key components for the new CASSIDIAN LTE mobile broadband communication for the security forces, for example, which has already been tested by the French police and army (world's first operational demonstration in the 400 MHz frequency band) as well as the police in Spain, Mexico, Kuwait and Saudi Arabia, with tests in Singapore being planned. EOLANE, a French consortium partner whose market is intelligent cameras and video surveillance for public transport, is exploiting the need for passenger security and driver assistance among bus, tram and metro services, such as the RATP and SNCF. ASELSAN is already in the process of deploying its SONEC cameras as part of border security contracts totaling more than $120 million.

The advances in surveillance capabilities demonstrated by SPY illustrate how end-users can gain global situation awareness without compromising on detail. Due to their inner scalability and interoperability, the concepts developed within the project could also apply to other markets such as traffic or industrial safety management. In such a case, the surveillance ontology should be enriched with data and metadata models to identify new situational representation and new metadata protocols.

This project, with its watermarking for surveillance and other innovative technologies, demonstrates how European industry is not only a pioneer as an 'inventor' of cutting-edge solutions but also a main player able to seize the high ground when it comes to valorising the results. The benefits from such advances will be felt in terms of industry and employment as well as in a wider societal context where such mobile wireless surveillance solutions will contribute to create a more secure and safe environment for citizens.