Enhancing video interpretation through context modelling
ViCoMo Innovation report
Electronic appliances, devices and equipment are becoming increasingly interactive and intelligent in respect of the environment or conditions in which they have to operate. Such systems and appliances can relieve humans from onerous work and improve efficiency, security and safety in areas like healthcare, lifestyle and surveillance as well as in production and transport system monitoring. Because of the enhanced interaction with users and the environment, the complexity of such systems is increasing rapidly.
The ITEA 2 project ViCoMo (Visual Context Modelling) is the successor to CANTANTA, which focused on behaviour and interaction analysis, and its predecessor CANDELA (single event detection). The general goal of the ViCoMo project is to find the context of events captured by cameras or image sensors, and model the context such that reliable reasoning about an event can be established. Hence, modelling of events and the 3D surroundings forms an integral part of the research agenda of the project. These goals also complement the strategic research agendas for the EU and individual countries, since the developed technology is advanced information technology that contributes to improving healthcare, security and safety, and the public infrastructure in society in general. The project also supports the development of emerging industries in surveillance, healthcare and data mining, concerning data storage, efficient retrieval and usage.
Rich mix of partners
The ViCoMo consortium was composed of 29 companies from 5 countries (France, Spain, Finland, Turkey and the Netherlands) with relevant experience in the domains related to the development of the ViCoMo concepts and services. This experience is based upon their business activities or from their participation in other European projects. The rich mix of the consortium included large high-tech companies (e.g. Philips, Acciona, Thales), smaller innovative SMEs (CycloMedia, ViNotion) and relevant research groups and departments from well-known universities (TU Eindhoven, University of Catalonia) and research institutes (INRIA, CEA List). The consortium set out to exploit context modelling in several domains: observation for surveillance, 3D modelling of the real-world environment, observation of human behaviour for system control, and logistics control for traffic and transportation.
Contributions to (new) standards
While the ViCoMo project has not developed any new standards as such, the results do contribute to helping to define new specifications for the ISPS (International Ship and Port facility Security) code that aims to improve security conditions in maritime sector. In addition, ViCoMo is a potential contributor to the standards in those sectors in which it is used, such as a new standard to describe behaviour patterns of humans as metadata. An example of this is the health sector, where behaviours or observations of the patients are described as metadata.
Technical standards for Mixed Reality do not yet exist, although some attempts have been made in this direction. VTT has traditionally brought Mixed Reality related research results to ISMAR, which is the most important conference in this research area. During ViCoMo, VTT continued this participation and monitored all the emerging standardisation initiatives, tracking the standardisation work initiated in the Augmented Reality area. Within the context of ViCoMo, VTT has produced a content description and production pipeline, which is one of the areas where a particular need for standardisation has been recognised.
Demonstrating innovation
The ViCoMo work packages produced a number of innovations. Firstly, an event simulation engine was linked to video processing for augmented reality and a common plug-in architecture was dedicated to video processing as a tool for video processing development and exploitation. Furthermore, a calibration framework to associate calibration information with video streams allowed the 3D localisation of detection and multi-camera processing. Secondly, the development of a number of robust content analysis algorithms that address a wide range of functionalities as well as multi-camera tracking, people re-identification and behaviour analysis make the use of context in video analytics more robust and context reasoned. Thirdly, innovation came in the visualisation of complex events by an active 3D scene, interactive digital signage with augmented reality, touching objects on stereoscopic displays, content creation tools and workflows for AR applications and augmented reality for interior planning in mobile devices. These three main innovations in the ViCoMo project – multi-camera and dynamic analysis, 3D environment modelling and the successful integration of new marketing concepts and emerging technologies – have been demonstrated in two scenarios, a port terminal and a shopping mall. In the port terminal, the context modelling techniques improved the monitoring and tracking of cargo throughout the harbour while visualisation of the 3D modelling improved the logistic operations and control. In the shopping mall, surveillance concerned the behaviour of individuals and groups. In addition, marketing objectives are enhanced by automatically collecting shopping statistics, adaptive advertising and an interactive kiosk. The two demonstrators share a commonality in terms of the common ViCoMo system architecture, the technology used for multi-camera analysis, the similarity of the concepts for modelling the context and the surveillance applications.
Exploitable results
With a focus on exploiting results, the ViCoMo project opted to take an approach that would maximise exploitation prospects. This approach was characterised by parallel development of the technical solutions and the assessment of business opportunities (by benchmarking, market research and by contacts with business stakeholders on requirements, needs, future interest) while the most attractive concepts were carefully selected for the demonstrators that were built at the end of the project to show the total outcome of all the work packages. More specifically, the partners plan to exploit the ViCoMo results through generating innovative commercial products (new surveillance and security products, new transport and logistics products focusing, for example, on camera image sensors, in 3-D solutions) and offering new and enhanced services (in the field of security, logistics, consulting and software development). Furthermore, improved design methods and tools are envisaged that can speed up the design phase and offer shorter time to market, flexibility, higher customer satisfaction, and low development costs while the new software architecture and middleware are expected to be integrated in products, services and industrial environments.
Another outcome of the project is the generation of future product options that will be commercialised by the consortium members. The excellent track record of many of the consortium partners working closely with technology centres, new campus companies and technology villages where the aim is to convert research ideas into business plays a significant role in the emergence of new companies.
Business boost, competitive edge
Among the examples of fast exploitation are an autonomous security system delivered by ViNotion to Charleroi Airport that monitors any movement to and from the runway using thermal cameras for distances exceeding 500 m. In fact, negotiations are ongoing to start a new company for public security in collaboration with public co-operatives. VDG has released an improved mono-camera tracking solution, ObjectRTM, in a HW-SW design and sales are increasing. CycloMedia has already sold a number of automatically generated lists with traffic signs, their type and 3D positions. The specially developed algorithms are also being applied for face and licence-plate detection for privacy protection. INRIA’s event recognition is being licensed to KEENEO and is currently being validated. As for mixed reality exploitation, applications have been licensed to a magazine publishing company and negotiations on licensing of augmented reality display for digital signage are ongoing.
By actively stimulating research and cooperation in several of the fastest growing businesses of the coming decennia, the ViCoMo project contributes in a positive way to employment. A broad spectrum of applications and services can be built using the ViCoMo knowledge, models and tools. Not only does the project enable existing companies to gain a new competitive edge, it also provides an excellent springboard for companies aiming to develop new business lines based on the products, applications and services developed and demonstrated in ViCoMo. Finally, it also generates opportunities for starting up new businesses, known for their high job creation potential.