Bringing reality to virtual worlds

METAVERSE1 Innovation report

The ITEA 2 Metaverse1 project has developed a standardised global framework enabling interoperability between virtual worlds such as Second Life, IMVU, OpenSim, Active Worlds and Google Earth and with the real world in terms of sensors and actuators, vision and rendering systems, and applications in areas like social and welfare systems, banking, insurance, tourism and real estate. Results of the project drove the MPEG-V virtual worlds standard published by ISO/IEC in January 2011.

Virtual worlds integrate existing and emerging media technologies which can support networked services and drive the development of new kinds of devices and applications. The emergence of virtual worlds as a platform for networked services is seen as an important enabler as it offers the power to reshape the way businesses and people interact with their environments.

Such virtual worlds are found increasingly in areas such as serious computer games and simulation models. However they are mostly stand alone and independent of each other with little or no connection to the real world. As such, they are isolated solutions with a life of their own. Metaverse1 set out to overcome this isolation – defining a standard to enable connectivity and interoperability between virtual worlds and with the real world.

Connecting worlds

The objective of the ITEA 2 project was to define interoperability in such a way that it would be possible to exchange information between virtual worlds. For example personalisation of an avatar in one virtual world could be applied to an avatar in another world. This would be useful for example in translating social skills to supply feedback to users established in one coaching system to another virtual world for a similar application. It would make it possible to transfer all sorts of things that have been added in one world to another.

Even more important and needed from an industrial point of view is the development of a standard interface between the real physical world and the virtual – simulation/serious games – world. This would make it possible to attach real world sensors – such as body parameter or environmental sensors – to provide input to simulations or alternatively obtain feedback from such models into the real world, for example to control comfort conditions in terms of lighting, temperature or ventilation in a room or for personal wellbeing or drive robotic aids for disabled persons.

Another example is the use of 2D digital (video) sources as input for 3D worlds. A lot of what is done today is already available in some kind of IT system. So a standard interface would make it easy to obtain input from all types of existing systems – such as traffic reports, weather forecasts, property details or tourist information – for a virtual world representation or simulation.

Diverse consortium

Israeli professor Yesha Sivan, head of the information systems programme at the Tel Aviv Academic College, brought the idea of standardisation in the field of virtual worlds to the ITEA 2 project outline event in Düsseldorf in 2007. Jean Gelissen from Philips Research teamed up with Sivan and took on the role of project leader.

Consortium members came from a range of different areas. Spanish partners focused on tourism and virtual travel applications. French partners were interested in technology simulation for museums, with a model allowing people to be present virtually in the space station and experience effects of low gravity. Dutch partners were more focused on ambient assisted living for elderly people – including connectivity with carers.

Finally, Alcatel Lucent in Belgium and France was interested in the symbiosis between video conference and meetings in a virtual world. Video conferencing has limitations – particularly in being static. Meetings in a virtual world can be much more dynamic. People can move around, change seats and participate in subgroups in a virtual world, but there is still a lack of reality – it is not possible to see facial expressions for example. So the idea was to combine the two approaches by projecting 3D footage of participants on their avatars in the virtual world – offering a new way of teleconferencing.

Many of the technologies required by Metaverse1 were not new but it was necessary to identify what was missing and develop suitable solutions. The project therefore defined a series of use cases and looked what was available in terms of virtual worlds and the very limited connectivity with the real world – mainly through display screen, keyboard and mouse.

Some 18 missing items were defined and the necessary technologies developed. Missing items, for instance, included:

• Being able to transfer data and actions between systems in terms of available sensor signals to avoid clicking a mouse and keying in information;
• Feeding real-time 3D video streams into a virtual world;
• Providing support for multiple languages – crucial in social contexts; and
• Support for inclusion of real audio input – for example taking original sounds such as fountains or on the beach at locations in Gran Canaria and integrating them into a virtual tourism application.

International standard

The project's key intention was to develop an international standard within the well-known ISO/IEC Moving Picture Experts Group (MPEG). A new standard was completed within the timeframe of the project including an international workshop around the standard held in Korea in January 2011.

The first version of the ISO/IEC 23005-1:2010 (MPEG-V, Media context and control) standard is made up of several parts referring to:

• Architecture and use case scenarios;
• Metadata to describe device capabilities, sensor and actuator data and user preferences;
• Metadata to represent sensory effects;
• Metadata to represent virtual-object and avatar characteristics; and
• Syntax and semantics for all the above data formats and reference software.

MPEG-V took into account all sorts of products already on the market that could be adapted to the standard. This included amBX, originally developed by Philips, which is a system allowing the addition of extensions to multimedia and computer games. amBX makes it possible for example to add thunderstorm effects to games or films and is used in the Korean RoSE system to automate special effects for stage performances. Other applications studied in Metaverse1 for instance included the use of sensors for ambient-assisted living.

In the standardisation context, work is already advancing on a second version of the standard to extend its application domains. There is a lot interest in biosensors – measuring vital body parameters and using them as inputs either for games or lifestyle-related applications.

Value in applications

MPEG-V defines the boundary conditions but the real added value is in the applications – transforming raw data into something useful. This is of interest to consumers, industry and even public authorities.

For example, local government in the Netherlands is interested in simulation models to help in urban redevelopment projects: currently, the public is invited to look at plans or a model in the town hall and give their opinion but this does not work well. Virtual worlds allow viewing from any angle or place – from where people live for example. So, when developing a new shopping centre, sports facility or apartment building, planning authorities can demonstrate the impact more directly.

Such a scheme was tried out in the Netherlands and a second version has been used to help students to find preferred accommodation. A similar approach can be used for all types of other urban planning – from modelling utility systems for maintenance or extension to locating new transport systems.

In addition, the virtual tourism, sensor-enabled ambient-assisted living and new generation teleconferencing systems mentioned earlier are application domains which have generated new business concepts.