MOS2S

Traditional media is losing ground to personalised experiences. Children of today, for example, don’t even know what it’s like to have a set of TV channels with fixed broadcasting timeslots for your favourite shows; they choose what to watch at the time they want. And they even produce thousands of pieces of content on their own each day. This trend in the entertainment business can also be seen in society, where city representatives no longer make decisions on their own. Everybody wants to be involved, or at least can be.

With many different innovations and highlevel sensor applications, the MOS2S (Media Orchestration — Sensor to Screen) project took the outdated broadcasting concept to the next level, adding a completely different dimension with features such as instant live broadcasting. The aim of the project was to capture as much sensor data as possible and use this data in various applications in order to eventually enhance the experiences of people.

Citizen and audience engagement

To bring this engagement to a higher level, 17 partners from four countries (led by TNO in the Netherlands) came together in the ITEA project MOS2S and have created world-first ways to engage with citizens and audiences of live events. MOS2S was centred around two use-cases: crowdsourced journalism and (sports) entertainment. The project focused on technologies that allow data and media streams to be orchestrated into an all-encompassing experience on various types of end-user devices. The common denominator in MOS2S is the media processing platform, which combines multimedia streams from different domains.

e-Democracy enhanced with AI

For e-Democracy, four components were developed by the MOS2S project in which each partner provided another step of the value chain:

• Babbelbox
• Online Debate
• Online Debate replay
• Hangouts

VRT’s Babbelbox is a mobile interaction booth which is set up at crowded locations and lets citizens give feedback on societal issues. In April 2019, reporter and photographer Yassine Atari and Belgian journalist Rudi Vranckx travelled for four weeks through Europe in search of the voice of citizens. They set up a Babbelbox in Palermo, Vienna, Budapest, Warsaw, Berlin, Brussels and Paris. Opinions of passers-by were gathered through various questions about, for example, inclusivity and climate change – all made available to people to answer in their native language. Since then, Babbelbox’s technologies have been used in multiple media campaigns like ‘De Warmste Week’ (‘The Warmest Week’), one of the biggest charity programmes in Belgium. The Babbelbox technologies brought more personal stories to professional media creators, enabling them to create more impactful broadcasts on radio, television and digital channels. Selected content has also been used in live debates. In the ITEA project CityStory , the Babbelbox technology has been upgraded and extended with video-based questionnaires in order to provide users with a more personal and inclusive interaction.

Gerade Software’s Online Debate is a tool with an integrated moderator role and editorial dashboard to set up online debates. Online Debate Replay allows editors to put the results of these debates into a media format that can be broadcast, incorporating a number of innovations, and this has already been implemented in Turkey. After the project, these tools were integrated into an existing solution, which attracted an investor who bought it.

Hangouts is a video chat system, co-developed by VRT and Kiswe, which is fully integrated into the editorial and productional flow of a programme. It was used by radio brand Studio Brussels during their yearly charity event ‘Music for Life’, an initiative to raise money during ‘The Warmest Week’, but also during the Football World Cup 2018 broadcasts from the television brand Sporza. Thanks to Hangouts, listeners and absent campaigners were able to dial in live and bring their story to air from anywhere in the world. The workflow ran completely automatically, saving the editorial team time and manpower. Consequently, they were able to welcome more campaigners for a personal interview. As such, Studio Brussel and Sporza were able to increase listener and viewer interactions. The Hangouts technology has been integrated into the flagship product offering, Kiswe Studio, in order to create more immersive live broadcasts. In 2021, Kiswe was named one of the World’s Most Innovative Companies by Fast Company and the company has grown by more than 70 FTE worldwide since their participation in the MOS2S project.

Unique, world-first sports and events experience

In the sports domain, several unique innovations were also developed.

Since the MOS2S project, Kiswe has been working with multiple sports leagues and entertainment and media companies worldwide, like K-pop group BTS, NBA, Universal Music Group and the Tour of Flanders to name a few. In close collaboration with OBS, the Hangouts technology was used at the Olympic Games in Tokyo 2020 (2021) and the Winter Olympic Games in Beijing 2022 through the novel media format ‘Athlete Moment’. Immediately after their performance, athletes were able to connect with family and friends, even before they were approached by the press. It provided beautiful and emotional video footage.

Through an exceptional SME collaboration, GameOn and Inmotio have implemented their video and sensor technologies in the Johan Cruijff ArenA. Thanks to this, visitors can get even closer to the spectacle, even from a distance. The live images can be streamed on a mobile device where you can watch live, replay, search for highlights, zoom in, draw, switch between cameras and play parts in slow motion. In addition, the technology is interesting for coaches because player movements can be easily analysed for training purposes. GameOn’s video technology has been licensed to 25 European clubs with a revenue of almost EUR 700 thousand for GameOn in 2019 (versus roughly EUR 80 thousand in 2016). The Inmotio Performance Centre is being rolled out for all 18 teams of the Dutch Eredivisie, potentially leading to millions of users following completion. As the first stadium in the world to adopt such technologies, the Johan Cruijff ArenA is boosting its reputation for innovation and opening up a new consultancy market, in turn making the technology accessible to new and existing sports hubs.

This innovative reputation was underlined in September 2018 when, for the first time in the world, a football match in the Johan Cruijff ArenA was broadcast in real time with only a 0.3-second delay from the pitch in Amsterdam to a viewing area in South Korea. Combining new Ultra-Wide Vision technology with a super-fast data connection enabled a crowd of South Koreans to experience the live event in an unmatched way; the entire stadium was displayed to them in panoramic high resolution, down to the tiniest drop of sweat and quivering blade of grass. This new technology makes it possible for people on the other side of the world to experience live events in real time as if they are at the event itself in person. Conducted by TNO, the Johan Cruijff ArenA, the Korean Electronics and Telecommunications Research Institute (ETRI) and SURFnet, the test broadcast was designed to demonstrate, amongst other things, whether European Championship matches can be experienced live worldwide as full stadium experiences.

In addition, MOS2S's technology was selected, out of 209 applications from 39 countries, to be demonstrated during the Eurovision Song Contest of 2020, which was unfortunately cancelled afterwards due to COVID-19. In the preparation phase for this event, organisations that had a solution to enhance the online experience of viewers and fans or a solution that supports or enhances the production of online content before and during the Eurovision Song Contest were invited to participate in this competition. The innovation challenge around the Eurovision Song Contest led to an overwhelming response from the international tech world. The entries varied enormously in terms of services and technology: from URL shorteners and hashtag generators to 8K virtual reality and AI. Eventually, three finalists were selected to actually implement their solution at the Eurovision Song Contest and two of these three solutions were derived from the MOS2S project:

• Ultra-Wide Vision, providing viewers all over the world with a live experience
• Live Hangouts, which has been integrated into Kiswe’s Studio product and enables a large number of customised streams aimed at different target groups with the choice of different commentators, such as social media influencers who are seeking interaction with the public and can show the results on the stream.

Strong international cooperation and social impact

One of the key successes of MOS2S was the ability to create strong international cooperation. They had national demonstrations led by the Netherlands, South Korea and Belgium in 2017, 2018 and 2019 respectively. However, in those demonstrations, almost all of project partners worked together to reach the ambitious targets of the project. The results of the demonstrations led to multiple commercial products and services.

For the crowdsourcing technologies, success is measured by audience. As younger people are increasingly unengaged with traditional media, the social impact of getting their unbiased views is high: it encourages participation in the democratic process and provides a new form of journalism that can combat fake news and help fight polarisation. Now that MOS2S’s applications have been tested in a Smart City Playground, the next step is to implement them on a wider scale to further revolutionise the role of citizens in both politics and entertainment.