TRACTION Pre-recorded Content Adaptation

Mohammed Amine Togou, Anderson Simiscuka, Rohit Verma, Noel E. O’Connor
and Gabriel-Miro Muntean – 1^st December 2021

Since the emergence of COVID-19, many opera houses have been temporarily shut down, pushing artists and artistic institutions to move their activities online to survive. This situation has contributed to the growth of digital art, where artists use new or customisable technological tools that enable them to collaborate remotely to produce captivating performances. These plays are recorded in the highest audio and video quality and are made available to the public through video streaming platforms (e.g., YouTube, Netflix).
To deliver these plays to viewers in the optimal way for their network connection, most streaming platforms use adaptive algorithms. These algorithms are based on the MPEG Dynamic Adaptive Streaming over HTTP standard, commonly known as MPEG-DASH. The standard uses a client-server architecture where videos are stored at the server in varying degrees of quality known as representations, each of which is split into multiple segments that are a few seconds long. To play a video, the client first requests the Media Presentation Description (MPD), which contains the list of all the segments that make the video along with the varying degrees of quality that are supported. With the help of an adaptation algorithm, the client then starts requesting segments with specific types of quality, considering the broadband capacity as illustrated in the picture below.

An example of how a DASH-based adaptation algorithm adjusts the video quality to meet the bandwidth capacity. Source: encoding.com, 2014

The goal of the adaptation algorithms is to provide the best overall playback experience, even when the speed of the broadband connection is slow. A good playback experience should ensure high quality for both audio and video while avoiding playback interruption. Yet, this is a challenging task.
Several adaptation algorithms have been proposed. Most of them only focus on adapting the quality of video in order to meet the broadband constraint. They do not adapt the quality of audio, which is an integral part of many performing arts pieces, particularly opera. Netflix proposed a new algorithm to allow audio quality to adjust to bandwidth capabilities during playback. It enables Netflix subscribers to enjoy studio quality sound, preserving the original creative intent of the show makers. Still, such an approach requires the use of the Dolby Digital Plus technology, which may not be available in many electronic devices.
Alternative approaches are being sought by large European Union Horizon 2020-funded projects such as TRACTION. One of the technological innovations of the TRACTION project is the development of the Co-creation Stage, a tool that connects communities and individuals in real-time and allows multiple co-located stages and participants to perform opera together. These performances can make use of both pre-recorded and live content. Nonetheless, the current version of the Co-creation Stage lacks the capability to adapt media content to bandwidth capacity to ensure a high quality experience. For this reason, the DCU team has been working on developing adaptation algorithms that consider both audio and video and can be easily integrated into the Co-creation Stage. One of these algorithms focuses on adapting pre-recorded media content. It selects the highest quality for audio and adapts the video quality according to the available network bandwidth.
Would the proposed algorithm provide better audio and video quality to viewers watching pre-recorded artistic content? This is the question that a 10-day study, which started on November 16th 2021, is trying to answer. Participants from various EU countries as well as non-EU countries such as India, Iran, and Morocco were invited to take part in the study. These participants are asked to watch four clips from “Só Zerlina ou Cosi fan Tutte?”, an opera play created by Portuguese partners SAMP.

Footage from “Só Zerlina ou Cosi fan Tutte?” by SAMP, Leiria, 2018. Photograph by Joaquim Dâmaso

They were then asked to fill in some quality of experience questionnaires. To allow participants to move seamlessly from one video to another and to do all the questionnaires online, a new TRACTION tool was created, as depicted in the images below.

Screenshot of the mobile version of the tool to be used during the experimental study.

Through this study, we intend to identify the needs of diverse audiences and investigate how viewers’ quality of experience can be improved when watching pre-recorded content. The study will also help us to collect information and understand in detail the requirements from a user-centred point of view. These will further inform the design adjustments that might be needed for the adaptation algorithm in order to best enhance the TRACTION Co-creation Stage capabilities.

Acknowledgements: This work was supported by the European Union’s Horizon 2020 Research and Innovation programme under Grant Agreement no. 870610 for the TRACTION project. The support of the Science Foundation Ireland (SFI) Research Centres Programme Grant Numbers 12/RC/2289_P2 (Insight) and 16/SP/3804 (ENABLE) is also acknowledged.