Mohammed Amine Togou, Anderson Simiscuka, Rohit Verma, Noel E. O’Connor
and Gabriel-Miro Muntean – 1st December 2021
Since the emergence of COVID-19, many opera houses have been temporarily shut down, pushing artists and artistic institutions to move their activities online to survive. This situation has contributed to the growth of digital art, where artists use new or customisable technological tools that enable them to collaborate remotely to produce captivating performances. These plays are recorded in the highest audio and video quality and are made available to the public through video streaming platforms (e.g., YouTube, Netflix).
To deliver these plays to viewers in the optimal way for their network connection, most streaming platforms use adaptive algorithms. These algorithms are based on the MPEG Dynamic Adaptive Streaming over HTTP standard, commonly known as MPEG-DASH. The standard uses a client-server architecture where videos are stored at the server in varying degrees of quality known as representations, each of which is split into multiple segments that are a few seconds long. To play a video, the client first requests the Media Presentation Description (MPD), which contains the list of all the segments that make the video along with the varying degrees of quality that are supported. With the help of an adaptation algorithm, the client then starts requesting segments with specific types of quality, considering the broadband capacity as illustrated in the picture below.
An example of how a DASH-based adaptation algorithm adjusts the video quality to meet the bandwidth capacity. Source: encoding.com, 2014
The goal of the adaptation algorithms is to provide the best overall playback experience, even when the speed of the broadband connection is slow. A good playback experience should ensure high quality for both audio and video while avoiding playback interruption. Yet, this is a challenging task.
Several adaptation algorithms have been proposed. Most of them only focus on adapting the quality of video in order to meet the broadband constraint. They do not adapt the quality of audio, which is an integral part of many performing arts pieces, particularly opera. Netflix proposed a new algorithm to allow audio quality to adjust to bandwidth capabilities during playback. It enables Netflix subscribers to enjoy studio quality sound, preserving the original creative intent of the show makers. Still, such an approach requires the use of the Dolby Digital Plus technology, which may not be available in many electronic devices.
Alternative approaches are being sought by large European Union Horizon 2020-funded projects such as TRACTION. One of the technological innovations of the TRACTION project is the development of the Co-creation Stage, a tool that connects communities and individuals in real-time and allows multiple co-located stages and participants to perform opera together. These performances can make use of both pre-recorded and live content. Nonetheless, the current version of the Co-creation Stage lacks the capability to adapt media content to bandwidth capacity to ensure a high quality experience. For this reason, the DCU team has been working on developing adaptation algorithms that consider both audio and video and can be easily integrated into the Co-creation Stage. One of these algorithms focuses on adapting pre-recorded media content. It selects the highest quality for audio and adapts the video quality according to the available network bandwidth.
Would the proposed algorithm provide better audio and video quality to viewers watching pre-recorded artistic content? This is the question that a 10-day study, which started on November 16th 2021, is trying to answer. Participants from various EU countries as well as non-EU countries such as India, Iran, and Morocco were invited to take part in the study. These participants are asked to watch four clips from “Só Zerlina ou Cosi fan Tutte?”, an opera play created by Portuguese partners SAMP.
Footage from “Só Zerlina ou Cosi fan Tutte?” by SAMP, Leiria, 2018. Photograph by Joaquim Dâmaso
They were then asked to fill in some quality of experience questionnaires. To allow participants to move seamlessly from one video to another and to do all the questionnaires online, a new TRACTION tool was created, as depicted in the images below.