"Autonomous and Robust Live Tracking of Complete Opera Performances"
Autonomous and Robust Live Tracking of Complete Opera Performances
Sprache des Titels:
This thesis is concerned with the challenging task of tracking complete opera performances in real time and reliably following along in the corresponding musical score. Robust real-time opera tracking (score following) would be extremely useful for many processes surrounding live opera staging and streaming, including automatic subtitling displays, camera control, or live video cutting.
So far, existing approaches to live music tracking have been demonstrated to be able to track full orchestral works accurately and robustly. However, these approaches fail to track operas. Such trackers must not only deal with a continuous musical recording but also have to consider a complex mixture of polyphonic music and singing voice, acting sounds, interludes, breaks, and also applause from the audience. All those parameters interfere with the tracking process and tend to provoke failures where the tracker is lost in the score.
To address this issue, we develop new methods to integrate different extra-musical knowledge sources into a state-of-the-art music score following algorithm to achieve robust and accurate opera tracking. We first collect and annotate real-world opera data (recordings, scores, and manual alignments between these) based on several performances of the opera ?Don Giovanni? by Wolfgang Amadeus Mozart, which will serve to evaluate our different trackers throughout the thesis. Then, we incrementally add our different contributions based on acoustic event detection, speech-specific features, acoustic modeling, and multi-resolution tracking to improve our opera tracking performances. Finally, we publish an open-access multimodal dataset including audio and video recordings of the opera ?Die Zauberflöte? by W.A.Mozart with corresponding manual annotations to stimulate research in that direction.
This project ends with the presentation of a first real-time opera tracking interface demonstrated on the opera ?Don Giovanni? with the playback of various YouTube videos. The interface is composed of a Score Viewer that highlights the currently played bar in the score book and a Subtitle Display that provides lyrics translation in various languages. We believe that such an interface has a very high potential, not only inside but also outside opera houses.