Merging symbolic, physical and virtual spaces: Augmented reality for Iannis Xenakis’ Evryali for piano solo
Pavlos Antoniadis (EUR-ArTeC, Université Paris 8; TU Berlin-Audiokommunikation, Humboldt Stiftung)
Aurélien Duval (EUR-ArTeC , Université Paris 8)
Jean-François Jégo (INREV-AIAC, Université Paris 8)
Makis Solomos (Musidanse, Université Paris 8)
Frédéric Bevilacqua (UMR STMS IRCAM-CNRS- Sorbonne Université)
The proposed paper will present interactive systems for the visualisation and optimisation of extreme score-based piano performance. The systems are founded on an ecological theory of embodied interaction with complex piano notation, under the title embodied navigation (Antoniadis, 2018a; Antoniadis and Chemero, 2020). The theory has materialised in a modular, sensor-based environment for the analysis, processing and real-time control of notation through multimodal recordings, called GesTCom (Antoniadis, 2018b; Antoniadis and Bevilacqua, 2016). The motion capture modeling is based on an one-shot learning Hidden Markov Model developed at Ircam and called Gesture Follower (Bevilacqua et al., 2010). At a later stage, mixed reality applications have been developed on the basis of existent visualisation methodologies for motion capture (Jégo, Meyrueis and Boutet, 2019), seeking to create a virtual concert environment.
Drawing on music performance analysis, embodied cognition, movement modeling and augmented reality, we consider the concert experience as embodied navigation of performers and listeners in a hybrid environment. This environment capitalises on the isomorphisms and decouplings of physical, virtual and symbolic spaces, which merge in static and dynamic relationships: the performer’s gesture shapes music notation, music notation becomes an integral part of the concert space, a virtual avatar of the performer allows the audience to experience multimodal aspects of the performance which usually remain private, and so on.
The main focus of this presentation will be on a recent performance of Iannis Xenakis’ solo piano work Evryali employing live motion capture and augmented reality. Preliminary documentation of this project may be found in the following link: https://youtu.be/D-vhOX88NfM. This particular work problematises usual notions of virtuosity and performability, bears extra-musical references and is encoded in a unique graphic design. These features justify the task’s characterisation as extreme and demand a rethinking of technology-enhanced performance that combines sensorimotor learning, symbolic interpretation and multimodal feedback in novel ways.
Antoniadis, Pavlos (2018a). Embodied Navigation of Complex Piano Notation: Rethinking
Musical Interaction From A Performer’s Perspective, PhD thesis. Strasbourg: Université
de Strasbourg – IRCAM, 2018.
http://theses.unistra.fr/ori-oai-search/notice/view/2018STRAC007, accessed 14.04.2021
Antoniadis, Pavlos and Chemero, Anthony (2020). “Playing without mental representations:
embodied navigation and the GesTCom as a case study for radical embodied cognition in piano
performance”, in the Journal of Interdisciplinary Music Studies, special issue “Embodiment in
Music” following CIM19 conference in Graz, Austria (eds. Andrea Schiavio and Nikki Moran),
season 2020, volume 10, art. #20101207, pp. 126-174.
Antoniadis, Pavlos (2018b). “GesTCom: A sensor-based environment for the analysis, processing
and real-time control of complex piano notation through multimodal recordings”. Invited talk
at Séminaires Recherche et Technologie, IRCAM, 15.10.2018.
https://medias.ircam.fr/x2253e1, accessed 14.04.2021
Antoniadis, Pavlos and Bevilacqua, Frédéric (2016). “Processing of symbolic music notation via
multimodal performance data: Ferneyhough’s Lemma-Icon-Epigram for solo piano, phase 1”.
Proceedings of the TENOR 2016 conference, 127-136. Cambridge: Anglia Ruskin University 2016.
http://tenor2016.tenor-conference.org/TENOR2016-Proceedings.pdf, accessed 14.04.2021
Bevilacqua, F., Zamborlin, B., Sypniewski, A., Schnell, N., Guedy, F., and Rasamimanana, N.
(2010). “Continuous realtime gesture following and recognition”. In Lecture Notes on Computer
Science, Gesture Workshop, pages 73–84. Springer.
Jégo, Jean-François, Meyrueis, Vincent, and Boutet, Dominique (2019). “A Workflow for Real-time
Visualization and Data Analysis of Gesture using Motion Capture”. In: Proceedings of the 6th
International Conference on Movement and Computing. ACM. Phoenix, Az, USA, 2019. p. 1-6.