What new possibilities does technological progress offer for our listening experience? The MetaPhase research project focuses on interaction between human and virtual musicians in a new world that encompasses both real and virtual space. As part of the European STARTS 2023 Award, the research has just received an honorable mention from the European Commission. MetaPhase was selected from over 1600 applications.
What is remarkable about this project is that the movements and gestures of the performers are captured with high precision by a motion tracking system and are embodied on the screen in the form of a digital avatar. This allows the audience to be swept along with every detail of the relationship between gesture and sound.
MetaPhase is the artistic result of a collaboration between Giusy Caruso and the innovative start-up LWT3 from Italy. Caruso is a concert pianist and postdoctoral researcher who focuses on the study of human-machine interaction (HMI) for the creation of futuristic multimedia formats and the analysis of musical performances. As chair of the CREATIE research group at the Royal Conservatorium Antwerp and affiliated researcher at IPEM-Gent University, she is officially music advisor to LWT3 Society Milan. LWT3 focuses on data analysis, visualization, IoT infrastructure development and human-machine interaction solutions.
This joint research was driven to bring the creative potential of data processing, human-machine interaction and biotechnological applications to life in an XR performance. The aim was to improve the expressiveness of the performers and the listening experience of the audience. The core is a portable, user-friendly system (prototype), developed by LWT3, that is capable of collecting biosignals.
[caption id="attachment_50279" align="alignleft" width="200"]
© Giusy Caruso[/caption]
The system maps gestures and provides quantitative data on displacement, acceleration and speed of movements. In this way, musicians gain valuable information about their physical approach to playing an instrument. This helps the musician better identify opportunities for improvement.
Based on the movements of musicians, the system also creates a virtual agent (usually an avatar) that moves on stage. The musicians are equipped with reflective markers on their body and a biosensor that capture the gestures and associated muscle effort in real time via infrared cameras. In summary, this form of co-creation of live music contains the first experiments with OptiTrack motion tracking, LWT3's biometric signal devices, and VR technology and the use of a Yamaha Disklavier piano.
From this perspective of visualizing bodily measurements, Giusy Caruso and LWT-3 developed the idea of also creating digitalized performances. For example, she plays Steve Reich's Piano Phase for two pianos through interaction with another avatar pianist who plays the first part of the piece. This second virtual person is animated by the expressive movement of the real pianist that was recorded earlier together with an audio track on a Disklavier piano.
The project underscores that we are in an era where increasing attention is being paid to hybrid approaches (physical+digital) in hybrid spaces (real+virtual). Contrary to what one might expect, people have become accustomed to living in virtual environments and communicating via smartphones or laptops. With the development of AR/VR projections, people can now also create interactions of their avatar in the 'metaverse' (a virtual world with avatars and tokens). This metaverse is only at the beginning of a development that still holds many mysteries and uncertainties but is gradually developing more and more with each step.