Enrique Tomás / Thomas Gorbach / Hilda Tellioğlu / Martin Kaltenbrunner (Eds.)

# **EMBODIED GESTURES**

Tomás / Gorbach / Tellioğlu / Kaltenbrunner (Eds.)

**EMBODIED GESTURES**

Enrique Tomás / Thomas Gorbach / Hilda Tellioğlu / Martin Kaltenbrunner (Eds.) EMBODIED GESTURES

This book has been published in the context of the Embodied Gestures research project financed by the Austrian artistic research funds PEEK programme (project number AR-G399).

Enrique Tomás / Thomas Gorbach / Hilda Tellioğlu / Martin Kaltenbrunner (Eds.)

# **EMBODIED GESTURES**

Cite as: Tomás, E., Gorbach, T., Tellioğlu, H., & Kaltenbrunner, M. (Eds.). (2022). *Embodied Gestures*. TU Wien Academic Press. https:/doi.org/10.34727/2022/isbn.978-3-85448-047-1

#### **TU Wien Academic Press, 2022**

c/o TU Wien Bibliothek TU Wien Resselgasse 4, 1040 Wien academicpress@tuwien.ac.at www.tuwien.at/academicpress

ISBN (print): 978-3-85448-048-8 ISBN (online): 978-3-85448-047-1

Available online: https:/doi.org/10.34727/2022/isbn.978-3-85448-047-1

(CC BY-SA 4.0). https://creativecommons.org/licenses/by-sa/4.0/

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License

Media proprietor: TU Wien, Karlsplatz 13, 1040 Wien Publisher: TU Wien Academic Press Editors (responsible for the content): Enrique Tomás, Thomas Gorbach, Hilda Tellioğlu and Martin Kaltenbrunner Production: Facultas Verlags- und Buchhandels AG

## CONTENTS


#### **PART I INSTRUMENTAL GESTURE**


**v**



#### **PART III EMBODIED GESTURE**




## PREFACE

We all learned how to communicate way before articulating our first words. Around the ninth or tenth month, humans begin to make requests with gestures. Eight months later children use both gestures and words at the same time. It is at this age when we find it useful to communicate different things with words rather than with gestures. We could assert that gestures help us to develop speech. Using gestures, children communicate ideas and obtain new words from adults in response to what they are calling for. Adults use gestures to communicate their ideas, needs, knowledge and feelings. This is why gestures are commonly understood as physical movements carrying information. Gestures are embodied language and cognition.

Gestures are well articulated movements. They usually have a clear beginning and an end we can easily recognize. Like any other communicative strategy, we can only understand what a gesture denotes in the context of a social or cultural practice. For this reason, gestures can even produce misunderstandings.

Gestures are spontaneous acts. It is not easy to identify where a gesture originates. Could all bodily movements be interpreted as gestures? Do we concatenate gestures while walking on the streets? What transforms the painter's moving hand into an artistic gesture? If gestures are embodied communication, why do we need them when we are alone? Gestures reinforce our inner thoughts, fears, reflections, etc. Gestures allude to mental imagery and memory. For instance, within the fields of music and visual arts, gestures are employed to describe musical pieces and artworks. Indeed, we are often invited to discover the gestural traces left by the artist with the paintbrush. In a similar fashion, Jimi Hendrix's gestures around his Stratocaster are already part of the social imaginary. His gestures carried communication but they mostly served Hendrix in driving his emotions on stage. These gestures were bonded to a flux of cognitive processes shaping the way he played.

Gestures are so deeply personal that they can be used to elaborate cartoonish representations. We can recognize people from their gestural pose. It seems that human life is driven by all these bodily movements we cannot easily control or understand. Just like the impossibility for Glenn Gould to forego chanting and whispering while playing Bach at the piano. Probably for Gould, musical scores triggered a complex sequence of mental images resulting in words, symbols and facial expressions that were absolutely necessary to define his sonic outcome. A good example of how much our mind and body are embodied in music and vice versa.

**xi**

This book uses the following hypothesis as its departure point: humans tend to deduce gestural and sensorimotor activity from auditory perception. In other words, we are inclined to determine both sound context and sound-producing gestures. First, we all instinctively have a tendency to identify potential threats from sonic events. Second, listening is not a passive action just happening to us. We need to actively learn how to give meaning to what we are hearing. In the embodied mind theory perception is not something we receive. It is something we actively do. During the action-in-perception loop, external stimuli would be incorporated as mental simulations, as reenactments of what we perceive. It is particularly important that these simulations can involve sensorimotor activations. That would mean that we all may have an internal representation of movement acquired from former experience that can be accessed from various sensory information. This cognitive model understands auditory perception partly as simulation and control of compatible gestures and bodily movements. We could say that listening extends aural information towards sensorimotor action. In fact, the opposite also occurs. We often find our feet tapping to rhythmic music. This spontaneous action results from our previous encoded sensorimotor simulations. Interestingly, this phenomenon can also be observed in other animal species. For instance, parrots who, without training, impulsively tap and shake their heads to the musical tempo.

The lack of appropriate language for describing auditory events forces us to verbalize changes in sound perception though gestures. For instance, people intuitively represent sonic changes by tracing imaginary lines in the air. This action––sound tracing––has been well studied. High frequencies are usually associated with gestures above our head. Low frequency sounds are located somewhere below our hips. Perceived temporal changes in sound––often called sonic morphologies––also refer back to sensorimotor sensations.

It is often said that sound and music have the power to create mental images, haptic sensations and artistic associations. This possibility affords artistic creation. It is the framework where composers and sound artists develop their practice. How has 'gesture' shaped the way we produce and understand music since the 20th century, especially since the experimental electroacoustic period? The development of technologies for audio recording, edition and reproduction inaugurated a new era for music research. Recorded sound could be edited in pieces and organized in time. In the 1960s, Pierre Schaeffer proposed a method for musical composition and analysis that departed from the auditory perceptual features of recorded sound. Schaeffer, influenced by Edmund Husserl's phenomenology, proposed separating the sound source from its context. Sound become disregarded from any external significations. Schaeffer built a musical solfege for recorded ` audio based on the notion of 'sonic object', the minimal auditive auditory perceptive unit, equivalent to the minimal perceivable sonic intention listeners can perceive. Sonic objects were characterized and classified depending on their perceptive character. This separation between object and subject originated a new kind of perception, the 'acousmatic listening', decoupled from the context in which the sound was recorded. As a reaction, Guy Reibel and Franc¸ois Bayle pointed out how this Schaefferian 'reduction' would interrupt the natural flow of communication between music and listener. For them, even in a reduced listening situation, listeners always associate sound with mental images, irremediably connected with their physical origin. Yet Reibel and Bayle proposed the extension of the notion of 'sound object' towards the concept of 'sound gesture'. With this strategy, it would be possible to overcome what they called the 'Schaefferian Sound Bazar'. They would use the listener's musical imagination as the channel of communication between composer and recipient. To this end, Reibel and Bayle proposed a compositional methodology based on the extensive use of sound produced as result of an action. Indeed, Franc¸ois Bayle, together with Jean-Christophe Thomas, wrote a book called Diabolus in Musica, in which around 150 different examples of the composition of sonic gestures are analysed and explained.

Following this methodology, Annette Vande Gorne, founder of the Studio for Musique & Recherche in Brussels, has been responsible for the development of new compositional methods based on the notion of 'sonic archetypes'. She described these major archetypes as: percussion-resonance, friction, accumulation of corpuscles, rebound, oscillation, swinging and swaying, flux, pressure-deformation, rotation, and spiral. For Vande Gorne, these archetypes would constitute a vocabulary of models, especially connected with the perception of physical features, which can be used to describe and design listening experiences. These sonic archetypes denote a flow of movement and sensorimotor action. Vande Gorne's influential methods can thus be understood as practical embodied cognitive research.

In parallel to Vande Gorne, Denis Smalley proposed a framework to describe the rich variety of sonic contents in electroacoustic music. He called it 'spectromorphology' and it consists of a set of tools for 'understanding structural relations and behaviours as experienced in the temporal flux of music'. Within this framework, the spectromorphology of a musical piece (i.e. temporal spectral flux of music) is mostly discussed in relation to gesture. For Smalley, gesture is an energy-motion trajectory creating spectromorphological life. Smalley specifically describes how listeners always tend to deduce gestural activity from sound and introduces the notion of 'gestural surrogacy', a scale of relationships between sound material and a known gestural model (e.g. first-, second- or third-order and remote surrogacy). For instance, in his third-order surrogacy level, gestures are imaged in the music. In the case of 'remote surrogacy', music is articulated from gestural vestiges. Developing his framework further, Smalley explains that listeners always attempt to predict the directionality of a morphological change. To illustrate this phenomenon, the author describes a sort of image schema (e.g. onsets, continuants, terminations) with possible metaphorical interpretations (e.g. for onset: departure, emergence, anacrusis, attack, etc). Smalley also illustrates processes for typical motion and growth processes (unidirectional, reciprocal, cyclic, multidirectional) and texture motion (streaming, flocking, turbulence, convolution). Similar categorizations are made in relation to spectral and spatial changes.

Some scholars argue that Smalley's image schemas are implicit embodied cognitive theory. The same would apply to Vande Gorne's methods. Under this hypothesis, electroacoustic and acousmatic music could be considered as embodied cognitive praxis extending its current theories. The practice of acousmatic music assumes the mental simulation of sound-producing gestures. A key aspect of Smalley's theories is that different types of gestures have different embodied-functional associations and, hence, causal dynamics.

With this book, our intention is visualizing how similar and compatible are the notions employed by scientists working in the field of embodied music cognition and the artistic discourses proposed by musicians working with 'gesture' as compositional material. 'Embodied Gestures' is also the name of the artistic research project developed between 2017 and 2021 by the editors of this book. Our aim was to study a new paradigm of interfaces for musical expression especially designed to emphasize a performer's gestural embodiment within an instrument. In order to achieve this goal, 'Embodied Gestures' explored the possibilities of shaping the physical affordances of designed digital instruments with the intention of inspiring particular forms of gesturality. Specifically, our objective was to study the implications of designing musical interfaces that can afford the same type of gesturality that a particular sound inspires. For instance, in order to control a sound passage composed from the circulation of 'rotating' sonic movements in space, we designed musical interfaces that afforded by themselves, and through their physical affordances, similar 'rotating' physical gestures to their performers.

Throughout the years of this artistic research project, we often found ourselves at an intermediate place between embodied music cognition, musical performance practice and musical interface design. After this project, we see both Smalley's gestural surrogacy and Godøy's sensorimotor models as complementary explanations towards describing the human tendency to deduce sound-producing gestures from what we hear. However, artistic practice can often better incorporate experiential aspects of the topics under research. Artists create knowledge by showing us unexpected realities and embodiments of sensory information. In the case of music, composers and musicians can question our understanding of why we use gestures, where they originate and their economies of production. For this reason, this book compiles manuscripts by researchers who have approached the issue of 'gesture' from both the scientific and artistic practice.

This book is structured in three parts called Instrumental Gesture, Embodied Object and Embodied Gesture. The first section includes four manuscripts. Cathy van Eck's 'Creaking Apples and s Singing Chairs: On Composing with Objects, Actions and Sounds' introduces two pieces by the author: *In Paradisum* and *Empty Chairs*. Van Eck explores the question of what are actions and objects to make music with, and what not. How do movements by the performer relate to what is sounding? And when all kinds of actions can be connected to all kinds of sounds due to the use of electricity, how do we decide which connections to use? Louise Devenish's 'Instrumental Infrastructure: Sheet Materials, Gesture and Musical Performance' discusses the entanglement of instrumental materials and gesture as compositional material in contemporary music, a field that increasingly explores the sonic and musical potential of a vast range of instruments, objects, materials and surfaces. Haize Lizarazu's 'Pre-gesture, Gesture and Sound on a [no] piano: Music from Somewhere' describes the process of performing a pianist work with 'no piano'. In it, the author demonstrates how by eliminating the instrument, composer and performer are able to focus on those prior moments to the production of visible gesture, the 'pre-gesture'. Through this artistic tactic they visualize a clue concept linked to memory during the study of a musical piece. The fourth manuscript of the first part explores ways to incorporate the notion of embodied gesture into Artificial Intelligence and User Interface Design. Koray Tahiroglu's 'Evolving Musical Expectations: Mu- ˘ tual Correlation between a Human Musician and an AI Musical Instrument' provides an overview of the current experiential techniques the author has used towards performing with artificial intelligence. Both his reflections as an artist and a musician are presented. In this article, Tahiroglu refers to a new technological-theoretical framework that can ˘ offer insight into the creation of compositions for musical instruments that are fully or partly autonomous, or autonomous in an unusual way. Through the composition, *Uncertainty Etude #2*, he discusses the use of artificial intelligence methods and a perspective from which a creative practice can be proposed to explore unusual musical expectations in a music composition.

In the second part of this book, Rolf Inge Godøy presents 'Timescales for Sound-Motion Objects". This article departs from a radical understanding of music as multimodal art, consisting of fragments of combined sound and body motion. Godøy extends the classical Schaefferian 'sound object' towards the notion of 'sound-motion-object'. The main idea is that timescales of Schaeffer's sound objects are closely linked with the timescales of body motion, and that thinking multimodal sound-motion objects in music would be useful for both analytic and creative work. Virginie Viel's 'Multi-form Visualization: A Method to Compose Acousmatic Music ' challenges the instinctive and natural tendency to draw connections between sonic and visual stimuli. In this chapter, Viel proposes to us an examination of the perceptual process, allowing us to examine the perception of qualia in music. Finally, the author demonstrates how conceiving the perceiver's mind as embodied could be the key towards the development of compositional practices based on the idea of multiform visualization, a personal method developed by the author. Pavlos Antoniadis's 'The Ocularcentric Objectification of Musical Embodiment in Cognitive Capitalism: Covid-19 as an Allegory on the Multiple Senses of Touch' is an essay triggered by the Covid-19 pandemic. This paper attempts a problematization of the notion of touch in musical performance. The de facto crisis of musical haptics due to physical and social distancing is here considered in the context of a wider phenomenon, namely the ocularcentric objectification of musical embodiment. By revealing a crisis of touch, the ongoing sanitary crisis invites us to further reflect on the meaning of musical haptics beyond the visual properties of embodied gestures and beyond tactility in the design of tangible user interfaces. In that sense, Covid-19 becomes a modern allegory on the multiple senses of touch, similar to the allegories of the senses in Flemish Renaissance painting.

The third part of the book is dedicated to presenting articles produced in the context of our Embodied Gestures project. In the first of these chapters, Enrique Tomas, ´ Thomas Gorbach, Hilda Tellioglu and Martin Kaltenbrunner present 'Embodied Ges- ˘ tures: Sculpting Sonic Expression into Musical Artefacts'. In this chapter, the authors describe the results of introducing a new paradigm of musical interface design inspired by sonic gestures. In particular, the authors discuss the beneficial aspects of incorporating energy-motion models as a design pattern in musical interface design. These models can be understood as archetypes of motion trajectories that are commonly applied in the analysis and composition of acousmatic music. The evaluation through composition and performance indicate that this design paradigm can foster musical inventiveness and expression in the processes of composition and performance of gestural electronic music. The second chapter in this section consists of an interview with Annette Vande Gorne conducted by Thomas Gorbach. During a summer workshop in 2020, Gorbach recorded the spontaneous and inspirational answers given by Vande Gorne to the question of how to engage the notion of gesture in acousmatic music. Additionally, she reflected on the possibilities of introducing the idea of energy-motion models into the physicality of the instruments employed to produce acousmatic music. Theodoros Lotis' 'Gestural and Textural Approaches in Composition and Performance with the Embodied Gestures Instruments" discusses a work composed for 'Embodied Gestures instruments', describing how they facilitate the creation of rhythmical structures and looping processes, since many of their gestural typologies occur as repetitive patterns. In his opinion, apart from the evident approach of the gestural behaviour, the instruments could also be used for the control of the spectral/textural evolution (micro- and macro-structural) of sound objects or sonic structures. Jaime Reis' 'Exploring Polyphony in Spatial Patterns in Acousmatic Music' introduces the readers to the use of space as a parameter in acousmatic music––along with some of the main perceptive features involved in such practice. After a brief description of acousmatic music techniques, Reis discusses these relations in regard to gestures. Finally, the author describes how he has enhanced spatial polyphony through the use of Embodied Gestures instruments in his compositions. Finally, Hilda Tellioglu's 'User-centred design as a model-based co-creation process' reviews the it- ˘ erative user-centred design (UCD) process as an adaptive and agile life-cycle for open innovation and successful development of socio-technical systems. In UCD, the user acceptance and usability of a digital system are central and determine the way to proceed at several stages of a design process. The question presented in this chapter is how to ensure a successfully realized user experience in a new design. In other words, how to proceed in a design project to understand the target users and their context, including their past experiences, and to consider this insight in the design of artifacts and interactions provided as part of the new design. The author's answer to these questions is applying modelling in all phases of the design process by creating models of all findings gathered after studying the target users and their past and current contexts, as well as by preparing and accompanying the design process as a reflective and self-critical practice.

We hope this book contributes to the ongoing discourse and discussion around creative technologies and music, expressive musical interface design, the debate around the use of AI technology in music practice, as well as presenting a new way of thinking about musical instruments, composing and performing with them.

ENRIQUE TOMAS´

*Linz, Austria October, 2021*

## CONTRIBUTORS

#### **Pavlos Antoniadis**

Humboldt Stiftung Fellow, Audio Communication Group, TU Berlin, Germany

## **Louise Devenish**

Sir Zelman Cowen School of Music and Performance Monash University, Australia

### **Rolf Inge Godøy**

RITMO Center for Interdisciplinary Studies in Rhythm, Time and Motion, Department of Musicology, University of Oslo, Norway

### **Thomas Gorbach**

Artifact-based Computing & User Research (ACUR), Institute of Visual Computing & Human-Centered Technology, Faculty of Informatics, TU Wien, Austria

### **Martin Kaltenbrunner**

Tangible Music Lab, Institute for Media, University of Art and Design Linz, Austria

**xvii**

#### **Haize Lizarazu**

Performer and Researcher, Universitat de Girona, Spain

## **Theodoros Lotis**

Department of Music Studies, School of Music and Audiovisual Arts, Ionian University, Greece

## **Jaime Reis**

ESML / IPL - Escola Superior de Musica de Lisboa, Instituto Polit ´ ecnico de Lisboa, ´ IDI&CA; INET-md / FCSH-UNL - Instituto de Etnomusicologia - Centro de Estudos de Musica e Danc¸a da Faculdade de Ci ´ encias Sociais e Humanas da Universidade ˆ Nova de Lisboa, Portugal

## **Koray Tahiroglu˘**

Department of Media, School of Arts, Design and Architecture, Aalto University, Finland.

## **Hilda Tellioglu˘**

Artifact-based Computing & User Research (ACUR), Institute of Visual Computing & Human-Centered Technology, Faculty of Informatics, TU Wien, Austria

## **Enrique Tomas´**

Tangible Music Lab, Institute for Media, University of Art and Design Linz, Austria

## **Annette Vande Gorne**

Composer, Association de Musiques et Recherches and the Metamorphoses d'Orph ´ ee´ studio, Belgium

## **Cathy van Eck**

Department of Music, Bern Academy of the Arts, Switzerland

## **Virginie Viel**

Composer and researcher, Paris, France

## ACKNOWLEDGMENTS

Publishing this book was possible thanks to the funding received from the Austrian Programme for Arts-based Research (FWF PEEK). The same programme financed four years of the Embodied Gestures artistic research project (PEEK AR-399).

We greatly appreciate the comments and contributions of our anonym reviewers and TU Wien Academic Press. We also thank Ross Ludlam for his exhaustive work proofreading these articles.

Finally, the editors would like to thank the contributors of this book. We are deeply grateful for your work and commitment during these difficult COVID-19 times.

**xix**

## LIST OF FIGURES


**xxi**


## LIST OF TABLES


**xxiii**

**PART I**

## INSTRUMENTAL GESTURE

## #1 CREAKING APPLES AND SINGING CHAIRS: ON COMPOSING WITH OBJECTS, ACTIONS AND SOUNDS

In my two compositions *In Paradisum*<sup>1</sup> (2019) and *Empty Chairs*<sup>2</sup> (2018) I use objects––apples and chairs––and simple everyday actions––eating the apple and replacing the chairs. In both pieces, these actions control a computer patch that produces all kinds of sounds, and this combination results in a composition for objects, actions and sounds. My two pieces *In Paradisum* and *Empty Chairs* came forth from several questions. What are actions and objects to make music with, and what not? How do movements by the performer relate to what is sounding? And when all kinds of actions can be connected to all kinds of sounds, due to the use of electricity, how do I decide which connections to use? To elaborate these questions and thoughts on what kind of interdependencies I try to develop during my compositions I discuss and compare works by three other composers––Nicolas Collins, Chikashi Miyama and Joanna Bailie.

An important starting point for these pieces is what I would like to call 'the origins of sounds'. These origins can be divided into three categories: the corporeal origins, the physical origins and the imaginary origins. I developed these categories because they are helpful for my compositional work with movements, sensors and sounds.

The corporeal origins can be defined as the movements by the musician causing the musical instrument to sound. With acoustic instruments this is obvious: a musician has to make movements to produce any sounds at all. These might be big movements, such as needed when playing the cello, or smaller movements, such as when playing the clarinet. Non-electronic instruments each demand a specific physical effort from the player (Craenen, 2014, p. 150). Controllers and instruments used to control electronic

**3**

<sup>1</sup>Some pictures of In Paradisum by Cathy van Eck: https://www.cathyvaneck.net/in-paradisum/ (Accessed: 1/12/2021)

<sup>2</sup>A documentation of Empty Chairs by Cathy van Eck: https://www.cathyvaneck.net/empty-chairs/ (Accessed: 1/12/2021)

sounds also demand a physical effort, albeit often with less constraints. In electronic music one can push a button or move a fader to trigger a complex sound. Additionally, the relationship between physical effort and resulting sound is not as linear (or even not linear at all) as is the case for acoustic instruments. Especially when using digital means, the same movement on the same controller could trigger a different sound, depending on how a computer or synthesizer is programmed. But also with these kind of set-ups the claim Craenen (2014) makes is valid: during sound production 'the visibility of synchronous physical action [. . . ] can augment auditory perception or push it in a certain direction' (p. 150). The movements visually perceptible when someone makes music influences how the sound is heard.

The second category is the physical origins. These are the movements through which the sound itself is produced. For acoustic instruments these are, for example, vibration by strings, pieces of wood, columns of air, etc. With electronic means these movements are nearly always executed by the diaphragms of loudspeakers. For acoustic instruments these physical origins are often firmly coupled to corporeal origins. The physical origins are defined by the design of the instrument. To bring the instrument into vibration, and thus to produce sounds, is achieved by specific movements of the body of the performer. The corporeal origins are merely predefined by the instrument itself, and thus by the physical origins. It is hard to produce sound by blowing on a violin, and not easy to bring out many sounds by hitting a clarinet either. This is different for instruments using loudspeakers for producing sound: the corporeal origins are not prescribed by the physical origins, but can be designed in many different ways. The physical origins are always the same in electronic music: loudspeaker diaphragms vibrating according to an electric signal. As Thor Magnusson (2019) states, 'the electronic or digital instrument *has* an interface, whereas the acoustic instrument *is* the interface' (p. 35). For electronic music you can choose whether you want to blow, hit, or bow the sound. This is much less easy, if not impossible, with acoustic instruments. The same movement always results in the same sound. But as I will argue in this text, this unreliability of the relationships between corporeal and physical origins in electronic music is an appealing feature of composing for movements and sounds.

The third category is the imaginary origins. Especially in electronic music this category is of much importance. As mentioned above, the physical origins will always be the same loudspeaker vibrations, but the imaginary origins of these sounds can be manifold. When listening to sounds I might recognize different sound sources: I hear drops of water falling, someone breathing or footsteps passing by. Evidently I can never be sure how these sounds have originally been produced, and that actually does not matter much for my perception. As Michel Chion (2010) describes, these imaginary sound sources replace the real sound sources, which are the ways in which a sound has been produced (p. 45). It is easy to simulate the sound of a train with the help of small objects in front of a microphone. Conversely, train sounds could be processed unrecognizably in a piece (Chion, 2010, p. 31). The imaginary origins do not have to refer to already known and identifiable sounds. These can just as well be abstract synthesized sounds. The recognition of imaginary origins is of course partly dependent on the perception of the listener. Whereas one person might identify different sounds as being from the same imaginary source, another person might identify several different imaginary sources. For composers this is of course also one of the big advantages used in many compositions of electronic sound: one can smoothly transform from one imaginary sound source to another. Many acousmatic pieces use this as a very important feature of their compositional technique. Composers such as Hildegard Westerkamp, Trevor Wishart or Annette Vande Gorne create all kinds of new imaginary origins, which can only exist in sound itself: a voice taking flight in *Red Bird* (Wishart, 1996, p. 166), falling raindrops becoming a musical rhythm (Norman, 2004, p. 80), or the buzzing of insects developing into a melodic line (Vande Gorne, 2021).

In the next few examples I would like to investigate how different compositions connect these different origins, and what kinds of compositional relationships these connections create.

#### **1.1 Using an everyday set-up as the corporeal origin for sounds**

In both my compositions I was interested in investigating objects and actions that have the least possible relevance in typical music practices. The objects used should neither be played expressively, nor be able to control electronic sounds in a virtuosic way; rather they should merely be used as they commonly are in their accustomed environment. I envisaged the idea of focusing on very simple actions and using them as the part of the set-up that controls the sound production. On instruments, whether standardized or newly developed ones, a performer can discover sounds and be able to practice, to control sound production very precisely, and eventually become a virtuoso. But I became curious about what would happen if I chose the opposite path: looking for an action that is not at all suitable for playing an expressive musical performance, which does not give me many possibilities for sound control and forces me to move in a certain way. For that reason, I chose eating an apple for *In Paradisum* and moving chairs for *Empty Chairs*. Both are actions with many possible associations. Besides being a very common fruit that is eaten worldwide, the apple has also a long cultural history. Eve in paradise, the judgment of Paris and the fairy tale of Snow White are a few examples. Chairs are very common furniture also used by many on a daily basis. The way chairs are positioned expresses a lot about how they are used. In a row, next to each other, in groups of two or in a circle; all these positions yield different associations of certain types of social gatherings.

A beautiful example of controlling electronic sounds with a very common everyday action is *In Memoriam Michel Waisvisz* (2009)3 by Nicolas Collins (Figure 1.1). In this piece Nicolas Collins uses the light created by the flame of a small candle to control electronic sounds. Michel Waisvisz was a composer and instrument inventor who worked a lot with all kinds of gesture and sensor interfaces, such as the *kraakdoos* (Dutch for cracklebox) (1974). The kraakdoos is played by making connections with the skin of your fingers between different parts of the circuitry. In his *In Memoriam Michel Waisvisz*, Nicolas Collins replaces the lively performer with a small candle, and the circuit is controlled by photoresistors that change their resistance depending on how much light they receive from the candle. Besides the candlelight getting closer the more of the candle has burnt down, the position of the flame also influences the sound. To change the position of the flame Collins uses a small fan. Four oscillators producing sounds that resemble

3A video of the work *In Memoriam Michel Waisvisz* by Nicolas Collins can be watched here: https://www.youtube.com/watch?v=sBlIRdnPciw (Accessed: 1/12/2021)

Figure 1.1: In Memoriam Michel Waisvisz (version 2015) *by Nicolas Collins (video still) (Video: ©Henrik Jonsson, 2015)*

the sonic character of a kraakdoos are controlled by the movements of the flame.4 As Nicolas Collins explains: 'the reason that circuit sounds the way it does is that the four oscillators are tuned in rough unison and when the flame moves around in the field of the four photocells it is detuning each against the others' (Collins 2021).

What makes the relationship between the movements of the flame and the electronic sound very strong is the connection of two phenomena which are commonly associated with two very different processes: the burning of a candle––an act reminiscent of the age before electricity––and sounds produced by oscillators––not possible without electricity. Evidently there is no such thing as a natural cause between the candle and the sound; here the electronic circuit design makes the analogue changes in flame and electronic sound possible. Even though the flame is of course not a human being, the movements of the flame can be regarded here as replacing the movements of a performer. Due to the synchronous changes in light and sound the corporeal origins of the sound seem to be the movements of the flame. A fascinating and poetic interaction between these two sources is the result.

Whereas the connection between the movements of the candle flame and the sound created by the oscillators is continuous, in my two pieces I used a less continuous connection between objects and sounds. I chose two discrete parts of the process of eating an apple for controlling the sound processing: biting into the apple (detected by a contact microphone placed upon the apple) and chewing an apple piece (detected by a contact microphone on a chewing muscle on my cheek). For the chairs the main parameter con-

<sup>4</sup>A description of a similar circuit can be found in Nicolas Collins' book Handmade Electronic Music (2020, p. 100).

trolling the sound processing is whether a chair is moving or not. These movements are detected and sent to a Max patch on my computer by iPhones attached to the underside of the chairs.5 All these controls are either simple triggers (biting and chewing) or on and off (chairs moving or not). This kind of control could also be done by a simple knob controller. The difference would be in the constraints added to the movements of the performer by using these objects. Eating an apple asks for a certain amount of time, and whereas a knob can be pushed very often and quickly, each bite of an apple needs a certain amount of chews to be swallowed. Chairs are bulky objects, and carrying and shoving them around also requires a different timing than pushing a button on a controller. The timings of these everyday actions become obligatory musical timings in my compositional process.

#### **1.2 Changing the mapping between the corporeal origins and the imaginary origins**

In *In Memoriam Michel Waisvisz* the movement of the flame controlled a different electronic circuit, and therefore completely different sounds. The relationship between corporeal origins and imaginary origins can easily be changed, especially when software is used instead of hardware. A good example of these kinds of changes in mapping between controller and synthesizer can be found in the piece *Black Vox* (2009) by Chikashi Miyama.6 He developed his own instruments because he is 'not so attracted by traditional instruments such as piano or flute, because they are too biased towards the existing musical scale' (Miyama, 2020). He constructed the Peacock, an instrument with 35 infrared sensors that control more than 300 synthesizer parameters in a PureData patch. The patch is a phase-bash-algorithm-based synthesizer (Miyama, 2010, p. 381). For physical instruments, parameters such as spectrum and pitch or volume and pitch are often coupled to a certain degree. Playing louder on a piano changes not only the loudness, but also the spectral characteristics of the sound. In Chikashi Miyama's instrument seven sensors in a row control one voice of the synthesizer, resulting in a maximum of five voices. Changes in distance to the performer's body will make their output value lower or higher (p. 382). All kinds of parameters of the synthesis process can be shaped by the hand and arm movements independently.

What makes Miyama's set-up significantly different from conventional instruments is that the mapping between body movements and sounds changes during the performance. The same hand movement will yield different results at the beginning, in the middle or at the end of the piece (Miyama, 2010, p. 382). The imaginary origins thus change, although the corporeal origins stay the same. This change of mapping is based on a timeline. Miyama can follow a score in his Pure Data patch, which lets him know both where in the piece he is and what the current mappings are (Miyama, 2020). Due to

<sup>5</sup>There are actually two short moments in the piece where the movement of a chair is controlling an oscillator continuously. This is done by tilting the chair, whereas during the rest of the piece the chairs are carried or shoved. There are also some sections in which the microphone is picked up by the performer and also used to trigger sounds. These sections are omitted from this article for clarity.

<sup>6</sup>A video of Black Vox by Chikashi Miyama: https://www.youtube.com/watch?v=QX1y3v3tk4w (Accessed: 1/12/2021)

Figure 1.2: Empty Chairs *by Cathy van Eck (Photo: Marije Baalman, 2019, CC BY)*

the fixed relationships between body movements and sounds in combination with the fixed timeline for the change of these mappings the piece can be rehearsed and every performance will be pretty similar.

The relationship between the performer's movement and the resulting sound––and thus between corporeal and imaginary origins––changes too during *Empty Chairs*(Figure 1.2). I mainly place three chairs in different positions on stage. In the beginning I position the chairs similar to how one would place chairs in a space used for a conference. During the piece the chair positions signify other social gatherings: placed close to the audience, or all three next to each other at the back of the stage, or two together and one alone––and by the end of the piece the three chairs form a circle. These different set-ups of the chairs control completely different sonic structures. In the beginning of the piece the sound of my footsteps is recorded every time I pick up a chair (the movement of the chair triggers the recording to start). When the chair is put on the floor again (and thus stops moving), this recording is played back at irregular intervals through the loudspeaker attached to the chair. The origins of these footsteps recordings are easily identifiable here, since the production has just been heard and seen before. During the piece the movements of the chairs control all kinds of different sound processes, starting with sounds close to the chairs itself, such as the sounds of chairs being shoved, but soon moving away from that recognizable sound by changing the parameters of a polyphonic sampler of more abstract sounds, and ending with acoustic feedback. The corporeal and imaginary sources diverge more and more during the piece. In many instrumental pieces the performer's movements can be heard in the sound. The corporeal origins are therefore often mirrored in the imaginary origins: fast body movements result in fast-changing sounds, slow movements in slow-changing sounds. In contrast to this instrumental practice, the sounds I control in *Empty Chairs* change much more than my actions would suggest. The imaginary origins are manifold, compared to what one would expect from the corporeal origins.

#### **1.3 Composing structures for imaginary origins of sounds**

The interweaving of different kinds of imaginary origins can be found in many of Joanna Bailie's compositions, for example 'Artificial Environment 5' from the piece *Artificial Environments 1-5* (2011).7 As in many of her compositions, Bailie uses field recordings in combination with live playing of instruments in this piece. She describes her way of combining field recording with instruments as being based on '[. . . ] a kind of Cageian/Duchampian belief in the power of *framing*, the act of transforming real-life non-art into art through placing it in an artistic context or by just seeing or hearing it in a different way' (Saunders, 2012). At the beginning of this part I recognize cheerful barrel organ music and the noise of happy children's voices. But then the sound of bells starts passing by in a similar rhythm to cars on a road. Since the former movement of the piece, 'Artificial Environment 4', starts with clearly recognizable recordings of cars passing by, my ears are already trained for this sound. I therefore easily recognize the car-passing rhythm in the bell sounds. These sounds are accompanied by acoustic instruments, which are often playing downward glissandi, similar to a Doppler effect, and therefore enhancing the effect of a car passing by. Pretty soon, though, I not only hear bells passing by as if they were cars, but also the barrel organ I heard before, and I seem to recognize some car horns. Then the rhythm returns to the barrel organ once more.

Listening to this mixture of very precisely composed sounds I am in constant doubt of what I am listening to. This sonic amalgam can only be created through sounds themselves; I can hear sonic events through structures that seem remnants of other sonic events. When listening to this piece, I find myself oscillating between first being able to categorize or recognize what I hear, and then not. It would be impossible to compose any corporeal origins for these imaginary origins. In my view, these 'imaginary sound sources' (Chion, 2010, p. 45) would immediately be reduced in their compositional possibilities and perceptible interpretations by 'the visibility of synchronous physical action' (Craenen, 2014, p. 150). To come back to the citation by Paul Craenen at the beginning of this text, in this piece, the corporeal origin would diminish the auditory perception and impoverish the acoustic fluidity of the sonic materials.

In *In Paradisum*, on the contrary, I try to work with these impoverishing qualities of the performer's movements: all kinds of different sounds are connected to one simple movement: chewing. Starting with just amplifying the common sounds of chewing an apple with every new bite in the apple, other imaginary origins appear. At the beginning

<sup>7</sup>A video with score of *Artificial Environments 1-5* by Joanna Bailie: https://youtu.be/UWrsqDSL eU (Accessed: 1/12/2021). The piece contains many more elements than described here. An important element is a spoken voice, which I left away in my analysis since it is not used in this particular fragment.

of eating the apple these are barely audible soft-pitched sounds, but during the performance the apple chews get many different kinds of imaginary origins: they are loud and percussive, or long filtered noises. The changes in the corporeal origins are minimal: I chew a bit more regularly when triggering the percussive sounds, or I chew more slowly when triggering the long filtered noises. One of the main aims of my composition is to compose this discrepancy between the movements of the performer and the resulting sounds. By creating essentially simple movement-sound relationships (every bite is a change in imaginary origin, every chew is a new sound) the changes in this relationship become my main focus. These ambiguities between what is heard, what seems to have caused the sound, and what is visually happening are all part of composing connections between corporeal and imaginary origins. How can the same movement be connected to completely different sounds? How musical can chewing an apple become? By composing relationships between objects, actions and sounds I try not to answer these kinds of questions. I try to raise these doubts through composing the origins of the sounds.

#### **REFERENCES**


## #2 INSTRUMENTAL INFRASTRUCTURE: SHEET MATERIALS, GESTURE AND MUSICAL PERFORMANCE

*Thin sheets of material are suspended and placed around the space: tracing paper, silver acetate and aluminium panels. Smaller aluminium panels rest on low plinths, overlapping but not touching one another, surrounded by layers of tracing paper laid beneath them. Before being sounded they are still, flat, bright. The flat panels are first activated by rice dropped in single grains, and then cascades. The paper is touched with fingertips and nails, swept and rustled, before being scrunched and dragged across other material surfaces. The larger aluminium panels and acetate sheets are suspended vertically, hung like mirrors in two rows. Speakers placed behind these hung materials project prepared electronics and acoustic sound. Miniature microphones hidden at the performer's wrists are brought close to the sheets of material, tracing lines and shapes on their surface, first simply picking up otherwise inaudible frequencies before later being used to induce feedback loops with speakers set behind the hanging layers of aluminium and acetate. Sound itself activates the acetate sheets, which tremor and rustle: they are sounded by sound. Shards of reflected light are thrown around the space, making sound vibrations visible. A performer moves between the corridors created by the hung materials that make up this instrumental infrastructure, glimpsed between the mirrored panels as they are sounded using hands, mallets, superballs. The flickering light now resembles rippling water as the performer moves forwards. Half hidden from view, a single sheet of tracing paper is folded into an origami paper boat and placed in the fading pool of rippling light.*

#### **2.1 Introduction**

Over the past 20 years, themes of practice focused on rapidly expanding the *tools* (instruments, objects, materials), *medium* (composed music, performed sound) and *methods* (collaborative practice, compositional practice, performance practice) of music creation and performance have become central to new Western art music. An increasingly significant theme of practice is the musical use of both sonic and non-sonic materials in the development of new work. Ciciliani suggests that this is driven by artists who are 'working from the understanding that sound alone is no longer sufficient to express their musical ideas' (Ciciliani, 2017, p. 24), yet still seek to root their work in musical practice (Walshe, 2016). It is particularly apparent in works with interdisciplinary or intermedial influences. As new compositional and performance practices exploring the sonic/nonsonic have developed, in some works an entanglement of material roles has taken place. In these works, materials that are used instrumentally are also used as infrastructure or props, or even as notation.1 In a number of percussive works, the distinction between instrumental object and implement (or mallet) is also less clear. This entanglement of roles is a key feature of an approach I refer to as 'post-instrumental practice'.2

The opening paragraph of this chapter describes *Permeating Through the Pores of Shifting Planes* (2019) by Annie Hui-Hsin Hsieh. *Permeating Through the Pores of Shifting Planes* (hereafter *Shifting Planes*) was commissioned for *Sheets of Sound*, a percussive performance project by Louise Devenish centred around the use of sculptural instrumental installations in combination with electronics.3 Although some standardized percussive items were permitted, the exploration of non-standard instrumental materials was prioritized. The starting point for *Shifting Planes* included exploration of the instrumentality of various sheet materials, from both a sonic and a non-sonic perspective. Three questions guided the development of the work: When using unfamiliar instrumental materials of varying scale, how does the relationship between the visual and the auditory change? How can performative gesture be used as compositional material when using large instruments distributed around a performance space? How can sheet materials fulfil plural roles on stage (instrument, infrastructure, prop, mallet, etc.)?

#### **2.2 Instrumental sheet materials**

*Shifting Planes* used a range of sheet materials including paper, silver acetate, and aluminium, as shown in Figure 2.1. The core of the setup was eight custom-made aluminium panels, cut at random in either square or rectangular sections from a 5 mm thick aluminium sheet. The four smallest panels were placed horizontally on plinths towards the front of the space, with several large sheets of tracing paper placed in layers beneath them. Upstage, the four larger panels were suspended vertically. Behind the suspended panels were four sheets of acetate, each concealing from view a large speaker on a stand.

<sup>1</sup>For examples of instrumental infrastructure, see Ash and Adam Fure's *The Force of Things: An Opera for Objects* (2017) or Matthias Schack-Arnott's *Everywhen* (2019). For an example of materials used simultaneously as instrument, infrastructure and implement, see Kate Neal's *Never Tilt Your Chair Back* (2017). For an example of instruments as notation, see Enrique Tomas' ´ *Tangible Scores* (2014-2018) or Mark Applebaums's Straightjacket, movement 4: *Taquinoid* (2009).

<sup>2</sup>This forms the basis of my Australian Research Council project 'The role of post-instrumental practice in twenty-first century music' (2020–2023).

<sup>3</sup>Other works commissioned for this project include *Percipience: After Kaul* (2019) by Louise Devenish and Stuart James, and *Catacomb Body Double* (2019) by Matthias Schack-Arnott. The premiere was presented by Tura New Music, with funding support from the Australia Council for the Arts.

Figure 2.1: *The instrumental setup of* Permeating Through the Pores of Shifting Planes *(Photo: Nik Babic, 2019, CC BY)*

The randomized sizes of the aluminium panels formed a unique and specific pitch set, with each panel capable of producing a different range of frequencies depending on how and where on their surface or edges they were activated. For example, the largest panel produced a clear, low frequency when struck in the centre with a soft mallet, as well as very high, sparkling frequencies when struck on the edge with a thin metal beater at a 90-degree angle. The various materials themselves encouraged different application of tools, techniques and movements to each sheet, further diversified by how they were set up, which is shown in Table 2.1. In addition to the usual percussive techniques of striking, rubbing, shaking, scrunching and dropping, the sonic palette was expanded further through the use of microphones and loudspeakers to incorporate non-contact, gestural activation techniques.

The use of microphones and loudspeakers as musical instruments is now a common practice, with numerous works using only these materials as their instrumentation. The evolution of this practice and associated repertoire has been well documented (van Eck, 2017), and is common enough within percussive practice that the musical use of microphones and loudspeakers has been included in recent pedagogical texts focused on percussive implements (Dierstein, Roth, & Ruland 2018). Notably, the use of small DPA microphones (such as the 4060 series of miniature omnidirectional microphones) attached to performers' wrists in performance is emerging as a standard 21st-century percussive technique, as microphones placed at the wrist facilitate close microphone access to sounds and sounding techniques at their source.<sup>4</sup> This can be particularly effective in works exploring varied proximity to sound, and in works with large or distributed set-ups.

In *Shifting Planes*, DPA wrist microphones are initially used to access otherwise inaudible acoustic sounds produced by very small movements, such as the tactile sound of fingers brushing on paper. In the central section of *Shifting Planes* the fundamental tones of the aluminium plates are sounded using mallets or hands, before an additional, otherwise inaudible melodic line is brought forward from the decaying resonance using non-tactile gestures to guide the microphone across different areas of the panel to amplify a range of frequencies. This sound world was supported by an electronics part, that could be played with or against during this section. As both the wrist microphones and the four speakers sounding the electronics are hidden from view, the distinctions between who or what is making the sound become blurred. For a short section, the three types of sound-related gestures that occur in music performance––sound-producing, soundaccompanying or sound-tracing––occur simultaneously (Godøy, 2006, p. 154). Later in the work, the wrist microphones are used to stimulate feedback loops with the concealed speakers, created by the performance of gentle, specific gestures that are determined in rehearsal during tuning of the PA to the performance space. The resonance also activates the hanging acetate sheets between them, shown in Figure 2.2. The non-tactile means of activation enables the audience to hear the vibrating sheet materials without also hearing attack or friction sounds. As the microphones and speakers are concealed and the performer does not make contact with the acetate sheets, again it is not immediately clear who or what is making the sounds. Here, a 'dance of agency' (Pickering, 2012) between human, instrument, microphone and loudspeaker occurs, as the sonic result is not entirely controlled by either human or materials. This is further impacted by the acoustic of each space in which the work is presented, and is monitored by an off-stage technician during performance.

Godøy hypothesizes that 'there is a continuous process of mentally tracing sound in music perception. . . mentally tracing the onsets, contours, textures, envelopes, etc., by hands, fingers, arms or other' (Godøy, 2006, p. 149). Hsieh seeks to explore this notion in her work, and with regard to the development of In *Shifting Planes*, states 'I'm very fascinated by the relationship between the visual and the auditory in performance. What am I seeing in the performance that informs me about what I'm hearing? How do you revert the expectation between *you-hit-something, you-hear-something*? What happens when you give a super big gesture, but the sound is almost inaudible [or vice versa]? Is that perceived as loud or soft?' (Hsieh in Devenish, 2020). With these questions in mind, additional compositional materials were drawn from the gestural possibilities that emerged from the choices of instrumental materials and how these materials were arranged for performance.

During the creative development of *Shifting Planes*, it quickly became apparent that gestures and techniques that were effective on unamplified materials were ineffective when the microphones were used and vice versa. As choreographer Jonathan Burrows notes, 'technique is whatever you need to do, to do whatever you need to do it' (Burrows, 2010, p. 68), and for *Shifting Planes*, this meant drawing on and adapting established

<sup>4</sup>In addition to *Shifting Planes*, some examples of recent Australian percussion works using this technique include Matthias Schack-Arnott's *Annica* (2016), and a number of the works in Speak Percussion's *Fluorophone* project (2015).

Figure 2.2: *Non-tactile activation of the silver acetate sheets via microphone and loudspeaker feedback loops (Photo: Nik Babic, 2019, CC BY)*

percussive techniques, microphone techniques, performative gestures and everyday gestures to develop a set of techniques specific to this work, both in the composition and performance stages.

#### **2.3 Gesture as compositional material**

In addition to the materials themselves contributing to the design of the work, a significant part of the compositional process included analysis of the performer's movement and physicality used in video and audio recordings of previous performances. In both a pre-concert artist talk and an interview, Hsieh described her early compositional process as being guided by the natural and musical physical gestures I use in performance:

Studying gestures tells composers a lot about what [the performers they are writing for] prefer to do, how you approach scenarios, what your musical sensibilities are. Following someone's gesture is a way of understanding your musical choices, the intention behind the musical decisions you made, as well as a way of guessing how you move and which compositional ideas could work for this performer. How you move is a way of showing what you're thinking. (Hsieh in Devenish, 2020)

Gesture thus became a source of compositional material in *Shifting Planes*, providing a rich platform for exploration of perception of musical gesture and sound. The use of


Table 2.1: *Activation techniques used on the sheet materials in* Shifting Planes

physical gesture as compositional material has been a cornerstone of percussion music for decades, cemented by the large body of instrumental theatre works composed for Trio Le Cercle in the 1970s and 1980s by composers including Mauricio Kagel, Georges Apherghis and Vinko Globokar. In recent decades, composers such as Jessie Marino, Natacha Diels and Jennifer Walshe have continued the trajectory of gesture as compositional material. Over time, the three primary methods for employing gesture in percussive music that have emerged are choreographic, tactile and virtual. Choreographic gesture does not directly contribute to sound making, in that it does not produce or capture sound, nor does it connect with any instrument other than the body. For example, Jessie Marino's trio *The Flower Episode* (n.d.) is a rhythmic group choreography of six hands, with most gestures not requiring the intentional production of sound. Similarly, Mark Applebaum's *Aphasia* (2010) and *Ceci n'est pas une balle* (2012) by Mathieu Benigno, Alexandre Esperet and Antoine Noyer require the performer to silently execute specific, precise gestures; however, in these works this is done in time with a pre-recorded tape. *Aphasia* is described by the composer as 'essentially a choreographed dance work' (Applebaum score 2010), with the gesture aligning with pre-produced sound, rather than generating live acoustic sound. In contrast, tactile percussive gesture is used to produce sound through connection with acoustic objects or instruments, or to capture sound from activated instruments using microphones, and these approaches are frequently used together. For example, in Juliana Hodkinson's *Lightness* (2015, rev. 2018), three percussionists strike matchsticks on ignition 'runways', rhythmically drag matchsticks across different grades of sandpaper, and precisely extinguish lit matches in small trays of water. The quiet, delicate, or proximal sounds that result from these gestures are captured by wrist microphones worn as described above. Finally, virtual gesture is common in the performance of MIDI instruments or software programs which rely on sensors to capture specific performative gestures in space to activate electronic sounds or to trigger samples, such as Jean Geoffroy and Thierry de Mey's Light Wall System.

Gødoy observes that gesture-based control of digital musical instruments (DMIs) can be seen as an evolution of musicians' relationships to traditional acoustic instruments (Gødoy, 2006), and the roots of these techniques can be connected to standardized instrumental techniques. Although all of the works described above are frequently performed by percussionists, any musician or actor could perform these works. Marino refers to the kind of performance skills required for her work as a 'virtuosity of the everyday' (Marino, 2020), using performance techniques that foreground everyday human gestures. In *Shifting Planes*, everyday gestures such as pouring from a bottle, brushing materials off surfaces, standing before a mirror, and folding a simple origami paper boat are woven together with percussive and microphone gestures. This kind of blended performance practice requires another type of virtuosity common in new music, which I have previously referred to as a 'new virtuosity' (Hope & Devenish, 2020).

#### **2.4 Instrumental infrastructure and notation**

In *Shifting Planes*, the multidimensional role of the sheet materials emerges through their performance. The sheet materials used in this work first appear as a theatrical set, within which a performance will take place. As the work progresses, the materials on stage are transformed (via their use in performance) from stage infrastructure to musical instrument and back again. The varied roles of the instrumental materials and their engagement in performance is notated using a hand-drawn action score that included a combination of text instructions, pictographic representations of physical gestures, and limited elements of conventional music notation. *Shifting Planes* is scored on graph paper, with each unit representing five seconds, and is relatively sparsely notated. An excerpt of this score is shown in Figure 2.3. Hsieh believes that the composer is never really in control of physical gesture, and that 'the more important the gesture, or the more featured it is in performance, the less it should be notated' (Hsieh in Devenish, 2020), as everyone has a unique body and unique ways of moving.

Each of the three sections of the work comprise notations addressing gestures and actions that the performer should execute. The first section is entirely text based, with written instructions such as 'slowly tip out', 'gently swirl the remaining grains' and 'listen and react to the texture heard in the fixed media'. The second section introduces line-based pictographic notation, which depicts gestures to be 'drawn' on the surface of various sheet materials using a range of indeterminate implements. Line-based gestures and their position on a surface or in space can be easily notated, either on standardized five-line manuscript or using graphic notation. It is particularly useful for music not bound by pulse or metre––that is, led by listening, movement and play. The act of drawing or tracing lines or outlines on surfaces or in space as a means to elicit sound appears in a range of ways in the percussive repertoire. For example, lines can be executed rhythmically using short rapidly drawn lines, as in my realization of Mark Applebaum's *Composition Machine No. 1* (2014) using a thick texta pen on amplified butcher's paper. Alternatively, they can be drawn texturally using large circles drawn using a slow, smooth motion, as in my realization of Cat Hope's *Tone Being* (2016) using cardboard tubes on a tam-tam (Devenish, 2018). In the former, the lines leave a trace on the butcher's paper, resulting in visible graphic drawings. In Hope's work, and in *Shifting Planes*, lines are drawn using various percussive implements and do not leave a visible trace.

Figure 2.3: *Excerpt of the score* Permeating Through the Pores of Shifting Planes *showing hybrid notation (Photo: ©A. Hui-Hsin Hsieh, 2019, used with permission). Dynamic graphic score created with Decibel Score application (2013)*

#### **2.5 Conclusion**

The increasing entanglement of materials as instruments, infrastructure, implements and increasing diversification of performance technique is effectively breaking down boundaries between musical practices: composition, performance, music, movement, installation. The design, layout and use of tangible and intangible instrumental materials is increasingly influenced by how performers might interact with these materials using musical, performative and everyday gestures and techniques and vice versa. *Permeating Through the Pores of Shifting Planes* was an initial exploration into the sonic and performative affordances of sheet materials, with particular consideration given to the relationship between the visual and the auditory in performance. The detailed investigations into gestures, a dispersed, installation-like setup, and individual performative movements at the early stages of development allowed almost all of the original ideas to be realized in performance. The use of my gestural, material and electronic instrumentarium was an effective way to 'grease the wheels' of a new collaboration, making use of all of the creative languages at our disposal. New music premieres always seem to take place with less time in the venue on the complete set-up than is ideal, and Hsieh's consideration of my performance style, interests in improvisation and graphic notations as well as her consideration of familiar percussive instruments and techniques in the early stages, resulted in an idiomatic work that came together quickly in the final stage and rehearsal period. Furthermore, this allowed deep and layered exploration of the questions that drove the work throughout the entire process.

The research questions surrounding visual-auditory relationships in performance, sheet materials as instruments and the plurality of material roles in performance, were explored on micro and macro levels over the course of *Shifting Planes*, using large gestures to expose small sounds, and small gestures that (with the assistance of amplification) conveyed large sounds. The notion of large-small and small-large gestural/sonic combinations and consideration of how the visual can affect the auditory have appeared in percussive practice and pedagogy for decades. For example, young orchestral players are encouraged to 'think loud, play soft' to achieve the perfect pianissimo triangle note, and to consider how much more effective a crescendo shake roll on the tambourine seems when the shake roll is accompanied by a gesture that raises the tambourine from low to high. Similar gestural concepts are embedded in all percussive techniques. When combined with the wrist microphones, these concepts can be explored further, particularly when working with feedback loops. When there is no attack to rebound from and the sound emerges from the aforementioned 'dance of agency', the performer can focus their gesture wholly on the care of the sound they are working with, and the ways in which gesture can guide listeners. In *Shifting Planes*, these concepts saw the transformation of an inanimate instrumental 'set' that a performer sounds, into an animated, vibrating instrumental infrastructure that a performer responds to, and this has formed the foundation of further collaborative artistic research.

#### **REFERENCES**


#### **MUSICAL REFERENCES**


## #3 PRE-GESTURE, GESTURE AND SOUND ON [NO] PIANO: MUSIC FROM SOMEWHERE

*...real space is neither its topos nor its instrument. This music is not here or there; as soon as it sounds, it is all-present and all-penetrating [...]. Therefore let us call this music, from a spatial perspective, a music from somewhere...*

—*(Craenen, 2014, p. 30)*

**21**

#### **3.1 Introduction**

Let this initial quote be the starting point to talk about *Music from Somewhere*, a piece written in 2017 (reviewed in 2019) by the composer Fran MM Cabeza de Vaca.<sup>1</sup> *Music from Somewhere*, for pianist's hands, lights and stereo audio tape, shows us a simple scenic display to amplify the previous moment to the sound emission of the piano, working with the silence as material that precedes the music that is yet to come, the one that is in this other place.

Bringing that specific moment to the forefront means that the movement and gesture become our main visual input, focusing on that virtual place where the performer is just before the sound materializes. The performer explores the different ways to connect the body to the instrument, linking it not only to the visible gestures, but also to the previous state where the invisible movement stays: the pre-gesture.

My background as a performer started with a classical piano education in a music *conservatoire*. Later on, I specialized in contemporary music, combining it with specific interest in free improvisation and sound art.<sup>2</sup> As the performer of this piece, I found that

<sup>1</sup>The video documentation of the work is accessible here: https://vimeo.com/449604575 2To learn more: https://www.haizelizarazu.com/en/

Figure 3.1: Music from Somewhere *by Fran MM Cabeza de Vaca (Video ©Marta Azparren, 2020)*

*Music from Somewhere* posed some questions from the very beginning of its conception: How do I relate myself to the piano when it is not physically present? How are my pianistic movements linked to the score when the resulting sound is not a consequence of the notation? All these lines converge in the self-consciousness of my moving body, on the proprioceptive process of the pianistic gestures that I have acquired through all my student years. *Music from Somewhere* challenges the pianist to rethink all the mechanic and automatized movements, the visible and the invisible, the different kinds of memories we develop, and creates a new space where sound, silence, gesture and pre-gesture are inter-connected.

#### **3.2 The invisible: score and pre-gesture**

There are many different kinds of scores, ranging from traditional staff notation systems to graphical scores, to video- or aural-scores. All of them are related to sound and show us various ways of representing it. When digging into other artistic disciplines, however, such as dance, theatre or performance, we can also find the score concept––not related to sound, but to movement and/or actions. Richard Schechner (2002) links the score to what he terms 'proto-performance' (p. 234). He describes the proto-performance (or 'proto-p') as something that precedes and/or gives rise to a performance: a starting point or, more commonly, a bunch of starting points (Schechner, 2002, p. 225).

The score in *Music from Somewhere* is twofold in this way: it is a video-score that is synced with the music (tape) that uses––mainly––two different kind of notations: gestural/positional (Figure 3.2) and traditional musical notation (Figure 3.3). The former clearly asks the performer for an imitation kind of response: do what you see. The latter,

Figure 3.2: *Gestural notation on* Music from Somewhere *(Score excerpt)*

Figure 3.3: *Traditional musical notation on* Music from Somewhere *(Score excerpt)*

apparently essentially traditional piano notation, asks the performer for a very different type of work: they need to find a gestural approach to the given piano score excerpts. Many of them belong to pieces that I have, at some point in my life, already studied and played; for example, works from the traditional, well-known piano repertoire, such as Beethoven's piano sonatas, Webern's op.27 Variations or Debussy's Etudes (among ´ others). All these different pieces are therefore linked to my own experience. To my own memory. Of course, we can't talk about memory as a single concept, as it has many different layers: aural, tactile, emotional, analytical, muscular. . . but all of these layers converge at one common point: the body. As Paul Craenen (2014) explains,

besides being a perceptual filter that determines what can be perceived and experienced, the body also functions as an active and reactive memory. Mimetic theories have already made us aware of the close relationship between action and perception, the ability to perceive, and muscle memory. (p. 265)

Furthermore, it is very important to mention that the score excerpts do not try to represent the resulting sound, as traditional scores do, but rather set a trigger to the action of approaching the written sound. Or, to put it another way, they explore the silence that exists just before the sound is produced (the music of the piece is in a great part a composition of the many different silences of many different piano performances). Hence, no actual piano performance gesture is desired in the first half of the piece. Taking all these considerations into account, performing the aforementioned score sketches lead me to re-visit them, remember them in all the possible ways: aurally, gesturally, emotionally. During the performance of the piece, no piano is there to respond with a resulting sound, or to feel its physical resistance. Therefore, the study process goes back to the real piano; not to practising the score itself, but to practising the feeling, the perception of playing those different piano scores. Very little time is given to change from one score to another, from one style to another, from one memory to another. Hence, the focus of the practice process relies on the awareness of your own body, on the previous state of playing the notated sound, on what I call the *pre-gesture*.

The pre-gesture is the invisible gesture that precedes the visible one. It is a starting point, an impulse. Much like the definition that Schechner makes about the protoperformance. It is very interesting to observe, too, how in this same line, the author gives a special emphasis to the concept of pre-expressivity (Schechner, 2002, p. 226), a term related to the theatre author and director Eugenio Barba (previously developed by his teacher, Jerzy Grotowski).<sup>3</sup> It all makes reference to this previous body state––body presence––that the performer needs to train in order to actually perform the play, text, work or dance.

The same way that a performer/actor trains the presence of their own body, the tensions preceding the actions (the in-tension) in order to acquire an incorporated knowledge through the body, the performer/pianist in *Music from Somewhere* needs to dig until the pre-gesture level, the preceding action, the previous mind-body state, in order to perform the pianistic gestures without either a piano or a resulting sound. Figure 3.4 shows how I recorded myself playing the different score excerpts; I used this as a helping tool to memorize the visual gesture as seen from the outside (added to the proprioceptive feeling of the pre-gesture).

Figure 3.4: Music from Somewhere: *screenshots of the video documentation during the study process (Photo: Lizarazu, 2020, CC BY)*

All these concepts (proto-performance, pre-gesture, pre-expressivity) circle around the same point. On the subject of this previous state of the body Paulo de Assis (2018) provides a very interesting analysis about Simondon's key concept of *transduction* and how the body could be understood as a transducer in music-making (pp. 137–157). Terms such as *structural germ* or *micro-haecceity* (related to the Simondonian notion of *individuation*) illuminate these energy and state changes of the performing body and give us a clue about how we might apply it to the musical realm. In his words,

<sup>3</sup>To learn more, see Barba, Eugenio & Savarese, Nicola. *A Dictionary of Theater Anthropology: the secret art of the performer*. Routledge, New York (1991)

Micro-haecceities are high energy-loaded and high-speed-moving singularities that carry a force of potential from one position to the next. They make up the visible or audible part of artistic transductive processes. In their functioning as radical becoming they never appear as stable *beings*, remaining an impulse of virtuality from one actualisation to the next. If one thinks, or does, or experiences artistic performances with these operations in mind, the Deleuzian notion of *capture of forces* becomes more graspable than ever: the virtual becomes actual in order to be instantly dissolved into the virtual again. The pianist playing Schumann perfectly exemplifies such a capture: he or she is not merely reproducing a stratified, pre-existing entity, but operating a capture of forces (from the virtual) that produces a new individuation (actual) as a highly intensive becoming, which immediately—as soon as it is generated—points forward to other virtual pre- and after-individualities. (de Assis, 2018, p. 149)

As we see, the continuous contingencies that happen within a performance (*virtualities*) are a key concept in understanding how our body moves and behaves in the music-making process. The author refers to the *structural germ* as a potentiality: it is a structural consecutive potential; that is, it carries some sort of information, which sets the basic conditions for an event to happen (de Assis, 2018, p. 147). It is clear now how all the mentioned terms and theories help to facilitate an understanding of the body and its movements not only in the performance moment, but also in the previous non-visible states. *Music from Somewhere* explores all these pianistic pre-gestures as the main musical and working material. It challenges us to focus and to be conscious of them not only on a conceptual level, but also as an interaction process between the body and the instrument at the practical level.

#### **3.3 The visible: gesture and space**

At first sight, when watching *Music from Somewhere*, one can see that two main spaces are created and shown: the public space in front of the white panel, and the semiprivate space in the background, where a dim light and a distant music carries us. The public space is presented as the performing stage, where the hands, fingers and arms are shown. A kind of concert stage. The semiprivate space is presented as an intimate study place (home, studio), where the performer usually finds themself alone (even though the audience is able to 'peep' them through the panel hole). The proposition, here, is one of outside and inside; what the audience does or does not see. And between those two places, once again, a common agent: the performing body.

These two spaces have different purposes. On the one hand, it serves as a visual and clear separation for and between the audience and the performer, who hides behind the white panel. On the other hand, it physically creates the different places where sound exists in a performance, as we will elaborate in the following lines (the moment just before the sound, when it sounds, after the sound is heard/played). This second partition leads us to talk about the space in aural terms: the sound-space.

The first case scenario is the sound-space that happens before the note is played, before the hand actually touches the keyboard. This starting point (which itself consists of many starting points) is linked with with the aforementioned pre-gesture, as we have seen. But that pre-gesture is not just a physical proprioceptive feeling, it contains all of the music that is yet to come. To be at the pre-gesture level, the performer needs to be also at the multiple levels and 'places' that the piece has been before, since the first reading until the performance moment. For Mine Dogantan-Dack (2011), 'the unity of the initiatory ˘ gesture and tone produced is also part of the listening experience, although the listener is not ahead of the music physically in the same way as the performer is' (p. 259). The audience doesn't just listen to the sound of the piano, or the various attacks of the pianist. The audience sees how the performer moves, how they approach every musical event, and also, how they enter the stage or enact in certain, idiosyncratic ways, the music that we're about to experience. As Dogantan-Dack (2011) discerningly points out, 'in fact, if ˘ we really are precise, it is not the attack that produces the sound, but the gesture bringing about the attack' (p. 259).

It should now be clear for the reader how paramount this previous state of the performing body is––not only for the performer themself, but also for the audience (that perceives it from a third person perspective) and for the resulting sound that comes after. As Dogantan-Dack (2011) explains: ˘

(. . . ) the performer starts to experience the tone much earlier not only mentally, but also physically, at the beginning of the fixating gesture, before the hammer contacts the string and the tone actually starts sounding. The kinaesthetic sensations that accompany the gesture result from the adjustment in muscular tonus that the pianist makes to prepare the impact, and this adjustment in turn is guided by an aural image of the desired tone, the goal of the gestural movement. Pianists know – must know – what kind of tone will ensure that their touch is able to produce the intended tone at all. (p. 258)

The sound-space concept has been examined for many decades now, from the architectural perspective as well as from both the listener's and a music-making perspective. Often sound is introduced in this discourse as an isolated element, with the argument being that sound creates the space itself, like a sphere without fixed boundaries, space made by the thing itself, not space containing a thing (LaBelle 2010, xxi-xxii). Though this argument may be true in context, we might also think about the impact of the musicmaking body in the sound-space creation. It is important to think about spatiality not only in physical, three-dimensional terms, but also as a musical metaphor to understand the different moments and events that occur when the sounding body intervenes. Paul Craenen (2014) provides an extensive analysis on this subject, in which he traces music as a spatial phenomenon through different categories, such as: a) space *surrounding* the music; b) space *for* the music; c) space *of* the music (pp. 20–25).

The first category refers mainly to the characteristics that make musical activity recognizable in a society, meaning the role that music plays in our social, cultural, political and economic context. Here, the human body is found in a multiple layer scenario where audience, politicians, promoters, programmers, musicians all interact in creating this surrounding of the music as a metaphorical category (Craenen, 2014, pp. 20–21). The second sound-space is related to the instrumental space. This is where we find the music halls, venues, stages and all the specifics needed for the music performance to happen. It is also where the performing body first appears as a main character. Lastly, the space of music itself defines what we understand as the musical playing field. In the words of the author,

. . . the musical playing field (. . . ) is a space that is constantly shifting, a space that behaves dynamically. It is an emerging space that unfolds as the music progresses. The musical playing field is a cultural and audio-motor space in which patterns of musical expectation, stylistic characteristics, and idiomatic sonorities encourage selection and variation. (Craenen, 2014, p. 23)

The space *of* music changes from piece to piece, from performer to performer, from instrument to instrument, even if the space *for* the music remains the same in all those changes. It is in this musical playing field where the listening and perceiving experiences carry us to other, more intimate sound-space metaphors: the music from *here*, music from *there* and music from *somewhere* (Craenen, 2014, pp. 26–30). Focusing on this last metaphor (which gives its name to and also partly motivated the piece we are analysing in this paper), this *somewhere* attempts to name that space where the music carries us, we listeners and performers, when sounding,. This place is formed not only by the sound, but also by the movement, the performing body that materializes that sound.

In *Music from Somewhere*, all these sound-space metaphors are presented through the gesture of the performer. First, the sound is being––wanting to be––materialized, present: *here*. As we have already seen, this is managed through the inner perception of the pre-gesture level. Secondly, once the performer's finger literally touches the table–– the [no] piano––another sound-space is presented, the one where the sound is played and long gone at the same time: *there*. In that place, the visible pianistic gestures are more recognizable than ever. The attack, the intensity, the tension, the release of the notes is being shown as a consequence of the proprioceptive memory of the performer and their embodiment. The travel through these two places lead us, the performer and the audience, to this other place: *somewhere*. From a spatial perspective, a music from *somewhere* is conceived as a fluid, phantasmal space experienced kinetically (Craenen, 2014, p. 30).

The awareness of the different sound-spaces in relation to the present body (performers and audience) enables us to analyse the performing body, its gestures and movements as an independent and well established concept for theorization. It is in fact the copresence of the bodies that makes the scenic realization possible and the one that set the ground for the first performative turns in the arts (Fischer-Lichte, 2004, p. 65). The importance of the gesture in relation to a musical instrument is a key component in the learning process of any musical piece at any musical level.

#### **3.4 Conclusion**

*Music from Somewhere* is a musical piece that uses the gesture and the performing body as musical material. It paradoxically explores the relation between the pianist and their instrument by eliminating the latter. This sets the focus on the movements that a pianist does not only when performing, but also when practising. The two main points that have been explored through this piece are the self-awareness of the pre-gesture and the creation of the sound-space through the gestures of the performing body.

Pre-gesture has been presented as an inner movement of the performer, a previous state of the performing body where all the music, information and possibilities live before the sound actually materializes. The concept is here connected to other artistic fields, such as theatre and performance, where the body has been a more extensive subject of study than in the musical field (Schechner, 2002; Barba & Savarese, 1991; Lecoq, 2003). I argue about the importance of focusing on this previous state to understand the embodiment of pianistic gestures and the cognitive processes enacted in the study and practice of a musical piece or instrument.

In addition, pre-gesture also serves as a practical tool to perform and elaborate that metaphorical place where the music exists prior to its sounding. The actual materialization of the sound leads us, then, to this other sound-space, where sound exists and is gone at the same time, a fleeting moment where sound is barely localized. And in all of those sound-spaces, the performing body acts as an interface, a filter, a bridge to carry us to that *somewhere*, where all the music, the audience, the performers exist in a *non-spatial* space realm.

#### Acknowledgments

Special thanks to Fran MM Cabeza de Vaca.4

#### **REFERENCES**


## #4 EVOLVING MUSICAL EXPECTATIONS: MUTUAL CORRELATION BETWEEN A HUMAN MUSICIAN AND AN AI MUSICAL INSTRUMENT

#### **4.1 Introduction**

Artificial intelligence is one of the most active areas of research in the music technology community. There are significant efforts to further our understanding of artificial intelligence's potential applications in music on behalf of many practitioners, artists, musicians, computer scientists, with their own particular approaches, goals and their contributions to the development of AI research in music. One of the main areas of focus has increasingly been on expanding the current use of artificial intelligence to the creative practice of musicians, making new technologies available and accessible to instrument builders, musicians and composers (Eigenfeldt & Kapur, 2008; Tatar & Pasquier, 2019). Currently, there is an ever-growing demand for tools and techniques that allow for the creation of autonomous devices and processes (Tahiroglu, Kastemaa & Koli, 2020). Following the demands of musicians, a considerable amount of research has been devoted to various approaches to manipulating and transforming musical instruments into musical agencies through the autonomous acts of the musical instrument in a collaborative music action between musician and the musical instrument (Tanaka, 2006; Karlsen, 2011). Such musical instruments can incorporate features that allow them to act in mutual cooperation with human musicians in the process of composing and performing music. In this joint activity, the development of algorithms that can contribute to compositional structures as well as composition methods is a promising path towards the integration of more advanced computational technology with the creative practice of musicians.

In this article, I intend to reflect on the potential use of artificial intelligence technologies in instrument building, and subsequently to address the specific challenges and

**29**

*https:/doi.org/10.34727/2022/isbn.978-3-85448-047-1 4.* This work is licensed under the Creative Commons Attribution-ShareAlike License 4.0 International (CC BY-SA 4.0).

opportunities that arise through these technologies. I focus on emerging musical expectations and musical demands in a practical research implementation of our AI-terity autonomous musical instrument. Relating advanced technology to music and performance is not a new phenomenon and has been discussed in more depth (Collins, 2007; McPherson & Tahiroglu, 2020; Magnusson, 2019; Tahiroglu, 2021), at the same time, ˘ not much effort has been spent in examining the following questions: How do artificial intelligence autonomous algorithms and human musicians contribute to new musical expectations? What can be realised through the equal contribution of AI instrument and human musician in a music performance?

Figure 4.1: *AI-terity instrument (Photo: Koray Tahiroglu, 2020, CC BY) ˘*

#### **4.2 AI-terity and the composition Uncertainty Etude #2**

In our Sound and Physical Interaction (SOPI) research group at the Aalto University School of ARTS we have been building, developing and performing with the AI-terity musical instrument (Figure 4.1), which is a non-rigid instrument that comprises a deep learning model, GANSpaceSynth,1 for generating audio samples for real-time audio synthesis (Tahiroglu, Miranda & Koli, 2020). Physical deformability becomes the affor-

1GANSpaceSynth is a hybrid generative adversarial network (GAN) architecture that we developed in our SOPI research group. It applies the computational features of the GANSpace method (Hark ¨ onen et al. 2020) ¨ dances of the instruments for handheld physical actions to be applied. The instrument uses an abstract form of an interface that is responsive to manipulation through bending or twisting, controlling parameter changes in granular synthesis.

We developed the GANSpaceSynth, specifically to provide more control over the spatial exploration of the audio features that are distributed in the latent space. This unique access to the GAN latent space gives musicians the ability to interact with the higher order structures to generate new audio samples. The advantage of this approach is the control of the directions for moving one point to another in spatial dimensions of the latent space. We call this point, the synthesis centre point (SCP).

Following the unique features of the instrument's deep learning model, we developed the instrument's autonomous features further to bring in alternative musical responses in music performance (Tahiroglu, Miranda & Koli, 2021). In this way we could approach ˘ the performance of a music composition as an entity providing an independent variable that could affect the musical context by changing the decision with a non-arbitrary way of generating new sounds. To do that, we built in autonomous features to change the direction of the SCP in the latent space. GANSpaceSynth generates audio samples based on the SCPs, and these points determine the audio characteristics of the samples. The musician can navigate through the latent space by interacting with different parts of the interface, and GANSpaceSynth receives each of the SCPs as input to generate a corresponding audio sample. The idea of the autonomous behaviour is to monitor the musician's state of performing with the instrument and change that confident state of performing to an intermittent state of performing. The autonomous behaviour allows for alternative sound-producing expressions to appear; these are then layered in the changing audio features of the real-time granular synthesis. In this way, the autonomous nature of the instrument can be seen as an autonomous behaviour that aims at keeping the musician in an active and uncertain state of exploration, which allows massive flexibility and instantaneous exploration of an instrument's playability.

We wrote a composition for the AI-terity instrument, aiming to idiomatically reflect the autonomous features of the instrument. The work is based on the idea of *uncertainty*, where the instrument moves the SCP across latent space, aiming to find a new target point, but never stays in one particular point long enough to allow the musician to stay in a comfortable and certain state of performing. The composition brings up some confusion and surprise. Figure 4.2 shows the studio recording of the piece.2 I should clarify precisely what is meant by 'uncertainty' in this case. It doesn't mean that the instrument chooses any 'random' points in the latent space; the jumps in between the SCP that the GANSpaceSynth uses to generate audio samples for the granular synthesis are not random. Instead, the autonomous features are designed to move in the opposite directions on the basis of the latent space centre and gradually introduce new audio samples in a smooth transition that allows the musician to explore the changing timbre of the audio samples during each jump. Through this process the music is composed––and it is unpredictable. You can hear it changing all the time. For a while follows the course of playing in an original set of generated audio samples, but the music is composed for each

2The studio recording is available at https://vimeo.com/514201580

on the audio synthesis features of the GANSynth model (Engel et. al. 2019) for organising the latent space using a dimensionality reduction for real-time audio synthesis. The open source code of the GANSpaceSynth is available at https://github.com/SopiMlab/GANSpaceSynth/.

jump with the intention of bringing in a new set of audio samples. It is possible to create a certain combination of sounds, each combination being a result of the SCP. What the performance of the piece attempts to do is to create unpredictable yet original musical expectations for the musician and the audience.

Figure 4.2: *Studio recording of the composition* Uncertainty Etude #2 *(Photo: Koray Tahiroglu, 2019, CC BY) ˘*

#### **4.3 Achieving musical expectations**

Titon (2009) describes a model for our interaction with music that presents musical performance as the process of creating the active experience of musical works. Following this model, we can see that music performance makes both those musical and aesthetic expectations appear that are already present before the act of the performance, as well as those that are not. In the context of music with an AI powered autonomous instrument, musical expectations appear in relation to the mutual connections between the instrument and the human musician. The performance of the piece *Uncertainty Etude #2* explores how the artificial intelligence instrument can serve as a musical partner, so that the human musician and the instrument can communicate to evolve musical expectations.

Musical expectations have been discussed in the context of melodic (Margulis, 2005), tonal (Bharucha, 1994), sound and meaning (Clarke & Cook, 2005), biologically learned (Huron and Margulis, 2010), memory and cultural (Curtis and Bharucha, 2009) patterns involved in music that may give rise to expectations and to 'affective consequences of expectations' (Huron & Margulis, 2010) for listeners or musicians. Huron and Margulis (2010) mention that 'familiar experience' occurs in the nature of musical expectation even between songs devoid of any structural musical relationships. It is an indication that familiarity could play a role in helping listeners to learn how to anticipate future songs, an ability which is often attributed rather to the listener's 'general knowledge' of the music. This expectation, however, is not a direct knowledge, and only shows to the listener the potential that is inherent in the music. By contrast, expectations have a proprietary quality, in that they can be learned from the specific events themselves. Regarding the indications for 'anticipating the future sounds', the question then arises, in what ways can we expect a particular musical expectation to be achieved in a correlation of sounds that even the musician has not heard before performing the music? For the composition *Uncertainty Etude #2*, we trained the GANSpaceSynth model with the audio dataset that has the overall textures of musical sounds, essentially inharmonic and atonal features with electronic ambience patterns. I have provided my own dataset for training the GANSpaceSynth checkpoint. The resulting checkpoint model in the composition tends to generate rather garbled approximations of the original dataset with smeared transients of unfamiliar sounds that do not follow the musical features of tonality or the context of melody as commonly discussed in the forms of musical expectations.

In the performance of the *Uncertainty Etude #2*, it might be argued that the ability to form and achieve expectations about unfamiliar sounds could still be more innate, while at the same time to some extent being dependent upon musical experience. This leads us to discuss whether musical expectation could be formed and developed on the basis of some other behavioural responses, such as intersubjective experience (Fuchs & De Jaegher, 2009), that is, from hearing other listeners. Perhaps it could be learned; indeed, as the structure of the musical language in *Uncertainty Etude #2* changes, some of the features required to form expectations are then learned through the performance of the piece. It might not be too much to argue that achieving musical expectations about the performance of the *Uncertainty Etude #2* will occur through direct acquaintance with the music. The mutual correlation between the human musician and the autonomous AIterity musical instrument will evolve unfamiliar and surprising musical expectations that the listeners will experience.

Bharucha (1990) questions whether the listeners or musicians give up the element of surprise in musical expectation when they prioritize what they have already known. I am not sure if there is any particular answer to that question, but it might still be worth mentioning that familiarity has the potential to provide an emotional boost to music listeners' enjoyment, which might counteract the effect of surprise in music enjoyment. If this is true, then, in contrast to the music enjoyment of listeners of unfamiliar genres, are the listeners of familiar genres more likely to experience surprise as an unexpected *loss*?

It could be further questioned whether unfamiliar genre-specific audiences have a greater expectation of novelty than listeners of familiar genres. I think this may be more plausible than it would be in the case of other kinds of anticipation and expectation; but even if the question is theoretically open, it would be a tough argument for the listener of a familiar genre to come to the conclusion that any song is more interesting than, say, a song that fits the familiar genre. In the performance of the composition *Uncertainty Etude #2*, there is a unique set of properties for musical expectations, which is the result of the mutual cooperation between the human musician and the autonomous instrument. This set of properties are part of the unfamiliar and surprise nature of the musical performance itself.

#### **4.4 Musical expectations as a framework for composition**

The shift of music performance from a process with a master performer to an activity performed by a human musician and an autonomous instrument can also be considered as a particular social expectation that can be used as a framework for a musical composition (Tahiroglu, 2021). In the performance of the piece ˘ *Uncertainty Etude #2* the human musician and the instrument become part of a collective, part of a 'performance ensemble' (Latour, 2005). There is a 'particularity' in this collective in its own way. The particularity, here, can be expressed in its musical and conceptual form, which can be derived from the composition. As part of the performance ensemble, the human musician and the instrument become a 'system of instruments' with their own musical demands, and yet also a 'system of musical compositions'.

Such performance ensemble opens a space for ways of music-making in which unfamiliar and surprise musical expectations become a part of compositional structures. We can consider this process as a compositional framework that offers insight into the creation of compositions that are fully or partly autonomous, or autonomous in an unusual way. We can speak of unfamiliar and surprise musical expectations with an immediacy that the performance would entail. There is a potential for musical exploration that arises from particular musical expectations, in which the musical structure and its material have to do with something that autonomously evolves with unexpected, surprise and unpredictable musical events.

#### **4.5 Conclusion**

The intention of this chapter was to present my reflections as an artist and musician performing with an artificial intelligence musical instrument. The question of the unusual musical expectations and further musical demands was also discussed. My main intention has been to introduce the unfamiliar musical expectations that appear through the performance of the composition *Uncertainty Etude #2*. In this composition the integration of an artificial intelligence method with the creative practice of a musician contributes to the autonomous structure of the piece. The performance of such a composition involves further challenges for the musical expectations and musical demands––both for the human musician and the audience. I intended to discuss in what particular way musical expectations and musical demands become present through an autonomous behaviour that was built to enable the appearance of alternative sound-producing expressions, which are layered in the changing audio features of the generated audio samples. I hope this article will contribute to an ongoing discourse about new creative technologies, and especially to the debate around the use of AI technology in music practice and a new way of thinking about composing and performing with musical instruments.

#### Acknowledgement

This work was supported by the Academy of Finland (project 316549).

#### **REFERENCES**


**PART II**

## EMBODIED OBJECT

## #5 TIMESCALES FOR SOUND-MOTION OBJECTS

#### **5.1 Introduction**

There can be no doubt that Pierre Schaeffer was one of the most influential figures in 20th century European music, both as a composer of electroacoustic music and as the developer of a new and extensive theory of music. We are much in debt to his artistic and theoretical achievements, and in spite of more recent advances in music technology, we can still find his contributions highly relevant for music making. In particular, Schaeffer's idea of *sound objects* as the basis for musical creation, perception, and music theory (Schaeffer 2017; Godøy 2021), was a remarkable change of paradigm in Western musical thought, and this sound object paradigm is still highly relevant for music theory, as will be highlighted in this chapter.

In brief, the sound object can be defined as a holistic mental image of a fragment of sound, typically in the approximately 0.5 to 5 seconds duration range, and the sound may be of any origin, vocal or instrumental, electronic or environmental, and from any musical culture. The focus on sound objects as the basis for both creation and theory, this chapter will argue, has the following advantages:


The main point of this chapter is that the object timescale has a privileged role in music, both in terms of intrinsic sound features and the associated body motion fea-

**39**

tures. The main source for this view of the object timescale is Schaeffer's monumental *Traite des objets musicaux ´* (Schaeffer, 1966), now also available in English and referred to as (Schaeffer, 2017). Other sources include Michel Chion's *Guide des objets sonores* (Chion 1983), an excellent overview (endorsed by Schaeffer himself) of the most important topics of Schaeffer's Traite (also available in English as Chion, 2009), and Schaeffer ´ and colleagues' *Solfege des objets sonores `* (Schaeffer et al., 1998), a collection of three CDs with sound examples and music theory topics narrated by Schaeffer. This work is very useful because of the sound examples, and can make Schaeffer's ideas accessible for a broader audience. For practical reasons, references to these sound files in this chapter will be given as follows: '*Solfege `* , CD#, track#'.

#### **5.2 Schaeffer's object focus**

The idea of sound objects as the basis for a music theory emerged from practical work with concrete music (sometimes referred to with the French label, *musique concrete `* ), when composers, before the advent of the tape recorder, used looped sound fragment recordings, i.e. *closed groove* (*sillon ferme´* in French) on phonograph discs, to combine different sounds in compositions. As the composers listened to innumerable repetitions of such sound fragments, they discovered that their attention shifted from the immediate and everyday signification of the sound to the more qualitative features of the sound, i.e. to the features that came to be the content of the so-called *typology and morphology of sound objects*, the elaborate scheme for feature classification in Schaeffer's theory. Later, Schaeffer and his collaborators came to realize that their modus operandi during the early years of the *musique concrete `* had involved a phenomenological shift in focus, known as *epoche´* in the writings of Husserl (Husserl, 1982). In retrospect, Schaeffer called this 'doing phenomenology without realizing it' (Schaeffer, 2017, p. 206). We can now see that Schaeffer's music theory has several other affinities with phenomenological philosophy, such as the procedure by sketches and the object-centred view of perception (see Schaeffer, 2017, p. 210).

The shift of focus from everyday significations (e.g. the squeaking door signalling that someone is coming) to the more qualitative features (e.g. the overall dynamic envelope and the upward glissando of the squeak sound), was called *reduced listening*, and it should be emphasized that this was a method for exploring sound features. This reduced listening was related to the idea of *acousmatic music*, 'acousmatic' here denoting music emanating from loudspeakers with no other visible sound source. Furthermore, a close reading of Schaeffer will show that any sound object is ontologically composite, i.e. it will usually have several different features and significations in parallel, but the overall dynamic and spectral features form the basis for the *typology* of sound objects, enabling an initial and coarse classification of sound objects based on their dynamic and spectral shapes.

The research method of Schaeffer was that of starting out with seemingly na¨ıve questions of what we are hearing, with a kind of Socratic approach of top-down scrutiny of what is in our minds. In the words of Schaeffer, this could be summarized as 'exploring the listening consciousness' (Schaeffer, 2017, p. 109), and with questions of the overall features of the sound objects like: What is the dynamic shape of the sound object? What is the mass (subjective sense of spectral shape and/or pitch) of the sound object? Is the sound object stable or does it fluctuate? Is it sustained or more impulsive?

A crucial point here is that the sound object is not a static entity, but first of all a mental image of a fragment of unfolding sound. Much effort in the writings of Schaeffer is devoted to what the sound object is *not*, and emphasizing that the sound object is a mental image resulting from attentive perception across multiple listening experiences, as well as being ontologically composite with a multitude of features in parallel.

Another vital point here is that a sound object may have a non-linear relationship with its acoustic basis, i.e. there may be a relationship of so-called *anamorphosis*, or warping, between the acoustic signal and the mental image. This often non-linear relationship between the acoustic features and the subjective percept is due to some perceptual-cognitive factors, primarily the following:


The point with anamorphosis is that there may not be a one-to-one relationship between the acoustic features and our subjective perceptions. In Schaeffer's method, this means taking our perceptions as primordial and not regarding perception as flawed, but instead exploring the correlations between acoustics and perception, correlations that also take this anamorphosis into account.

What is crucial here is the internal coherence of the sound object in the sense of temporal bi-directionality, i.e. that present is tainted by past and past is tainted by present (as well as by future expectations), as was concretely documented by Schaeffer with the so-called *cut bell* experience, which showed how removing the attack segment could totally alter the sound of a bell. This past-present-future tainting is yet another reason why the *closed groove* is such a powerful tool for research, as it documents the workings of context at the sound object timescale. From this primacy of the subjective perception, the next step was to study the dynamic shapes in the so-called *typology*, extended also to pitch and spectrum-related shapes, as well as later to various internal features of the sound object in the so-called *morphology*, and the combination of these in the *typology and morphology* summary diagram (Schaeffer, 2017, pp. 464–467).

#### **5.3 General object cognition**

The focus on objects in perception and cognition is also found in other domains of thought (see e.g. Shinn-Cunningham, 2008; Starrfelt, Petersen, & Vangkilde, 2013; De Freitas, Liverence, & Scholl, 2014 for some interesting cases), and it could be useful to have a quick overview of some generic ideas on object-centered perception and cognition to see how they may contribute to the idea of sound objects.

Two strands of thought in the late 19th and early 20th century stand out: Gestalt theory, and the previously mentioned phenomenological philosophy of Husserl. From the pioneers of Gestalt theory such as von Ehrenfels, Stumpf, Wertheimer, Koffka, and Kohler (just to mention the most prominent ones) up to Bregman in the late 20th century ¨ (Bregman, 1990), the idea of holistic perception and cognition has been prominent, and principles such as *belonging* and *exclusive allocation* have contributed much to understanding what we call object cognition. Another relevant case of Gestalt theory is that of motor control (Klapp, Nelson, & Jagacinski, 1998; Klapp & Jagacinski 2011), suggesting that body motion can be understood as consisting of pre-planned chunks similar to gestalts in perception. It could also be suggested that motion chunks contribute to sound object formation, as is the point of so-called *motormimetic cognition* (Godøy, 2003) and the idea of extending Schaeffer's typology categories to body motion (Godøy, 2006).

In parallel with early Gestalt theory, the phenomenological philosophy of Husserl provided important contributions to the epistemological reasoning about objects in our experiences. According to Husserl, we need to step out of the continuous stream of sensations in order to constitute meaning, and this happens by means of a series of socalled *now-points*, i.e. points in time where we interrupt streams and lump together past, present, and expected future sensations into somehow meaningful chunks (Husserl 1991; Godøy 2009).

In an experimental vein, we have seen research (since the pioneering work of Miller (1956) on chunking) on the workings and effects of different kinds of chunking in human behaviour (Gobet et al., 2016), and more neurocognitive views on chunking and sensations of presence can be found e.g. in Poppel (1997), in Varela (1999), and in Wittmann ¨ & Poppel (1999). Additionally, we have seen work on perception and cognition of audi- ¨ tory objects by holistic integration of sensory data (Bizley & Cohen, 2013); lastly, there is an extensive theory of objects and shapes as fundamental elements in human reasoning to be found in *morphodynamical theory* (Thom, 1983; Petitot, 1985; Petitot, 1990; Godøy, 1997).

#### **5.4 Sound object features**

What constitutes a sound-motion object is first of all a sense of energy shape, of starting and ending within a timespan of approximately 0.5 to 5 seconds (sometimes longer, but that is more exceptional). This overall energy shape, or envelope, is one of the main features of Schaeffer's typology, the typology being a first and coarse sorting of soundmotion objects. The main categories of these energy envelopes, called *facture* in French, a term designating the way they are made, are as follows:


But also a first and coarse sorting of sound content is included in the typology with the three categories of so-called *mass*, denoting the frequency domain (and not just perceivable pitch) as follows:


These mass and facture types are combined in a 3 x 3 matrix, and can be applied to most types of sounds within the typical duration range for sound objects, and several examples of these categories can be heard in the *Solfege `* , CD3, tracks 19–42.

There may be so-called *phase transitions* between these typological categories, e.g. an impulsive sound may turn into a sustained sound if extended beyond some duration threshold, or an iterative sound may turn into a series of impulsive sounds if the sound is slowed down, etc.; in short, there are categorical thresholds here due to duration and density of events. And importantly, these musically meaningful categorical thresholds are all related to sensations of body motion, to both motor control and bodily effort. The overall energy envelopes of sound objects are usually the most perceptually salient, capable of triggering sound-accompanying body motion in listeners, such as in various cases of so-called entrainment (Clayton et al., 2013).

There are in addition a number of internal features of the sound objects in what is called the *morphology*, organized in a system of main feature dimensions, each in turn with sub-feature dimensions, and sometimes also with sub-sub-feature dimensions. These dimensions concern the frequency domain features and their various sub-features, and some dynamic features and their sub-features. The most important are those concerned with textures:


There are also other morphology dimensions such as *mass* (overall spectral content), *dynamics* (overall loudness), *harmonic timbre* (spectral distribution), *melodic profile* (pitch-related shapes), and *profile of mass* (spectral shapes); however, the typology contains the most prominent features for sound-motion objects, because it includes shapes at the sound-motion object timescale.

Large-scale forms may also be conceived of as objects, as has often been the case in Western music theory, but Schaeffer was clear that his focus was on the materials of the sound objects and not on large-scale works (Schaeffer, 2017, p. 17). Schaeffer introduced the concept of the *suitable object*: neither too short nor too long, and in practice (as evident in the examples of the *Solfege `* ) typically in the 0.5 to 5 seconds duration range, as well as some other criteria of information density, i.e. neither too complex, nor too simple, to keep the attention of the listener.

With too-long fragments, it would not be possible to focus on salient features, because they would change. With too-short fragments, perceptually salient features would not exist (would not have time to become manifest). In exceptional cases, sound objects in the *Solfege `* last for up to 30 seconds. But the main duration criterion is that of content, i.e. that the sound object manifests a clearly perceivable salient shape. This means that the typological categories mentioned above become the main criteria of timescales, as can be heard in *Solfege `* , CD3, track 19–22, as well as examples from various sources, instrumental and electronic, in the *Solfege `* , CD3, track 22–42. To make this point about suitable objects, we are in *Solfege `* , CD3, track 42–59 presented with other examples of sound objects deemed not to comply with suitable objects criteria by being either too long, too short, too dense, having too much redundancy, being too unpredictable, or too chaotic.

#### **5.5 Continuity vs. discontinuity**

What the preceding object principles boil down to is the relationship between continuity and discontinuity in musical experience. The question is: To what extent does our organism work by continuous or discontinuous processes and decisions? For more general points of view on this, see e.g. Miller (1982), Sergent and Dehaene (2004), Spivey (2008), and Reybrouk (2021); but in motor control, continuity vs. discontinuity has been debated for more than a century, and has engendered various models of how we plan, trigger, and control body motion in different contexts (Elliott, Helsen, & Chua, 2001).

The conundrum of continuity and discontinuity in motor control can in a sense be bypassed by seeing how triggering and control may be discontinuous, while the results may yet be continuous, i.e. the motion trigger may happen at discrete points in time, but the resultant motion may be extended over longer stretches of time. Discontinuity in motor control is often believed to be based on constraints, i.e. that our organism is too slow to enable continuous control, and that continuous motion is an emergent feature of our organism's implementation of discrete control impulses, as suggested by the following:


An interesting development in motor control here is the theory of so-called *intermittent control*. This theory suggests that human body motion may be controlled discontinuously, i.e. in a point-by-point manner called *serial ballistic* control (Loram et al., 2011). The reason for this serial ballistic control scheme is again that our organism seems to be too slow to enable totally continuous control, and needs to work by anticipatory control, i.e. in a point-by-point manner by so-called *open loop* control. However, the perception of the output motion may be more continuous, hence the idea of 'observe continuously, act intermittently' (Loram et al., 2011, p. 317). Intermittent control is an ongoing research topic, but we may hypothesize that it could be a plausible way to reconcile continuity and discontinuity in sound-producing body motion as well, and hence also in our mental images of sound-motion objects.

#### **5.6 Concluding remarks**

The mentioned categories and criteria of a combined sound-motion object theory could make us aware of, and give names to, various perceptually salient features of the music. This could provide us with both analytic and generative tools, particularly useful for enhancing our grasp of the different sound-motion object types derived from Schaeffer's work (see Godøy, 2018 for details), such as:


Beyond the suitable objects mentioned above, there are also various other objects that can be named, and there are very many possibilities of object combinations and concatenations into more extended collage compositions. All this can have useful applications in musical analysis as well as in creative tasks, such as:


Throughout these processes, thinking sound-motion objects and various typological and morphological features, could be a systematic approach to handling otherwise ephemeral material.

Hopefully, sound-motion objects, combining the remarkable insights of Schaeffer and his collaborators from more than half a century ago with current research on musicrelated body motion, could be a vehicle for further exploration of the holistic nature of musical experience. This could be expressed as a theory of *musical quanta* (Godøy, 2013), denoting a holistic object-centred approach combining the overall sound and motion features in a way that also reconciles continuity and discontinuity in musical experience.

Notably, such an object-centred approach would not exclude more macro-level approaches as can be seen, for instance, in projects of music information retrieval. Exploring features of larger collections of sound-motion objects or of more extended, macrolevel works of music, e.g. *spectral centroid, spectral flux, harmonicity,* etc. using available software e.g. the *MIRtoolbox* (Lartillot & Toiviainen, 2007), could be combined with the more local sound-motion object feature studies. This could also provide us with quantitative information about various typological and morphological features, providing the acoustic correlations of these subjective features as was indeed the stated long-term aim of Schaeffer's music theory.

#### **REFERENCES**


## #6 MULTI-FORM VISUALIZATION: A METHOD TO COMPOSE ACOUSMATIC MUSIC

#### **6.1 Introduction**

In acousmatic music, the sonic realm, traditionally qualified as immaterial and ungraspable, is manipulable and concrete, thanks to Schaeffer's (1966) examination of musical listening and the factors of psychological conditioning that influence it. The composer's practice of sound recording, and her ability to intervene in the structure of sound in ways that transcend the actual physical process of sound production and its sound source, challenge the instinctive and natural tendency to draw connections between sonic and visual stimuli. When the actual physical sound sources are absent and sound is apprehended solely through loudspeakers, the listening experience becomes acousmatic. In this expose, an examination of the perceptual process allows us to examine the perception of ´ qualia in music. I then demonstrate how conceiving the perceiver's mind as embodied is key in the development of my compositional practice, which is based on the idea of a multiform visualization.

#### **6.1.1 The perceptual process**

The process of perception is complex and has relevance to philosophy, cognitive sciences, anthropology and geography. The primary sense of proprioception which, through continuous sensations triggered by the muscular activation of the body, helps us to be aware of our physical existence as organized entities. Therefore, perception is a first person experience (Lamedica, 2014) and the perceiving body is the origin of an ego-centred system of coordinates that evolves in an environment (De Monticelli, 2013). Delivery of information from our sensory apparatus, such as kinaesthetic sensations, makes us con-

**49**

scious of our position within the environment such that we can actively explore the world around us.

According to Fugali (2013), the sensation of being physically anchored in the world relies essentially on touch, which is *exteroceptive*<sup>1</sup> and *interoceptive*2. That is to say, touch is both outwardly and inwardly oriented, providing information about the world's material features and an awareness of our bodily posture, balance and inner condition. The perspectival way we gain access to the world—determined by the dynamic interaction between our sense organs and defined as the mobile parts of the sensory system—shapes our perceptual experience (Gibson, 1983) .

The lived environment provides us with different forms of stimuli that we generally define in regard to the different sensory modalities that characterize our body: visual, aural, tactile, olfactory or gustative. Our sensory modalities represent the different channels through which information emitted from the physical environment is received by the perceiver. Once information has been processed, the perceiver, in turn, sends a bodily response to the environment through one or several sensory modalities. Therefore, the perceptual process can be defined as interactive. Gibson (1987) considers perception as a direct, mutual and continuous relationship between an organism and its environment. The complexity of our constant interaction with the world around us invites the interweaving of consciously and subconsciously collected and processed information. Thus perceivers constantly adjust their own set of perceptual skills to the features of their environment.

#### **6.1.2 Sensory engagement**

Cognitive sciences see the nature and functioning of our mind as mainly determined by our bodily constitution and the way we engage our sense organs in an active exploration of our environment. O'Regan and Noe (2001) conceptualize the sensorimotor aspect of ¨ our perceptual process to examine the nature of our sensory engagement. They observe perceptual experience as a mutual co-determining relationship between body and environment and explore the idea of phenomenal feel (Torrance, 2005) through two features: *corporeality*<sup>3</sup> and *alerting capacity*4. These two properties help to elaborate sensory consciousness. They are based on the skills involved in perception—namely the articulation of specific patterns of sensorimotor interdependence between perceivers' sensing-

<sup>1</sup>A sensory modality that is outward-oriented. It turns to an object in the external world or to material features. 2A sensory modality that is inward-oriented. It has a self-reflective structure, which allows the perceiver to be bodily self-aware.

<sup>3</sup>A feature used by O'Regan and Noe (2001, pp. 939-1031) to define phenomenologically perceptual experi- ¨ ence as a mutual co-determining relationship between body and environment. It is a characteristic whereby bodily movements affect how the environment offers information to our sensory apparatus. For instance when I observe a forest from far away or from inside, it is clear that my bodily position influences the way I relate to the forest. From a general perspective, a corporeal quality can define anything that has a body or material existence.

<sup>4</sup>A second feature used by O'Regan and Noe (2001, pp. 939-1031) to define perceptual experience phenomeno- ¨ logically as a mutual co-determining relationship between body and environment. It considers any perceived change in the environment, like the arrival of the rain, as an invitation to take a sensorimotor action and adjust my bodily attitude, for instance by covering my head with my umbrella to continue observing the forest. In this situation, I adapt my bodily position to the environmental change to remain focused on the forest as the object of my attention.

moving body and their environment. But is this theory applicable to acousmatic music listening?

In fact, when I listen to a piece of music, my bodily engagement towards the music in an acousmatic context is fundamentally different to the perception of my multi-sensory environment. The mono-sensory quality of my acousmatic experience––for example, I am listening to music, sitting on a chair in the darkness of the concert hall or in the studio––does not require the use of my sensorimotor skills as much as when I perceive the environment. Instead, it invites me to engage in a mental process and to adapt the focus of my attention through my cognitive skills. In these terms, a helpful way to widen the perspective offered by O'Regan and Noe is to examine the sensory dimension of the ¨ perceptual experience and to look at the concept of embodiment to articulate the impact of bodily experience in musical perception. In the context of an acousmatic concert, the corporeality and alerting capacities are not literally applicable to bodily engagement, yet they may be useful when transferred into the cerebral domain, using mental projection.

#### **6.1.3 The embodied experience**

In phenomenological terms, embodied experience is achievable in a unique and structured moment of presence. It intrinsically connects the perceived object to the perceiver in an interactive and constructive mediation. I am bodily aware of the multi-sensory structure of my living environment thanks to the inextricable intertwining of my senses rather than by the independent action of each sense organ. The combination of stimuli, either modal (visual, tactile, smell, aural, olfactory) or bodily perceived (proprioceptive, kinaesthetic, vestibular) captured during my sensorimotor exploration and then processed, provides essential information to phenomenally ground the perceiver's body in a *transmodal*<sup>5</sup> experience and understand the world in which I live. For example, when I look at a flying bird, an active sensorimotor cooperation between sense organs occurs. The capture of supra-modal information (such as proprioceptive, kinaesthetic, exteroceptive ones) and visual information positions my body as a stationary and grounded self while I perceive the bird as a non-self body, moving in the air. Each modality explores the environment in its own style, picking up and pairing multi-modal information. Reflection on how this multi-modal information can be held together in the image of a world apparently coherent, solid and meaningful like our living environment becomes an important way in which we explore the nature of our senses themselves and the way they contribute to our understanding of the world and our place in it.

The notion of embodied mind is conceived as relational, distributed over body, brain and environment (Scarinzi, 2014), without being attached to any physical structure. From this perspective, the sensorimotor contingency theory seems to apprehend only partially the whole experience of life because it focuses just on objective instrumental actions without accounting for the non-physical aspect of life experience, which is the organism's subjectivity. Our subjectivity, which gives its unique colour to each of our experiences, is according to Fuchs necessarily embodied (ibid., p. 74), that is to say, conditioned by how

<sup>5</sup>Perceptions in which one sensory modality triggers those of another. For example, sound and vision can provide me with spatial, tactile or kinaesthetic information, whose intermeshing imbues a particular scene, or a landscape, with multi-modal qualities.

our body is shaped and how connections between senses are physiologically constructed. So in relation to the idea that our perceptual experience relies on inner representations of the external world, as enactivism positions, our body determines our subjectivity—the way we singularly interact with the world around us through perception, thought, feelings, desires and imagination. This defines us as autonomous and self-determined selves capable of generating meaning out of our experiential interactions with the world, revealing themselves as an entanglement of objective and subjective experiences. In this regard, understanding the notion of *qualia*<sup>6</sup> (Godøy, 2006) is key to understanding my compositional method.

#### **6.1.4 Musical perception: vectors of** *qualia*

The subjective sensations that accompany sensory experience are referred to as *qualia*. Our perceptual activity, thoughts and feelings as ensured by the constant dialogue between sensory information, practical knowledge and our subjectivity—are all contributary to the emergence of meaningful experience.

Like the world around us, a piece of music is essentially an object in constant motion, composed of intermeshed, sequential, overlapped gestural formations (Godøy, 2006). Our knowledge of the sounding world is rooted in motion, fundamentally based on our sensorimotor skills and bodily experience, which can be interpreted as gesture. In listening to acousmatic music devoid of anecdotal content we can still experience a certain degree of connection to and familiarity with the shaping influences and structures of the environment.

According to Smalley (1997) this could be due to the ability of sounds to communicate proprioceptive and motivating sensations that refer to our personal experience of sounding gestures<sup>7</sup> through the perception of their spectromorphology8.

Perceiving any qualia in music relies fundamentally on our body-centred and ecological perceptual behaviour, led by our instinctive tendency to seek coherence and meaning though connection between the unknown and our experiential knowledge of the world. Thanks to the electroacoustic tools available to composers to record, create and alter sounds, pieces of electroacoustic music give listeners the possibility to phenomenologically perceive infinite numbers of typo-morphologies9 and spectromorphologies. The relationship between bodily experience in the living environment and music perception inherently colour the way listeners may experience various qualia in musical listening, invoking our experiential knowledge of the physical properties of our living environment.

For example, the perception of qualia of motion could give the feeling one is virtually witnessing the making of a bodily gesture of a cellist to create a sound, or that one is observing a dancer moving in space. Following Smalley's thinking the propriocep-

<sup>6</sup>Qualia results from our capacity to feel and subjectively experience the materiality of the world thanks to our embodied mind and the dynamic performance of our senses. They are subjective sensations that accompany sensory experiences. They are personal and phenomenal (Huron, 2006).

<sup>7</sup>Smalley (1997) defines gesture as an energy-motion trajectory which excites the sounding body, creating spectromorphological life.

<sup>8</sup>Smalley (1997) defines spectromorphology as the interactive relationships between sound spectra (-spectro) and the way they change and are shaped through time (-morphology).

<sup>9</sup>The typo-morphology has been defined by Schaeffer (1997). It analyses and categorises each sound based ist type and a detailled description of its morphology.

tive sensation of gesture aurally perceived refers specifically to a sonic gesture related to physical sound sources like, for example, the opening of a door or footsteps. Sometimes, the motion perceived is not explicitly familiar and may mislead the listener's understanding of the composer's intentions and also lead her to forge imaginative connections. This is particularly the case when sound materials are abstract and their spectromorphology not 'firmly rooted in known sounding gesture' (Smalley, 1996). The qualia of physical presence are communicated through qualia of distance that can be associated with visual perception of the depth of field—both phenomenologically.

The quale of substance, perceived phenomenologically, is fundamentally based on spectromorphology, and features the proprioceptive sensations of density, mass and texture that relate to very intimate sensations of touch, invoking our experiential knowledge of the physical properties of our living environment, such as liquidity, hardness, smoothness, or ruggedness. The notion of substance also echoes the Schaefferian notion of *facture*10. However, this category of quale, less abstract than the other quale of physical presence, seems to be more connected to the materiality of our living environment and draws a substantial link to the recognition of anecdotal features, or physical sound sources, triggering mental images that connect instantly to our multi-sensory experience of the world.

#### **6.2 In my practice: the capacity to imagine**

The experience of listening to acousmatic music is a multi-layered process. It starts in the aural domain with the perception of sounds, which I generally consider either as abstract or as variably representative or evocative of my experiential knowledge of the sounding world—that is to say, through recognition of particular physical sound sources (like the sound of running water or a voice) or bodily sensations (for example, qualia of motion or substance). Listeners rarely perceive acousmatic music as exclusively abstract, anecdotal or figurative. Their experience is a continuous attunement to their feelings about what they perceive, supported by a constant dialogue between phenomenological features presented by a piece and their embodied mind. As a listener and a composer, I am naturally drawn to acousmatic pieces that feature abstract rather than explicitly anecdotal sound materials, as they engage listeners in a 'feel-relationship' with music based on perception of qualia. This has the potential effect for listeners of immediately engaging their being in perceiving transmodal sensations, rather than calling on a response based primarily on the cognitive process of sound source recognition. When I compose music, I see each project as an opportunity to create a new sensory experience, inspired by mental images based on perceptual experiences—real or imaginary—rather than inspired by the sound materials I have in my sound library or others I may like to record.

From a general perspective, my sources of inspiration are multi-modal—I conceive vision, touch and hearing as channels or containers capable of transmitting similar messages in different ways. That is why my source of inspiration can be as broad as the qualia of vast space, the qualia of the motion of the sea, the vision of darkness, the tactile

<sup>10</sup>In the Schaefferian typology facture is a criterion to describe the way that energy is communicated and evolves through time. It determines whether the energy of a sound is maintained continuously, iteratively or weakly.

sensation of a fizzy texture, the qualia of violent gesture, and so on. To tackle and implement these mental images into music, my method consists in analysing them through tables of key words, reducing them through sketches to explore how I could feel them through my bodily gestures and associating them with video clips whose observation becomes a way to absorb and remotely feel them. All these methods help me to make them 'mine', and to associate them with a spectromorphological design, an element of sound material, a process, a sequence or even a way to structure the form of a piece. The shaping of sound materials, like the visual dimension of my working process, is conducted using this feel-relationship. The sounds I choose and shape also profoundly influence the way I compose sound sequences by triggering new mental images. They emerge in turn from the perception of the phenomenological features of the processed sound materials and the sequences in progress. Therefore my compositional process can be perceived as a constant and recursive dialogue between mental images, qualia and phenomenological features, using the multi-sensory dimension of my bodily experience as an inspirational tool to compose music that is evocative of this multi-sensory dimension of human perception. I call this method the multi-form visualization.

In this very personal way of mapping and projecting mental imagery onto a practical compositional approach, the coupling of mind-body then becomes central to the compositional and listening experience in acousmatic music.

The ability to imagine is essential when the gestures we detect via spectromorphological characteristics are 'firmly rooted in known sounding gestures' (Smalley, 1996) or, on the contrary, when the spectromorphology has been so much processed that its initial gestural impetus becomes dissolved, losing all intimate connection it could have with our personal experience of life, we may feel completely detached from a work, 'as if *observing* invisible phenomena displayed on a screen or in space' (ibid.). Yet a feature of acousmatic music is that composers choose to deliberately shape a listening experience with no equivalent in the real world, leading listeners to build bridges between phenomenological features of sounds, associated bodily sensations and the mental images they experientially or imaginatively connect to sounds in order to find meaning. Moreover, when no realistic connection can be drawn between a piece and our physical environment, trying to picture a meaningful sound image of it cannot only be based on our bodily experience of the world but must also be supported by our ability to imagine. Indeed, a particular texture, the impression of a gesture, or a particular pitch can invite us to experience a conscious dream (*une reverie consciente ˆ* (Bachelard, 1971)) that progressively influences us and leads us in a flow of thoughts, images, and sensations in our mind. Between daydream and the sensory reality of a piece, this type of reverie comes to our mind as an alternative space, reassuring and open to the world but also intimate: it is a space where the aural sensations communicate sensations through various qualia.

#### **6.3 The use of multi-form visualization for An Ocean on the Moon**

#### **6.3.1 Building the project**

The use and design of visuals in my practice of composition is central. I primarily use sketches––but also images, photos or videos can work together with tables of keywords to help me in visualizing which sound sources and sonic qualities I should employ in a piece and how I could shape the spectromorphological evolution of a composition over time. For my piece *An Ocean on the Moon*<sup>11</sup> (2017), I also employed multiform visualization to stimulate the production of raw sound materials by instrumentalists, giving them the freedom to improvise based on multimedia resources I provided to guide them in the expression of particular qualia.

First of all, *An Ocean on the Moon* is a multichannel piece, composed for a 16-channel sound system that recreates the shape of a dome in order to offer an immersive listening experience. The sea is the central theme of this project. When I started it, my perspective on the sea was very much influenced by all the mental images I had in mind, like the joy of being rocked by the waves, the fear of drowning, the contemplation of its beauty or the experience of living at the seaside. At the start of the project I recorded various field recordings, sketches, videos and photos in order to find an angle to attack my piece. I spent time wandering in the city of Corfu and on the seaside at different times of the day to be in touch with the diverse atmospheres, colours and moods of my living environment at this time. As I accumulated pictures and video clips and repeatedly reviewed all of them, I started to connect each of them to particular keywords, representative of their sensory features and qualia. This led me to build the following table (Table 6.1) gathering these features (some are repetitive), which I organized into two different categories, corresponding to different energetic temperaments expressed by the sea, namely qualia of calm (A) and of violence (B).

However, rare were the moments when I could perceive a continuous violent or calm energy from the beginning until the end of a recording. In fact, the sea movements were very much similar to a breathing pattern, a sort of oscillation between calm and violence. Inspired by my observations, I took a pen, a piece of paper and let myself dive into the realm of my imagination where I reduced the sea to a phenomenon of pure motion. Drawing this line (Figures 6.112) was for me a revealing step because it made me experience proprioceptive sensations that I could relate to a sea wave, like continuity of motion, fluidity, freedom, softness, calm, or impetus of expansion. This led me to consider this waveform as the form of the piece, and furthermore invited me to contemplate how I could consider the temporal and spatial dimension of music in relation to this waveform and how I could integrate the opposition between qualia of calm (A) and of violence (B) (Figure 6.3).

<sup>11</sup>*An Ocean on the Moon* is an acousmatic piece that explores a maritime theme through the evolution of a soft yet shiny, joyful and destructive instrumental netting texture. A particular emphasis is placed on the diversity of sensations that may overwhelm a viewer in front of the sea, between memories of joyful moments, lonely wandering and sudden fear. This piece is an evocation of visual and kinetic experiences of the sea. Through this project, I studied the rich expressivity of instrumental gestures, sometimes methodically, sometimes impulsively. I also explored composerly gestures, which at each stage of the creative process works from the fluidity of drawn shapes: curved, voluptuous, and sometimes extreme. This project resulted from collaboration with several instrumentalists who performed musical sequences based on visual images and video clips. It was partly composed at the Ionian University, located in Corfu, Greece in the context of an Erasmus+ Exchange and it was completed in the Music Technology and Innovation Research Centre, De Montfort University Leicester, United Kingdom.

12The pattern of Figure 1 represents an oscillatory movement between qualia A (black curved line) and qualia B (red straight lines) evolving over time. The x axis refers to time, the y axis does not refer to any sonic parameters. While the curves evoke a cyclic, evolutionary, round and soft motion, the straight lines aim to illustrate percussive, aggressive and repetitive sounds


Table 6.1: *Sensory features and qualia featured in the visual materials collected for* An Ocean on the Moon

Figure 6.1: *First attempt to represent the breathing cycle of the waves*

Figure 6.2: *Curved line, resulting from a gesture*

Slowly, the shape, the sound qualities of the piece became clearer in my mind. I imagined a blending of instrumental and vocal sounds that would create various textures, gestures and motions, methodical and impulsive. In this regard, my collaboration with performers was key, as it allowed me to observe and integrate in the project musicians whose sensitive responses to the same multimedia resources and mind-maps of keywords were different. These mind-maps were essentially used as a trigger, and invited them to explore how they could gesturally and sonically embody the mental images, the imaginary or experienced multi-sensory sensations they associated with each mood. Each recording session was individual and consisted of seven improvised performances of 2-3 minutes each. Each resource was related to either qualia of calm or qualia of violence. I divided the qualia of calm into three different moods and the qualia of violence into four.

#### **6.3.2 The structure of the piece**

The recording sessions provided me with a significant quantity of sound materials. I created different working spaces in the digital software, called Reaper, that I was using to compose the piece. This allowed me to mix and compose sound clips to see which energy and sound quality would result from this. Then I started composing different textures without applying much sound processing. In this process, I was quickly confronted by different types of gestures regardless of the type of mood. They reflected the greater

(a) *Qualia of calm (A) described as Mood A-1, Mood A-2, Mood A-3*

(b) *Qualia of violence (B) described as Mood B-1, Mood B-2, Mood B-3, Mood B-4*

Figure 6.3: *The mind-maps used during the recording sessions: qualia of calm (subfigure a) and qualia of violence (subfigure b)*

or weaker fluency of each performer in improvising—some sequences were short, featuring expressive, assertive gestures with a strong sound presence (drums, harpsichord, violin and voice) while other sounds reflected hesitant gestures, a lack of dynamic and not much expressivity. I used this project as an opportunity to explore a new aesthetic and restrict the amount of sound processing to a minimum in order to explore how different instruments could interact with one another and create a surprising mix. To this purpose, I only used a few effects on my initial sound materials, such as noise remover, some filters, pitch, time stretch, and also very little reverberation. Then I started to mix these different sequences on Reaper, testing different spatial positioning and automating different motion trajectories with Zirkonium13. I was also very much influenced by the performer's gestures I could hear within the mix. While it was necessary to abandon some sequences––because of the way I designed some movements or because their combination with others did not convince me––I was satisfied with others because I could clearly associate them with specific qualia. For example, the combination of different timbres, pitches and gestures14, in some sequences managed to transmit the vision of the shimmering and moving surface of the sea, shiny and colourful.

Figure 6.4: *Different textures imagined before starting to record the improvised sequences (horizontal axis refers to time)*

This stage in my compositional process was particularly exciting. The sound sequences I obtained invited me to intuitively rearrange the initial structure (Figure 6.215) and refer back to the initial waveform (Figure 6.1) to propose a new structure based on this curved line. Following the idea of a flowing gesture, the new structure imaginatively describes the deployment of the energy-motion of the waves through time. I illustrated this perspective visually through four sketches corresponding to the final four-part structure of the piece (Figure 6.5), exploring the dichotomy between qualia A and qualia B. I used this presentation as a guide to achieve the composition of this piece. The colours

<sup>13</sup>Zirkonium is a digital interface employed to spatialize sound.

<sup>14</sup>An example: An Ocean on the Moon (from 0'40 to 1'44)

<sup>15</sup>Figure 2 is a reductive representation of my mental image of the sea, its the temporal motion.

used in the figures aim to illustrate the diversity of instrumental timbres I would like to use in these different sections. I chose these colours for their visual aestheticism, away from any synaesthetic purpose.

(a) *Part One* (b) *Part Two*

(c) *Part Three* (d) *Part Four*

Figure 6.5: *Proposal of structure of* An Ocean on the Moon

#### **6.3.3 The spatial composition**

I tackled the spatial composition of the piece as I went along by mixing and composing the different textures with a clear idea of which proprioceptive sensations I wanted listeners to experience throughout the piece. Accordingly, I designed the start of the piece with sparse sounds giving the sensations of sparkles of water drops that would come from the sky. Falling randomly on us, these sounds would progressively accumulate to create a sonic texture that would immerse us in a sound bath, marking the beginning of our journey into an imaginary sea world, made of gentle and sharp rocking movements, some voluptuous gestures that evoke sea streams, and even unpredictable spatial trajectories of some elements.

The Table 6.2 synthesizes all the features that inspired the final version of the piece. It is a guide that helped me to complete the composition of the piece. It combines the structure proposal, a summary of the mental images that emerged while I was listening to and mixing the recordings—which part corresponded to which quale (A or B), how the notion of knitting, texture and the quality of the sound mix have been thought through, and finally, how I conceived the work's general spatial composition.

#### **6.4 Conclusion**

Multi-form visualization is a powerful method to assist composers of electroacoustic music in their creative process by forming essential reductions of imagined sound into something that can be embraced and extended through direct auditory sensation in the studio, as a stimulus for production, an aid for the processing of raw sound materials and also as a guide to building a piece. It is a method I invite composers of electroacoustic music to follow in order to profoundly connect with qualia and also engage in the shaping of sound materials, in such a way that their phenomenological complexity leads to an intertwining of the realms of memory and the imaginary.

#### Acknowledgments

I am indebted to the Institute for Sonic Creativity (MTI2), in Leicester and John Young, for their support throughout my PhD research. I thank Theodoros Lotis for his warm welcome at the Ionian University, for his listening and support during the accomplishment of this project. I also thank Andreas Mniestris and Dionissis Batjakis for their help, and I thank the beautiful people and talented musicians who participated in this project: Alex Retsis, Elesa Papakosta-Smyri, Giorgos Stavridis, Rrezarta Krougia, Sevastianos Motorinos, Niki Kokkoli, Ektoras Remsak, Nafsika Karagianaki, Stelios Michas-Eglezos, Anna Katagi and Sofia Ketentzian.


Table 6.2: *Final inspirational guide designed to compose the piece* An Ocean on the Moon

#### **REFERENCES**


## #7 THE OCULARCENTRIC OBJECTIFICATION OF MUSICAL EMBODIMENT IN COGNITIVE CAPITALISM: COVID-19 AS AN ALLEGORY ON THE MULTIPLE SENSES OF TOUCH

#### **7.1 Introduction**

Triggered by the Covid-19 pandemic, this paper attempts a problematization of the notion of touch in musical performance. The de facto crisis of musical haptics due to physical and social distancing is here considered in the context of a wider phenomenon, namely the ocularcentric objectification of musical embodiment. This reduction of musical embodiment to its visual dimension has been a long, historical process, accelerated by the political economy of cognitive capitalism, including Covid-19 as a catalyst of pre-existent tendencies. By revealing a crisis of touch, the ongoing sanitary crisis invites us to further reflect on the meaning of musical haptics, beyond the visual properties of embodied gestures and beyond tactility in the design of tangible user interfaces. In that sense, Covid-19 becomes a modern allegory on the multiple senses of touch, similar to the allegories of the senses in Flemish Renaissance painting.

#### **7.2 The ocularcentric objectification of embodiment in contemporary musicology**

The role of embodiment in musical performance has been emerging as a central theme in the context of musicology's performative' and 'embodied cognitive' turns.<sup>1</sup> Tradi-

1For a good overview of the performative turn in musicology, please refer to Lalitte (2015); Clarke and Cook (2004); and Pace (2017). For the embodied cognitive turn, please refer to Leman (2008).

**63**

*https:/doi.org/10.34727/2022/isbn.978-3-85448-047-1 7.* This work is licensed under the Creative Commons Attribution-ShareAlike License 4.0 International (CC BY-SA 4.0).

tionally, the body was considered as a transparent tool for the realization of composers' intentions that were codified in notated musical works; today, it is valued in its own right as the central mediator between matter and mind, between culturally diverse musical phenomena and meanings, and between musicians and listeners alike.

In previous work (Antoniadis, 2013), I have claimed that this focus on embodiment takes specific forms of ocularcentric<sup>2</sup> objectification, which privilege the visual dimensions of musical performance. I have suggested that the historical development of post-WWII discourses on the performing body follows the general scheme proposed by the cultural anthropologist Jean-Jacques Courtine (2006): 'Where once we had subjects without bodies, now we find bodies without subjects' (p. 166). Vivid examples of this ocularcentric objectification of the body may be found in Stefan Drees' (2011) overview of body discourses in music after 1950, as well as in Harry Lehmann's (2012) philosophical examination of the digital revolution in music.

According to Drees (2011),

the body is set on stage not only in terms of the sonic outcome of performative acts, but also with regard to its visual aspects as artistically relevant *object*. This results in the conception of the body as a medium [...](p. 13., Translation and italics by the current author.)

Thus, the liberation of the body from the performative restrictions of the past coincides with a liberation from the monopoly of disembodied sonic ideals. Through the visual perception of bodily actions and images, music becomes an affair of the eyes as much as of the ears. For Drees, this implies that musicology can expand to include previously neglected genres, such as installations and performance art. This shift from the bodiless compositional subjectivity of the past to an audiovisual projection of the musical body corresponds to Courtine's objectification schema.

Similarly, and with direct reference to the French curator and art critic Nicolas Bourriaud (2002), Lehmann considers this expansion as the emergence of a 'relational music'. Absolute music is explicitly judged to be irrelevant in a digital culture, and music is understood as forging relations to images, performative actions and words, or what Lehmann (2012) describes as the strategies of 'visualization, theatralization, semanticization' (Visualisierung , Theatralisierung, Semantisierung) (p. 218). A shift from the traditional musicological dichotomy between absolute and programmatic music towards a new one, between visible and invisible music, seems to have emerged.

Moving on towards more systematic approaches to musical embodiment influenced by 4E cognition (embodied, embedded, enactive and extended), this ocularcentric objectification is manifested in the very theories of embodied gestures. I will refer here to Jensenius et al.'s (2010) overview of relevant literature, as well as to Shaun Gallagher's 'integrative theory of gesture' (Gallagher, 2005) and Marc Leman's communication model through corporeal articulations (Leman, 2008). In all these cases, I am interested in the fuzziness between the components of musical gestures that are visually conveyable, representable and communicable (defined as 'body image' by Gallagher), as opposed to their components that are irreducible to internal or external representations (defined as 'body schema').

<sup>2</sup>The term 'ocularcentric' is adopted here from Jay (1994).

Typically, bodily gesture assumes the role of a link between physical movement and meaning, or as Jensenius et al. (2010) put it:

(..) the notion of gesture somehow blurs the distinction between movement and meaning. Movement denotes physical displacement of an object in space, whereas meaning denotes the mental activation of an experience. The notion of gesture somehow covers both aspects and therefore bypasses the Cartesian divide between matter and mind. (p. 13)

To explain musical gesture's hybrid nature, Jensenius et al. adopt typologies and methodologies that bear unmistakable iconic elements. For example, the distinction between communicative, control and metaphoric gesture (Jensenius et al., 2010, p. 14), adopted by McNeil's (2000) approach in linguistics, reveals ocularcentric characteristics that permeate these distinctions: Regardless of whether gestures accompany speech to various degrees, according to the *Kendon continuum* (Kendon, 2004) in communicative models, whether they are an integral part of computational systems (the control model of gesture in human-computer interaction), or they are abstracted as musical concepts through metaphor, they are invariably objectified in visual terms. A good example of this objectification is offered by Jensenius et al.'s methods of musical gestures' analysis, which include interlinked spatial and functional components. Spatial components include performance scenes, body positions, and frames of action, similar to Laban's notion of the *kinesphere*, whereas functional components include the well-known distinctions between sound-producing, communicative/expressive, sound-facilitating and sound-accompanying gestures (Cadoz, 1988; Dahl et al., 2010). Strikingly, the visual analysis of gesture is maladapted to the open-ended, nested, coarticulative nature of dynamic musical gestures developing in multiple temporal planes.

The aporias cited above (hybrid nature of gesture as bridging the mental and the physical and its functional compartmentalization and fragmentation) are addressed in Shaun Gallagher's 'integrative theory of gesture', based on a distinction between body image and body schema. His theory is developed through the merging of two families of gesture theories (motor and communicative) and experimentally confirmed through the study of a deafferented subject, Ian Waterman, who despite his lack of proprioception, is able to gesticulate even in a blind condition.

Gallagher (2005) takes an important step away from the ocularcentric constitution of gesture, through the distinction between body image and body schema:

I defined body image as a (sometimes conscious) system of perceptions, attitudes, beliefs, and dispositions pertaining to one's own body. It can be characterized as involving at least three aspects: body percept, body concept, and body affect. Body schema, in contrast, is a system of sensory-motor processes that constantly regulate posture and movement—processes that function without reflective awareness or the necessity of perceptual monitoring. (p. 38)

This distinction becomes fundamental for his integrative theory of expressive gesture, whereby it manifests as a distinction between *morphokinetic* and *topokinetic* properties: the former are related to linguistic, cognitive, communicative, body image properties, whereas the latter are related to proprioception and are controlled by body schema. The fact that Ian Waterman can control morphokinetic (but not topokinetic) properties of gesture under non-feedback conditions indicates that expressive gestures rely on a completely different mechanism than instrumental or locomotive actions. They are inextricably linked to communicative and linguistic mechanisms that require no proprioceptive or visual guidance, even though they may themselves be visible. As for their topokinetic characteristics––the ones that firmly place gestures inside an objectified visual space of spatial coordinates––they are controlled by the 'blind' processes of body schema. In that sense, the relation between gesture and visuality is more convoluted than many taxonomies of gesture indicate. A final touch on the ocularcentric objectification of musical embodiment may be found in Marc Leman's (2008, pp. 160–162) model of musical communication between performers and listeners, based on what he terms 'corporeal articulations'. Corporeal articulations are bodily movements that encode the performer's musical intentions and become transmitted to the listener in the form of biomechanical energy through a mediator: the musical instrument. First, the performer's biomechanical energy is transferred to the instrument, a part of it transformed into sound and another part bounced back as haptic feedback. This haptic feedback, in combination with sonic and visual feedback, creates a closed loop, which is crucial for the performer's control of the instrument and for the illusion of body transparency, the fact that the instrument feels like a prosthetic extension proper of the body. Then, the performer transmits the sonic and visual energy to the listener, who can decode its meaning through mirror processes, meaning the imitation, explicit or implicit, of the original corporeal articulations and her mimetic resonance to them. This model allows for great interpretational latitude in that corporeal articulations may carry semantic meanings that are different for the performer and the listener, but universal in their sensory materiality. Crucially, this materiality irreducibly includes the visual modality, which is at least as central as sound in the transmission and meaningful decoding of corporeal articulations between performer and listener. As for touch, it invariably remains attached to the notion of haptic feedback, a fact to be problematized shortly.

#### **7.3 Covid-19 as a catalyst for the ocularcentric rendition of embodied experience into data in cognitive capitalism**

The ocularcentric objectification of the musical body is further illuminated through its biopolitical origins and neoliberal mutations, which lead to the political economy described under the rubrics of cognitive capitalism (Moulier Boutang, 2007; Neidich, 2013) and surveillance capitalism (Zuboff, 2019), and accelerated through the ongoing sanitary crisis of Covid-19. We will describe these developments in reverse order, starting with the current crisis and gradually unfolding the wider historical horizon that contains it.

The imposition of biopolitical lockdowns, curfews and socio-physical distancing measures on a global scale since March 2020 has been almost unequivocally justified across the political spectrum as an inevitable necessity in order to relieve systematically underfunded national health systems under neoliberal regimes during the pandemic stress, and to flatten the epidemiological curves to compensate for limited testing and ICU capacities. In the context of the ensuing financial meltdown, art forms based on an economy of physical presence and performer-spectator co-existence, such as live music performance and theatre, have entered an existential crisis unprecedented since World War II.3 Performing musicians' financial resources are collapsing, due to both the elimination of income from live concerts and the reinforcement of streaming services' domination, which was already a factor in the growing precarization of professional musicians prior to Covid-19.<sup>4</sup> The 'wet markets' of wild animals,<sup>5</sup> epicentres of modern epidemics since the end of the 20th century, are revealed to be intimately linked to the 'wet markets' of music: the immoderate and avaricious proximity of humans and animals, as the generator of *zoonotic diseases*<sup>6</sup> in the context of the wider ecological crisis, threatens inter-animality (*interanimalite´*) and inter-corporeality (*intercorporeit ´ e´*) (Boccali, 2019 after Michel Foucault) as the foundation of the musical act. The physical co-existence between musicians and listeners/spectators, a 'wet market' of impulses and products of the musical body, of excited emotions, sometimes of tears, sweat and blood, is now though at the disposal of yet another 'wet market': the musical body is appropriated, 'slaughtered' and turned into an object of transaction in the markets of streamed digital data among interconnected brains. The French economist Yann Moulier Boutang (2007) defines this configuration as 'wetware' (brains) and 'netware' (network) the biological and social layers complementing the traditional distinction between software and hardware, in the context of what he terms 'cognitive capitalism'(p. 89).

More generally, Moulier Boutang defines cognitive capitalism as the third stage of capitalism, after mercantilism and industrial capitalism. Its main feature is the appropriation and capture of the multiplicity of human experience by digital forms of capitalism. This general definition is articulated through fifteen different markers (Moulier Boutang, 2007, pp. 85-94), which outline the new relationships between advanced information and communication technologies, new forms of consumption and production defying the patterns of industrial capitalism, and the ubiquitous importance of the appropriation of *tacit* or *implicit knowledge*, including, for example, the knowledge of how to play an instrument. In his own contribution that is more oriented towards the culture industry, Warren Neidich (2013) defines four basic characteristics of cognitive capitalism, namely, the predominance of knowledge as commodity, the new conflicts between capital and immaterial labour, the new forms of computational machinery and a new relationship between cultural and neural plasticity (p. 15). It is here explicitly claimed that not only does the appropriation of knowledge shape culture and economy, but also that the outputs of this interaction feed back to the very constitution of the human nervous system, producing positive externalities such as innovation and seamless human-machine virtuosity, as well as negative ones, such as the psychopathologies commonly associated with cognitive capitalism (lethargy, stress, depression, tunnel vision and burn out). As far as the political economy of the arts is concerned, these characteristics of cognitive capitalism are usually associated with questions surrounding intellectual property, copyright, open access, intangibles, innovation and enterpreneurship––a good example would be the cur-

<sup>3</sup>https://www.culture.gouv.fr/Sites-thematiques/Etudes-et-statistiques/Publications/Collections-desynthese/Culture-chiffres-2007-2020/L-impact-de-la-crise-du-Covid-19-sur-les-secteurs-culturels (access 07.01.2021)

<sup>4</sup>https://www.weforum.org/agenda/2020/05/this-is-how-covid-19-is-affecting-the-music-industry/ (access 07.01.2021)

<sup>5</sup>https://en.wikipedia.org/wiki/Wet market (access 07.01.2021)

<sup>6</sup>A zoonotic disease (or zoonosis) is an infectious disease that has jumped from a non-human animal to humans. https://www.who.int/news-room/fact-sheets/detail/zoonoses (access 20.05.2021)

rent discussions around the exchange value of streaming in relation to the precarization of musicians during Covid-19. Nevertheless, this text aims at a different target, namely at the 'molecular' level of musical performance as embodied interaction and how these political economies do and will shape it in the future.

This molecular level of performance-related data takes us to the latest twist of cognitive capitalism, defined by the US social psychologist Shoshana Zuboff (2019) as 'surveillance capitalism'. In her research over the last 20 years, Zuboff offers a cartography of the unregulated 'wild west' of what she calls 'body rendition'. The term 'body rendition' expands on the appropriation of human knowledge by the apparatuses of cognitive capitalism, as Moulier Boutang and Neidich had already described it. It can be defined as the appropriation of embodied manifestations of human behaviour in the form of a 'behavioural surplus' of interaction data, which are used by the GAFAM7 corporations for behavioural prediction, modification and eventually control. As Zuboff puts it, 'ownership of the new means of behavioral modification eclipses ownership of the means of production as the fountainhead of capitalist wealth and power in the twentyfirst century' (Zuboff, 2019, p. 18). The human body is re-imagined as a behaving object to be appropriated for indexing and research, through a variety of data, which range from interaction data in the form of click-rates to GPS location data, movement acceleration data and intimate biometric monitoring.

Before speculating about how such performance-related data may be changing musical performance in the near future, it is important to stress musicians' central contribution to the very development of these interactive technologies. Crucially enough, the increasing virtuosity and performativity required at the user level through the historic development of human-computer interaction, from command lines to graphical user interfaces and eventually to tangible user interfaces and forms of augmented or virtual reality today (the so-called third wave of human-computer interaction), has been directly influenced by music performance. As the Canadian computer scientist and designer William Buxton (2008) puts it,

the real objective of the system's designers was to study human-computer interaction, not to make a music system. The key insight of Ken Pulfer, who spearheaded the music project, was that to do this effectively he needed to work with users in some rich and potent application domain. And he further realized that music was a perfect candidate. Musicians had specialized skills, were highly creative, what they did could be generalized to other professions, and perhaps most of all––unlike doctors, lawyers and other "serious" professions––they would be willing to do serious work on a flaky system at all hours of the day and night. (cited in Holland, 2013, p. 3)

Just as today's users interact in clicks and steps or tweets and notifications, performing musicians have always been interacting in breaths and beats, cues and signs, and the relation between the two is reciprocal, meaning that an increased 'musicalization' of cognitive capitalism interfaces and an increased rendition of performance data shape each other. Given the added fact that the most democratized systems today feature combinations of graphic user interfaces on the internet, it becomes clear that an ocularcentric constitution of these musical behavioural data is the norm rather than the exception.

7Acronym standing for 'Google-Amazon-Facebook-Apple-Microsoft', as used in the relevant bibliography.

Before moving on to the variety of musicians' responses to Covid-19 and how they affirm the central role of ocularcentrism in the audiovisual cultures of the new era, it is worth stressing the historical background of these developments, namely the fact that the objectification of musical embodiment has been a constant in music history through its technological mediation: technology has always been performance's ontology.

Firstly, one may consider the very biopolitical origins of musical performance itself, through the formalization of disciplinary techniques of the body, the increasing institutionalization of music education, the abstraction and symbolization of the musical act in the form of notation, the emergence of the notion of musical work and the strict hierarchies between composer and performer. Wolfgang Lessing (2014) has vividly shown the shifting meanings of the notion of performing technique, from a disciplinary method enforced from the outside to an internalized self-monitoring, a surveillance technique of the self, and it is here claimed that such developments foreshadow developments in human-computer interaction. Secondly, the history of music technology itself, defined by Douglas Keislar (2009, after Marshall McLuhan) as a series of 'mutilations' and 'augmentations' of materials and agents, affirms that the shock of Covid-19 is nothing but a catalyst for diachronic processes of abstraction and absorption of musical performance in the current apparatuses of cognitive and surveillance capitalism: 'mutilating' the physical co-existence and 'augmenting' its digital liquidization, the current crisis extends the historical process of an 'alchemical transformation', from pure praxis, to symbols, to the registration of physical energies and to their final rendition as digital data. Following the reflections of the French philosopher Jean-Luc Nancy on the ontology of technology, one could consider these diachronic processes of abstraction in musical performance as a 'dehiscence8': not an opposition to nature, but rather a bifurcation of the organic nature of musical performance, the creation of a relationship to itself. In this conception, the body forms a constant between real and virtual wet markets, whether in terms of an animality to be slaughtered or of networked and diffused brains.

#### **7.4 The last unassimilable frontier in cognitive capitalism: an enactive conception of touch**

This double crisis in the economy of the performing arts, which are based on physical presence, has provoked a multitude of creative responses, both artistic and technological, which fuel speculation about the future of live music. From solidarity concerts improvised on balconies to the virtual concert halls of established festivals; from the anarchic proliferation on social media of concerts by precarious musicians in their most intimate private spaces to the near monopoly of the teleconference platform *Zoom*, which has become the new wall of our online communications, including rehearsing and teaching; from the struggle against latency in the live transmission of auditory signals to the efforts

<sup>8</sup>Dehiscence indicates the spontaneous splitting in plant structures in order to release their contents, such as seeds or pollen. Here is the original quote by Nancy: 'La technique ne peut etre oppos ˆ ee´ a la nature, elle ne ` peut meme se manifester comme d ˆ enaturante ou comme destructrice de la nature qu' ´ a partir de sa provenance ` naturelle. . . . . Cette difference n'est pas une simple distinction de prori ´ et´ es : elle se pr ´ esente comme une ´ dehiscence, c'est- ´ a-dire comme le d ` ecollement interne d'une m ´ eme ligne ou d'une m ˆ eme surface ( ˆ a la mani ` ere ` dont s'ouvrent les antheres d'une fleur).' (J.-L. Nancy, 2020, p. 65) `

for a rapid democratization of new interactive technologies and networked performance, various questions arise. What would be musical performance's destiny in a situation of generalized digital mediation of audience-performer interactions? Could a culture of physical and social interaction simulations in a virtual hall ever be a substitute for real experience? Can we distinguish between physical distancing and social distancing? Beyond telematic performances in the form of '*Zoom* concerts' with poor image and sound quality, what exactly could the potential integration of physical interaction, for example movement or haptic data, bring about in the context of the Internet of Things (IoT), or of an augmented/virtual reality of the concert? What are the repercussions concerning the remuneration of virtual musicians or the copyright for live concerts that remain online?

My provisional answer to these questions lays bare the ocularcentric constitution of musical performance in the Covid-19 era: an audiovisual abstraction of musical performance, through democratized but low-quality machinery of live signal transmission, or even high-quality virtual concert halls, in the condition of a diffused Bentham *panopticon*<sup>9</sup> safeguarding a two-way surveillance, both of the musician performing for the invisible crowd of solitary eyes and ears, and of the audience, whose metadata are constantly tracked, indexed, evaluated, deanonymized and sold by the invisible 'data barons' of GAFAM, remains agnostic as to a dimension that interconnects physical presence, intercorporeality, social interaction, sonic vibration, energy circulation and affective potential. This dimension is touch.

One should not rush though into simple conclusions as to potential remedies of this lacuna of touch in the Covid-19 era, including the integration of haptic interactions in the current audiovisual apparatuses or in forms of virtual and augmented reality: the notion of touch I am referring to here encompasses a range of phenomena beyond sheer tangibility, vibration, or haptic/force feedback design for virtual instruments, as explored for example in Papetti and Saitis (2018).

Drawing on my previous work on piano touch from a continental philosophy and radical embodied cognition point of view (Antoniadis, 2021), I attempt a deconstruction of the normative perception of touch as physical contact, through its enactive rethinking in terms of movement coarticulation, multimodal diffusion, limit experiences and body transparency. A final word will be on the relationship between touching and listening, which opens these reflections to the communicative, social and deprivatized aspects of musical performance. In that sense, touch will be considered as a real, non-metaphorical feature that permeates through the communicative chain composer-performer-listener, and as a metonymy for a musical ecology, which is invariably physical, mental and social (Guattari, 1989).

From a philosophical point of view, touching has never been a transparent concept. Jacques Derrida's main contribution in what is a virtual encyclopedia of the philosophy of touching (Derrida, 2000/2005) is the deconstruction of a rudimentary phenomenology: touching is not simply about physical contact with a surface, about tactility, about immediacy and presence, about a specific modality, or a specific sensory organ. Touch is

<sup>9</sup>The panopticon is a type of institutional building and a system of control designed by the English philosopher and social theorist Jeremy Bentham in the 18th century. The concept of the design is to allow all prisoners of an institution to be observed by a single security guard, without the inmates being able to tell whether they are being watched. https://en.wikipedia.org/wiki/Panopticon

rather a cascade of mediations in all senses, which render accessible something untouchable. It is about interruption, tact, discreetness and tangents, as opposed to penetration and violence, especially so in the case of palpable effort against the resistance of a limit and its non-invasive transgressing. Paraphrasing Aristotle, Derrida (2005) writes,

(..) but ever since Aristotle suddenly hit on the manifold aporia of touch (*aporia*, he said then, and *aporeseie*); ever since he, Aristotle, foresaw all the obscurities of the tangible: touch isn't clear, *ouk estin endelon*, he says furthermore; it's *adelon*, inapparent, obscure, secret, nocturnal. (p. 4)

He further summarizes the qualities of touch that render it obscure, its Aristotelean aporias, as follows:


Through this summary, Derrida testifies to the multimodal diffusion of the sense of touch. He indeed comes to respond to Aristotle's aporias in the following passage:

(...) though it is obvious or 'clear' [*delon*] that, first, the 'organ' of touch is 'inward' or internal; second, flesh is but the 'medium' of touch; third, 'touch has for its object both what is tangible and what is intangible [*tou haptou kai anaptou*]' (ibid., 424a), one keeps asking oneself what 'internal' signifies, as well as 'medium' or 'intermediary', and above all what an 'intangible' accessible to touch is - a still touchable un-touchable. (p. 5)

The issue of an untouchable becoming touchable is inextricably linked to the experience of a limit:

How to touch upon the untouchable? Distributed among an indefinite number of forms and figures, this question is precisely the obsession haunting a thinking of touch––or thinking as the haunting of touch. We can only touch on a surface, which is to say the skin or thin peel of a limit (and the expressions 'to touch at the limit', 'to touch the limit' irresistibly come back as leitmotivs in many of Nancy's texts that we shall have to interpret). But by definition, limit, limit itself, seems deprived of a body. Limit is not to be touched and does not touch itself; it does not let itself be touched, and steals away at a touch, which either never attains it or trespasses on it forever. (p. 6)

In other words, according to Derrida, touching has an integrated failure of accessing what it actually reaches for, as it by default stops at a non-bodily, non-invaded limit. The destruction of this limit, say through penetration or violence, would immediately signal the destruction of the very notion of touch. The limit is to be transgressed otherwise.10

10Beyond the current focus on touch, a complete theory of sense-making in musical listening as transgression of immediate perception is offered in Reybrouck (2017). In this, it is argued that the surpassing of first-hand

Haptic perception has also been a focus of study in radical embodied cognitive science11 through the notion of dynamic touch. In his overview, Anthony Chemero (2009, pp. 154–160) has focused on work by Shockley, Carello, and Turvey (2004), who define a touch-specific affordance. According to this research, common illusions in the perception of objects through touch, such as the size-weight illusion,12 can be addressed not through a supposed erroneous computation, or judgment of the object's weight as analogous to its size, but through touch-specific information directly accessible in the object. To show this, Amazeen and Turvey (1996) experimented with the so-called tensor objects, which are specially designed objects of identical shape, size and overall weight, where the weight is, however, distributed in different parts of the object. The different distribution of weights produced different moments of inertia when the subjects attempted to wield them, whether having visual contact with them, or even when the objects were occluded and the subjects could only feel them. As a result, the researchers showed that humans perceive correctly the weight through the object's inertial potential as felt on their wrists. The point of Shockley et al. is that this inertial potential, or as they call it *moveability* of an object, is a touch-specific affordance: information available in the environment, into which humans can effectively tap through dynamic touch rather than through visualizations and representations.

Having investigated some basic features of touch (its multimodal diffusion, its mediating, transparent and transcendent nature, and its role in dynamic perception through active exploration), we will now see how it relates to listening as theorized in embodied cognition. Setting aside the fact that direct cross-modal correspondences between touch and sight have already been documented (Blakemore, 2005) in the context of mirror neuron research (Gallese, Fadiga, Fogassi and Rizzolatti, 1996; Rizzolatti, Fadiga, Gallese and Fogassi, 1996; Rizzolatti, 2002), that is, the listener may be feeling touch just because of seeing it, I will rather pursue the ecological idea that touch as action is encoded in the different modalities involved in Marc Leman's communication model already presented in the first section. In light of Leman's theory, an enactive conception of touch allows for its non-metaphorical, energetic transmission to the listener, beyond the narrower sense of touch as the performer's haptic feedback.

First, touch is transmitted in terms of movement coarticulation. The modes of tactile contact in musical performance are hardly decouplable from coarticulated bodily movement and from the design of the instrument as a prosthesis to the performer's body. For example, the normative *legato* touch in piano playing cannot be considered aside from a proper synchronization of the several anatomic parts, which allows for a certain pattern of energy transmission to the hammers and the dampers of the instrument. In that sense, touch is an organic, inextricable part of what Leman calls corporeal articulations.

12Given two objects of equal mass, people (both children and adults) judge the one with a smaller diameter to be heavier. For example, they judge a comparatively small pound of lead to be heavier than a comparatively large pound of feathers.

multimodal perception involves mediate knowledge based on cognitive and affective categories, spanning a continuum between concrete representation and abstract symbolization and involving distinct temporal categories ('in time / outside of time').

<sup>11</sup>The main feature of radical embodied cognition in relation to (non-radical) embodied cognition is the rejection of mental representations and mental computations as explanatory tools. In their place, radical embodied cognition employs tools from ecological psychology, describing the interactions of organisms and their environment, and dynamic systems theory, describing the way systems are changing over time.

Second, we defined touch in terms of its multimodal diffusion. Beyond the stimulation of tactile mechanoreceptors, the qualities of touch as movement are codified in other modalities, predominantly sound and vision. The expanded palette of touch in these actions is not only felt by the performer, through the resistance they induce, but also by the listener, through the transmission of their multimodal blueprint.

Finally, touch in the sense of experiences of a limit is transmitted through the listener's empathetic resonance and mimetic interaction. For example, forms of physical constraining of the performer require physical effort against the imposed resistances, which is literally felt and re-enacted by the listener. Moreover, this social dimension is further amplified by the bodily existence of many listeners in the same physical space, an idea initially developed by Maurice Merleau-Ponty (1964) under the notion of *intercorporeality*. In short, the cross-modal qualities of touch, the visual components of the related actions and the empathetic resonance to the exerted effort do not require a semantic representation of the psychophysical resistances, but create the conditions for a primordial experiencing by the listener.

The enactive definition of touch in terms of movement coarticulation, multimodal diffusion, limit transgression and body transparency offers a model for energy circulation in the 'aesthetics of presence', as formulated from a theatre studies perspective by Erika Fischer-Lichte (2004). In what she calls 'the soft concept' of presence, the sheer appearance of the phenomenal (as opposed to the script-related, semiotic) body of the performer and its coexistence with that of the spectators is a sufficient condition for an effect of presence to arise (one may note here the resonance with Merleau-Ponty's notion of intercorporeality cited above). Later on, it is not the sheer bodily existence, but rather the surrounding physical space and the spectators' active attention, which grants the event an enhanced quality of presence, what she calls a 'hard version' of presence. In the last twist of Fischer-Lichte's argument, a 'radical concept' of presence consists in the activity of actually sensing the embodied mind in its unity and the production and distribution of performative energy to the audience through techniques of the body. She cites the work of the Polish theatre director Jerzy Grotowski and the US theatre director Robert Wilson, whereby the musical qualities of physical movement enable the primordial experience of touch discussed above.

In Grotowski it was the concurrence of impulse and reaction, in Wilson there were the techniques of slow motion, rhythmicization and repetition, which evoke to the spectators the impression of a specific presence (*Gegenwartigkeit ¨* ) and enable them to energize themselves (Fischer-Lichte, 2004, p. 170. Translation by the current author.)

More importantly in relation to the audiovisual abstraction of Covid-19, she concludes that '[a]n aesthetics of the performative is in this sense an aesthetics of presence, not of presence effects, an aesthetics of emergence, not of appearance' (ibid., pp. 171, 175). This sort of presence and energy circulation that pertain to an enactive perception of touch are crucially not representable and not simulable through the screen-based interfaces of cognitive capitalism.

#### **7.5 Conclusion: Zoom walls as a modern allegory on the multiple senses of touch**

In the previous sections, an overview of the ocularcentric objectification of musical embodiment was attempted. Starting off with historical and systematic aspects in contemporary musicology, the phenomenon was further situated in the Covid-19 period. This current period functions as a catalyst for weaving together several biopolitical threads in music performance, ranging from the origins of the body's objectification in Western art music to embodiment's appropriation through analogue and digital music technologies. It was argued that amidst these developments, the sense of touch in music remains an unassimilable frontier. Due to its qualities beyond tactility, namely its enactive definition in terms of movement coarticulation, multimodal diffusion, limit transgression and body transparency, it can hardly be reduced to its visual dimension. In this way, it functions as a model for a genuinely anti-visual and anti-representational corporeality in music, which remains multimodal, interactive and dynamic, tailored as a diffused complex system of energy circulation.

The equally decentralized *panoptica* of *Zoom* walls intensify the current sense of touch deprivation in musical performance. Similar to Flemish Renaissance allegories, they evoke the sense of touch through its very lack, capitalizing on mediation and representation.13 Unlike the conscious limitation of means nurturing creativity though, as in the case of these allegories, our networked visual reductionism tends to operate as an enforced substitute of embodied experience: rather than attempting to explore ways of re-enacting the sheer range of action and energy circulation that define touch beyond tangibility, these means reveal a lacuna, which paradoxically and profanely renders desirable a 'musical contagion' in the midst of a pandemic.

In one of her responses to Covid-19, Catherine Malabou elaborates on the ambivalence of touch as contagion and its political dimension. Her object of analysis is Giorgio Agamben's notion of the contagion as 'a touch that disenchants and returns to use what the sacred had separated and petrified'<sup>14</sup> (Agamben, 2007 in Malabou, 2020, p. 221). In this sense, the imposed distancing in musical performance is both an act of consecration of touch, a juridico-political abstraction and purification of the musical act in a state of exception, as well as the catalyst for a 're-contamination', for the restoration of the primacy of touch in the musical communicative chain. Malabou's point is that 'exception cannot function without its aura, that is without the accursed share that constitutes it as exceptional. Contagion is transgressive. Instead of repressing it, let's make transgression contagious again' (p. 226). It is exactly the transgressive nature of an enactive notion of touch that becomes palpable through its absence in the current crisis and the promise of its return in the world after.

<sup>13</sup>A telling example is to be found in Jan van Bijlert's masterpiece A Courtesan Pulling the Ear of a Cat, Allegory of the Sense of Touch. Beyond the central depiction of an act of touch (the playful pulling of the cat's ear by the courtesan), it is rather the future expectation of the cat's violent reaction (communicated through its angry facial expression), as well as the suggestive nudity of the courtesan's back (potentially triggering the fantasy of a tender, sexualized touch), that communicate exclusively through the visual channel a complex experience, essentially multimodal and dynamic. The depiction of touch transgresses the painted surface and tells another story (an allegory, from Greek allos=another and agoria=story-telling) with potential moralistic overtones. https://eclecticlight.co/2017/04/15/painting-the-impossible-touch/ (access 07.01.2021) 14Agamben (2007), Profanations, Brooklyn: Zone Books

#### Acknowledgements

This research was funded in part by EUR-ArTeC, Universite Paris 8 through a post- ´ doctoral contract and by the Alexander von Humboldt Stiftung Research Fellowship for Postdoctoral Researchers, hosted at TU Berlin - Fachgebiet Audiokommunikation.

This chapter is dedicated to Jean-Luc Nancy, who passed away on the 23rd of August 2021.

#### **REFERENCES**


**PART III**

## EMBODIED GESTURE

## #8 EMBODIED GESTURES: SCULPTING SONIC EXPRESSION INTO MUSICAL ARTIFACTS

#### **8.1 Introduction: Sonic gestures and acousmatic music**

The cultural context of the 'Embodied Gestures'1 2 artistic research project is acousmatic music. Within this musical field, sonic artworks are commonly described through 'gestures'. For instance, a composer could assert that a musical passage was produced from the combination of sound gestures (Paine, 2004). The concept of 'sound gesture' is tied to the aural perception of sonic dynamics. The changing characteristics of a sound event during a period of time can be perceived as a trace, as a gesture (Van Nort, 2009).

Composers in acousmatic music (Schaeffer, 1966; Smalley, 1997; Vande Gorne, 2018) have described the tendency that listeners exhibit to deduce gestural activity from sound material. They observed how perceived temporal changes in sound materials––often called sonic morphologies––would always refer back to sensorimotor sensation. This particular effect has offered acousmatic composers a creative playground for exploring musical inventiveness, creating suggestive mental images, sonic sensations and associations. Interestingly, these observations are compatible with the experimental findings in embodied music cognition. In the embodied mind theory (Noe, 2004), perception is not ¨ something we receive. It is something we actively do. During the action-in-perception loop, external stimuli would be incorporated as mental simulations, as reenactments of

**81**

*https:/doi.org/10.34727/2022/isbn.978-3-85448-047-1 8.* This work is licensed under the Creative Commons Attribution-ShareAlike License 4.0 International (CC BY-SA 4.0).

<sup>1</sup>This article is an extended and revised version of the paper *Embodied Gestures: Sculpting Sonic Expression into Musical Artifacts* previously published by the authors at the International Conference for New Interfaces for Musical Expression NIME 2021, Shanghai. This article extends the sections *Interface Design*, *Musical Outcomes* and *Discussion* providing further documentation.

<sup>2</sup>The Embodied Gestures project, by Enrique Tomas, Thomas Gorbach, Hilda Tellio ´ glu and Martin Kaltenbrun- ˘ ner, was funded by the artistic research programme PEEK of the Austrian research funds FWF (PEEK-AR399).

what we perceive. It is particularly important that these simulations can involve sensorimotor activations. For instance, neuroscientists (Haueisen & Knosche, 2001) have ¨ observed how pianists activate their motor cortex when they only listen to piano music. If auditory perception can also be tied to sensorimotor sensations, a natural explanation for the perception of 'sonic gestures' would be the inherent production of sensorimotor simulations. This is probably why it is so natural to describe sound morphologies (e.g. temporal changes in pitch, volume and timbre) as physical activity, as movements.

How do acousmatic composers practically deal with the sonic gesture notion? Annette Vande Gorne developed a theory of *energy-motion* models building on previous work by Schaeffer, Bayle and Reibel (Vande Gorne, 2018). According to her theory, *energymotion* models are motion archetypes inspired by natural actions like oscillation, friction, flux, pressure, etc. For Vande Gorne, the application of these energy-motion models must begin at the very early stages of the musical piece's conception. Composers should devise sound materials following a well-defined energy-motion model. For instance, during a recording session, the composer first chooses the model and then performs the 'sounding body' (e.g. objects or musical instruments) having this model in mind3. The objective of this process is the production of expressive gestural sound materials for an acousmatic composition. Citing Anderson (2011),

through the energy model, the composer can develop a voluntary awareness of the internal stimulus which motivates and governs the energy flow unfolded through physical movement that results in gesture. Gesture would be articulated by and at the service of a particular energy model.

#### Vande Gorne methodically identified the following energy-motion models: *percussionresonance, friction, accumulation of corpuscles, oscillation, swaying/swinging, rebound, flux, pressure-deformation/flexion, swirls, rotations* and *spiral*.

Another relevant framework especially conceived to describe sonic gestures is Denis Smalley's (1997) 'spectromorphology'. In electronic music, audio processing can result in sound materials displaying remote relationships to any known sound-producing source. For instance, a recorded human voice digitally processed through convolution can be morphed into a radically different sound. Addressing this issue, Smalley proposed a framework to describe the rich variety of sonic content in electroacoustic music. He called it 'spectromorphology', as it would consist of a set of tools for 'understanding structural relations and behaviours as experienced in the temporal flux of [electroacoustic] music' (1997). Within this framework, the spectromorphology of a musical piece (i.e. temporal spectral flux of music) is mostly discussed in relation to 'gesture'. For Smalley gesture is an energy-motion trajectory creating spectromorphological life. Smalley specifically describes how listeners always tend to deduce gestural activity from sound and introduces the notion of 'gestural surrogacy', a scale of relationships between sound material and a known gestural model (e.g. first, second or third order and remote surrogacy). As we have seen, the notion of gesture is central in two of the most influential frameworks for composing and analysing acousmatic music. In the following section we will discuss 'sound gesture' from the perspective of embodied music cognition.

3For a better understanding of the concept of 'sounding body' we refer the reader to Thomas Gorbach's interview to Annette Vande Gorne in this book.

#### **8.2 Sonic gestures, artefacts and embodied cognition**

Many scholars have studied the multi-modal gestural images created by auditory information. The fundamental question of these studies has been elucidating what kind of physical gestures listeners associate with various musical sounds. The central hypothesis of these studies is that sound affords a form of memory recall to cultural and physical referents that themselves afford certain kinds of actions. One major problem in Western musical thought is the lack of an appropriate apparatus for describing holistically experienced musical sound. For this reason, researchers have often employed graphic methods, which facilitate the characterization of aural experiences. For instance, Godøy recorded participants drawing spontaneous gestures to various musical excerpts (Godøy, 2008). In this case the intention was studying the gestural rendering of what participants just heard. Musical experiences can be very complex and densely packed with events. With this experiment Godøy showed divergent results for steady pitch sounds with timbral changes, with some listeners drawing just a straight line and others drawing various curved lines. Indeed, some listeners expressed frustrations when they were asked to draw multi-dimensional sketches of what they experienced.

Caramiaux and Susini (2011) studied causal perception through movement. In particular, they tracked people's movements while listening to identifiable environmental sounds. Their results indicate that when the causing action is highly identifiable, participants mainly mimed the sound-producing action. When no clear action could be associated to a sound, participants traced contours related to sound acoustic features (e.g. pitch, volume, density, timbre, etc.). These dynamic features are typically called the temporal morphology of a sound.

There are also studies conducted towards understanding bodily gesture during sound production. In particular, Godøy, Haga, and Jensenius have developed experiments for analysing how people move while they mime musical control (Godøy et al., 2006). This gestural 'mimicry' has been described as performing 'air instruments', or making soundproducing gestures without making physical contact with any instrument or object. Nymoen developed a study for tracking participants' hands while they played 'air instruments' (Nymoen et al., 2011). He showed that the most significant parameters mapped by participants were pitch, frequency centroid and amplitude dynamics (volume).

Due to the lack of language for describing sonic events, participants often showed a tendency to look for association in order to understand sound. Users tried to describe sound examples in terms of familiar objects. Even in the absence of an object, they described sound in terms of artefacts. Tanaka (2012) asserts that the cognitive mappings enacted during these types of studies are always informed, mediated and inspired by the actual materiality of the controller used (i.e. size, material, shape, acoustic properties, etc.) According to Clarke we all have some ecological knowledge on how soundproducing actions relate to sound (Clarke, 2005). As Caramiaux has shown, musical cognition is always situated and sonic memories allude to certain objects to explain interaction. In sum, during the spontaneous rendering of movement people also envision artefacts (Caramiaux et al, 2015).

In conclusion, all these examples illustrate a clear tendency: humans tend to deduce gestural and sensorimotor activity from sonic gesture. In other words, we are inclined to determine the sound-producing gestures from what we are hearing and the possible objects producing this sound. Sound perception would also be referred back to some type of material or artefact. This tendency became the working hypothesis of the 'Embodied Gestures' project. With the aim of exploring a new possible paradigm for interface design, our research builds on the parallel investigation of embodied music cognition theory and the praxis of acousmatic music.

#### **8.3 The Embodied Gestures Project**

We argue that a fruitful path for approaching musical interface design––especially towards the creation of gestural music––could be the incorporation of archetypal sonic gestures as design patterns. Musical interfaces following this design paradigm would afford the same type of physical gestures that a sound material inspires when it is listened to. Our hypothesis is that such interfaces would be especially suitable to emphasize a performer's gestural embodiment within an instrument. For instance, for performing a sound passage made from the circulation of elastic sonic movements in space, we would design musical interfaces affording by themselves, and through their physical affordances, similar 'elastic' physical gestures to their performers.

The crucial question at the outset of this project dealt with finding successful ways of shaping the affordances of specific objects for suggesting particular body gestures. First, it was necessary to understand how listeners spontaneously envision sound-producing actions and physical materials from specific sound morphologies. After gaining this knowledge, we could then develop a number of interface designs. For this reason we planned a methodology based on user-studies and experiential evaluation which could help us identify suitable solutions according to design patterns. In particular:


#### **8.4 Ideating interfaces from spontaneous cognitive mappings**

In the early phases of this research project, we planned a study on 'gestural mimicry' especially designed to emphasize the material aspects of listening experiences. The aim of this user-study was to understand how people envision and materialize their own soundproducing gestures into physical characteristics when designing musical interfaces. Our hypothesis was that if participants are asked to mime sound-producing gestures while they listen to acousmatic music examples, they would also envision artefacts to produce that music. If we were able to find a quick way to mock up those envisioned objects, we could collect a repertoire of compatible sound-producing actions and artefacts for particular sonic gestures. This information would inform the subsequent phases of our project.

In the user-study that we originally created, we asked participants to produce physical mock-ups of musical interfaces directly after miming control of short acousmatic music pieces. We composed five sonic examples in the form of short acousmatic compositions emphasizing one of the following motion energies: *oscillation/rotation, granularity, attack/repetition/resonance, friction* and *pressure*. 4

Figure 8.1: *Resume of the Embodied Gestures user-study*

4These sound examples can be accessed from our website: https://tamlab.ufg.at/blog/embodied-gesturesmethodology (accessed: 1/12/2021)

As illustrated in figure 8.1, the user-study begins with a warming-up session where participants verbalized material aspects of the sonic examples (of each of the five short compositions). In particular, participants filled online forms to outline the gestures they perceived and the possible properties of the physical materials used to produce the sounds. After this phase, they were invited to stand up and mime control over the compositions. Directly after this, they spent approximately ten minutes producing a mock-up with clay. Once they finished it, they were video interviewed to obtain an explanation about their cognitive processes and the objects they envisioned (Figure 8.2). This process was repeated at least four times with four or five musical examples.

Figure 8.2: *Participants of the Embodied Gestures study miming control of acousmatic music and explaining the mock-ups produced (Photo: Enrique Tomas, 2018, CC BY) ´*

During four workshop sessions, 60 participants from five different creative backgrounds (music, arts, dance, computer science and administration) modelled more than

Figure 8.3: *Examples of mock-ups produced for four different energy-motion models (Tomas & Gorbach, 2019) (Photo: Enrique Tom ´ as, 2019, CC BY) ´*

200 physical artefacts (see examples in Figure 8.3). Participants were filmed and interviewed for the later analysis of their particular multimodal associations about music. Participants were divided in groups of three to five persons during the experiment. Each session had a duration of approximately two hours.

From the analysis of our video recordings, we were able to categorize patterns of physical gestures produced during the mimicry phase (Table 8.1).


Table 8.1: *Repertoire of sound-producing actions per energy-motion model observed during the Embodied Gestures user-study*

*Sound-producing gestures* have been well studied by Godøy, Haga, and Jensenius who identified two main types (Godøy el al., 2006). First, those human movements made with the intention of transferring energy to an instrument (excitatory gestures). Second, those human movements made with the intention of modifying the resonant features of the instrument (modulatory gestures).

From the results, we can say that participants firstly (and quickly) envisioned excitatory gestures. In the great majority of the cases, it took them less than ten seconds to spontaneously find a compatible sound-producing gesture for the sonic gesture they were listening to. Interestingly, participants easily engaged their bodies into various possible actions and internally evaluated whether these movements were compatible with the sonic gestures they were listening to. During a second phase, approximately twenty to thirty seconds later, and only after self-confirming the central sound-producing action, participants introduced additional features to their repertoire of movements. Each participant added other necessary bodily movements to perform the sonic transformations present in the music (i.e. progressive changes in pitch, volume, timbre). For instance, frequency changes were accommodated by producing the excitatory gesture at different heights. Volume was often controlled by modulating the speed of the sound-producing gesture. Certainly, this logic would allude to the causal schemes found in traditional musical instruments. We also observed the ways sound transformation was engineered in the artefacts that participants envisioned. Usually, they added an additional or complementary affordance to the initial form or configuration of the artefact they imagined (i.e. a new degree of freedom to the object) like knobs, sliders, buttons, additional sensors, acoustic effects, change of materials, etc.

Figure 8.4: *Embodied Gestures interfaces produced (Tomas & Gorbach, 2021). Note: ´ oscillation and granularity (top), friction and flexion (bottom) (Photo: Elisa Unger, 2020, CC BY)*

#### **8.5 Interface design**

The following phase in our artistic research project was centred around designing and building musical interfaces emphasizing four sonic gestures used during the user-study: *oscillation, granularity, friction* and *flexion*. <sup>5</sup> From the knowledge gained with the userstudy, we proposed a different solution pattern for each of the sonic gestures present in the music. These solution patterns are compiled and explained into detail in the table 8.2.


Table 8.2: *Solutions adopted for designing Embodied Gestures interfaces*

From these solution patterns, we built various technical versions during the project. The visual aspect of the interfaces produced in 2020 are shown in Figure 8.4.

The interfaces' technical core is an Espressif ESP32 WROOM microprocessor. It captures data from sensors and transmits this information wirelessly to a host using the Open Sound Control protocol. In our implementation, the host is always in charge of defining a sound synthesis strategy.

5Although a fifth sonic gesture, (attack and resonance) was used in the user-study we did not implement it

Figure 8.5: *Theodoros Lotis performing* Voices *with the friction interface (Photo: Elisa Unger, 2020, CC BY)*

#### **8.6 Musical outcomes**

To evaluate our project, we commissioned three musical works for 'Embodied Gestures' instruments. The first work was commissioned to the composer and performer Theodoros Lotis. In parallel, Jaime Reis composed an acousmatic piece with our instruments. Finally, the ensemble Steel Girls (Angelica Castell ´ o, Tobias Leibetseder and ´ Astrid Schwarz) prepared an improvisation for three instruments. Additionally, two of the authors (Enrique Tomas and Thomas Gorbach) produced two improvisations for two ´ of the instruments. All of these works were premiered and performed on various occasions in festivals in Austria and Greece.

We contacted these artists eighteen months before their respective premiere concerts. After a one-week training workshop, the artists worked independently for more than six months with copies of the four musical interfaces. Some captures of these musical works can be observed in figures 8.5 and 8.6.

#### **8.6.1** *Voices***: composition and live performance by Theodoros Lotis**

Theodoros Lotis composed and performed a musical work for one friction interface and interactive music system.6 Most of the sound material in *Voices* (Figure 8.5) consists of recordings of syllables and phonemes of an invented proto-language and audio recordings

6For a complete review of this musical piece, we refer the reader to the chapter *Gestural and Textural Approaches in Composition and Performance with the Embodied Gestures Instruments* in this book

of dancers' movements. Lotis studied the friction interface and introduced a taxonomy of possible trajectories in what he called the *gesture-field*, the spatial limits of the energymotion model. They are illustrated in figure 8.6.

Figure 8.6: *Taxonomy of motion profiles and gesture fields as they were used by Theodoros Lotis in* Voices *(Tomas & Gorbach, 2021) (Photo: Theodoros Lotis, 2020, ´ CC BY-NC 4.0)*

Theodoros Lotis explains that the gestural typology in *Voices* does not seek to divide time into small or larger linear temporal structures but rather to establish a style of floating narration.

The accompanying interactive sonic system in *Voices* consists of a Markov Chain model which stochastically selects the sound contents to be played. The interface's rotation and pressure values are sent to a mapping network application where they are weighted. This strategy is intended to mimic the overlapping one-to-many and many-toone gesture-to-sound mappings found on acoustic musical instruments.

#### **8.6.2** *Magistri Mei - Bruckner***: composition for fixed media by Jaime Reis**

The composer Jaime Reis explains the origins of this work:

'I had this idea for ages to think about polyphony of gesture and space, and then to actually have a lot of layers and polyphony and so on. This is one of the conversations that I so often had with Annette [Vande Gorne] which is, what are the limits of space lines? How many movements can you listen to at the same time?'

*Magistri Mei - Bruckner* is a sixteen channel acousmatic composition. Interested in exploring Anton Bruckner's sonorities and polyphony, Jaime Reis extensively used our interfaces to generate sound materials for this composition. In particular, following the acousmatic compositional method, Reis recorded many hours with a particular sounding body: our interfaces sculpting the sound of a number of GRM audio players loaded with a recording of Bruckner's *Missa solemnis*. After this, Reis worked on the organization of the recorded sound materials and on a complex spatialization strategy inspired by Bruckner's idiosyncratic use of polyphony.

For Reis, the process of sound material generation was comparable to the ones he usually develops with acoustic instruments and objects. However, he described the difficulties he found in defining 3D spatial trajectories with our interfaces. Reis usually elaborates them in a highly parametric way, calculating complex spatial trajectories on the computer. Reis would have required the development of a dedicated computer application able to map his movements to complex 3D spatial trajectories.

#### **8.6.3 Improvisation for Embodied Gestures instruments by Steel Girls**

The Steel Girls is an electroacoustic improvisation group formed by Angelica Castell ´ o,´ Astrid Schwarz and Tobias Leibetseder. With a long experience in the scene, the Steel Girls members show a clear physical and acoustic approach to improvisation as they usually perform with amplified objects. In this case, our interest lay in evaluating how our interfaces could be used by a small ensemble of improvisers.

Figure 8.7: *Steel Girls performing with oscillation, granularity and flexion interfaces during the Embodied Gestures premiere concert (Photo: Elisa Unger, 2020, CC BY)*

The Steel Girls prepared an improvisation for three of our interfaces: oscillation, granular and bending (Figure 8.7). Castello controlled the oscillation interface and mapped ´ the handheld device movement to a tape speed effect resulting in a typical sound-scratching effect. The device's angle and orientation were mapped to the central frequency of a number of resonant filters. Leibetseder performed a bending interface for controlling six parameters of a granular synthesiser. Schwarz played the granular bowl in order to trigger and transform the pitch of cascades of short sound recordings (100 milliseconds approximately) which were previously taken from the same bowl. Their improvisation resulted in a brilliant exercise of musicianship and communication on stage. As they did not plan anything apart from how to begin their performance, each member of the trio explored the different dynamic ranges of gestures afforded by the interfaces. Angelica Castell ´ o,´ who usually does not perform with digital instruments, asserted before the premiere:

For me, performing with computers is not sexy, but these instruments, they really are. Maybe they will reconcile me with the digital world!

#### **8.7 Discussion**

What are the main differences between composing or performing with these interfaces or with other musical controllers? From the interviews we carried out with our collaborators we can consolidate the most important observations:


these instruments have limits, and, after the limitless computer, it is good to go back to limits. All acoustic instruments are limited, like their tessitura and possibilities to articulate sound. And these interfaces have limits too. The way you push, the way you move around the objects, dictates how far you go with your time, with your temporal structures of music, and with the gestural structures. This was a good thing for me.

The apparent simplicity of our interfaces constituted a meaningful creative constraint for the collaborators. It stabilized crucial aspects of interaction which fostered musical exploration and inventiveness.

4. Tacit knowledge: quoting the words of Tobias Leibetseder, of the Steel Girls ensemble:

'These instruments tend to put you immediately in a specific bodily movement, and I like that because it is like beginning to perform or dance with the instrument with a really clear plan.

In our opinion, these types of interfaces benefit from the extraordinary tacit knowledge that many performers usually carry. In the case of Tobias Leibetseder, he is not only an experienced musician but also a dancer. For a performer who has experience in exploring bodily gesture there will be many possibilities for transforming musical intention into physical movement, and then finally through these interfaces, into expressive synthesized sound. In that regard, we observed how the straightforward functionality of our interfaces lowered certain early barriers. No manuals, no menus, no special computer music culture is required to operate these interfaces. If the devices are well set up and powered, any group of people can benefit from their tacit knowledge to create or perform gestural music.


#### **8.8 Conclusions**

Within the field of NIME and HCI we sometimes address complex and overwhelming issues. For instance, designing digital systems that enhance a performer's embodiment with the instrument. In this project, we escaped from elaborating complex or intricate interfaces. Our methodological approach began with experiencing––as opposed to understanding––the idiosyncratic ways of doing in our musical field. In other words, we first collected experiential expertise in what performing acousmatic music concerns (e.g. user-studies, workshops with composers, studio visits, concerts, building speaker systems, etc.). Only then were we able to define what a possible and intuitive solution for the issue in question could be. This is what Andrew Koenig called 'idiomatic design', advocating a solution not only by understanding the nature of the problem but also how the solution will be used, taking into account the constraints and cultures hindering its implementation (Koenig, 1996). We consider the incorporation of sonic gesture models into interface design as an idiomatic solution to the complex issue of disembodiment within the field of acousmatic music. This was done, in fact, at the risk of limiting and filtering the affordances of the physical artefacts we built. These limitations were perceived in this case as idiomatic, as creative constraints. However, we are aware that they could be evaluated differently from the perspective of other musical genres. Not all musicians who compose or perform digital, electroacoustic or even acousmatic music are interested in producing music from a gestural viewpoint. For instance, our interfaces will not be effective for the production of textural, ambient and drone music. Therefore, our interfaces could be described not only as idiomatic, but as highly specialized for the production of gestural acousmatic music. Finally, it is important to remark that our interfaces were specially designed to emphasize only one sonic gesture. As a consequence, composers and performers may require sets of these interfaces for composing from a diversity of models. Although this issue could be understood as a limiting factor, we also see it as an opportunity for the creation of interface ensembles, a plausible solution towards improving onstage communication between performers of digital music.

#### Acknowledgements

The Embodied Gestures project was funded by the artistic research programme PEEK of the Austrian FWF (PEEK-AR399).

#### **REFERENCES**


## #9 THOMAS GORBACH INTERVIEWS ANNETTE VANDE GORNE

#### August 29, 2020 in Ohain, Belgium

Following her initial classical studies first at the Royal Conservatories of Mons and of Brussels and later with Jean Absil, Annette Vande Gorne chanced upon acousmatic music during a training course in France. She became convinced by the quality of the compositional work developed by Franc¸ois Bayle and Pierre Henry and the revolutionary nature of this new art form: disruption of perception, renewal of composition through spectromorphological writing and listening conduction. Vande Gorne took a few more training courses on acousmatic music and then she studied musicology at ULB, Brussels and electroacoustic composition with Guy Reibel and Pierre Schaeffer at the Conservatoire National Superieur in Paris. ´

Vande Gorne founded the research group *Musiques & Recherches* and the *Metamorphoses ´ d'Orphee´* studio in Ohain. In 1984 she launched an acousmatic music festival called *L'Espace du Son* in Brussels, assembling a 60-loudspeaker system ––an acousmonium–– derived from the sound projection system designed by Franc¸ois Bayle. She is the editor of the musical aesthetics review *Lien* and *Repertoire ´ ElectrO-CD ´* (1993; 1997; 1998), a directory of electroacoustic works. She founded the composition competition *Metamorphoses ´* and the spatialized performance competition *Espace du Son*. She has gradually put together Belgium's only documentation centre on this art form, available online at *electrodoc.musiques-recherches.be*

Annette Vande Gorne was professor of acousmatic composition at the Royal Conservatory of Liege (1986), Brussels (1987), and Mons (1993). In 1995 she was awarded ` the *Prix SABAM Nouvelles formes d'expression musicale* (SABAM Prize for New Forms of Musical Expression). She still gives numerous spatialized acousmatic music performances, consisting of works both from her own repertoire, and those of other interna-

**99**

tional composers. Her current work focuses on the study of energetic and kinesthetic archetypes. Nature and the physical world are models for her abstract and expressive musical language. She is a passionate researcher of the various relationships created between spoken word, sound and meaning through electroacoustic technology. Relevant works include the *Tao cycle* and *Ce qu'a vu le vent d'Est*, which have renewed electroacoustic music's ties with the past. She has also made a few incursions into other art forms, including theatre, dance, sculpture, etc.

#### Part 1

*Thomas Gorbach (TG):* In your 'Treatise on Writing Acousmatic Music on Fixed Media' you build on the notion of 'play-sequence'. What is 'play-sequence'?

*Annette Vande Gorne (AVG):* Play-sequence is the result of playing a 'sounding body' or 'sound body' with a special category of energy-movements in your head and the adaptation to its surface. One sound body can create different energy-movements.

*TG:* I would like to ask you about what you refer to as a 'sound body'. Can you define its role in the setting of a play-sequence and an energy-movement?

*AVG:* It is the instrument, or better the surface, to play on. It has no relationship to the body, and it can be anything that makes sound. I prefer to reject the notion of 'instrument' because it is too close to the traditional conception of an instrument.

*TG:* Is it possible to call it a sounding object?

*AVG:* I reject the term 'object'. It is a term imported from the phenomenology of Husserl and Schaeffer which mostly deals with the perception of the 'objet sonore'. The 'objet sonore' is tied to our listening perception.

It is the notion of the 'archetype'<sup>1</sup> that interests me more, not the notion of the sound body. This idea was initially developed by Franc¸ois Bayle, although the choice of archetype comes from Guy Reibel's practice. The central question is: What can I communicate directly to the imagination of the audience so that they can immediately recognize the archetype's category I propose? For instance, falling, flying, rubbing, friction, oscillating, etc. In order to make a play-sequence I always have in mind a special archetype and then I play the sound body in such a way that I can produce this archetype. It is not a question of the physical gesture that I produce with my body. It is about the archetype's category, imported from the physical world, that I choose for the piece. I always look for 'sound bodies' that can produce this kind of energy in the best possible way.

However, it is necessary to add a musical layer to the 'play-sequence'. This musical layer is what I improvise from my personal experience as an instrumentalist, in my case the piano, but always in the kind of energy-movement I have chosen. The choice of an

1Note: Vande Gorne refers to energy archetypes that can be perceived in sonic events.

Figure 9.1: *Annette Vande Gorne in her studio performing a play-sequence with a 'sound body' (Photo: Annette Vande Gorne, 2021, CC BY-NC 4.0)*

appropriate musical layer is as important as the energy-movement and the sound body. But the play-sequence is not the composition. When I work in the studio exploring the recordings of a play-sequence, the miracle is that the musical layer of the play-sequence is constantly conserved during the process of composition. This the reason why I always start from a play-sequence.

*TG:* Let's talk about the embodied gestures instruments we have built. One of the project's basic ideas was building surfaces that could by their appearance and physical design alone afford an idea of how they might sound. Like when we see a violin, we also know how it sounds.

*AVG:* Yes, although you have put them in a sonic layer, because they do not sound by themselves.2 You made a step more that is the programming of the microchip, and I see that you have searched for a relationship between the acting bodily gesture and the sounding result.

2Vande Gorne explains here that, as the instruments are digital, they have to be connected to a synthesizer to sound.

Figure 9.2: *Annette Vande Gorne's objects for producing play-sequences (Photo: Annette Vande Gorne, 2021, CC BY-NC 4.0)*

*TG:* Talking about the embodied gestures, it would interesting if you could comment on the following instrument (Figure 9.3). We designed it with the idea of friction in mind. Its activating gestures would be pressing and rotating. Do you observe these energies in the surface of the controller?

*AVG:* I observe that there are two kinds of pressure. One is friction with few iterative results and the other is pressure understood as deformation. In the latter, there is a change in the sonic spectral compound during the direction of the pressure. Like the instrument called the jaw harp. So this instrument could in fact perform two types of energies. Although, for the instrument, it would be the same movement. It depends on the mapping.

Figure 9.3: *Embodied Gesture Instrument: The 'friction' interface (Photo: Elisa Unger, 2021, CC BY-NC 4.0)*

Figure 9.4: *Embodied Gesture Instrument: Vessel interface for granular control (Photo: Elisa Unger, 2021, CC BY-NC 4.0)*

*TG:* The following instrument is a vessel (Figure 9.4). Which energy comes to your mind?

*AVG:* Rotation and Spiral. The difference is that in the case of rotation pitch is constant while at a spiral the frequency goes higher when the movement accelerates.

*TG:* Rotating could also be possible by displacing the vessel around us. . .

*AVG:* No, then it is oscillation. It does not depend on the surface but on what is inside the vessel. Ah, correction, it is not oscillation, because it is possible to produce a balancing movement with this sound body. It is difficult to control the balancing. Oscillation is always mechanic, quick and regular. It is always possible to hear two poles but not the trajectory. Balancing is not regular and it is possible to hear the trajectory between the poles.

*TG:* Let's observe this instrument made of metal (Figure 9.5). What energy-movement do you associate with this surface?

*AVG:* It is a singing saw and I see in that the energy-movements of flexion and pressure/deformation.

Figure 9.5: *Embodied Gesture Instrument: The 'bending' interface (Photo: Elisa Unger, 2021, CC BY-NC 4.0)*

Figure 9.6: *Embodied Gesture Instrument: The 'oscillatory' (noisy-rotation) interface (Photo: Elisa Unger, 2021, CC BY-NC 4.0)*

*TG:* Finally, this is the first controller we created (Figure 9.6).

*AVG:* This is more complex. It is possible to make friction. It is also possible to use the tension and vibration of the string holding the intermediate object. It would also be possible to produce rebounding sounds.

*TG:* The string is not elastic, so rebound doesn't seem possible to me.

*AVG:* Then friction is the right energy because it is possible to go up and down and left and right with the handle in the middle. It is also possible to combine panorama with left and right, and pitch with up and down.

*TG:* You have experience with tape speed modulation in analogue studios. It is obvious to combine left and right movement with the speed of the tape, isn't it?

*AVG:* Yes absolutely. It is possible to use left-right movements to control the speed of the tape in correlation with frequency change.

#### Part 2

*TG:* Do you think these interfaces could be useful for composing in the studio?

*AVG:* It depends on which parameter you use during this process.

*TG:* Maybe for any special transformation?

*AVG:* Yes, for example I could use them for progressively changing the central frequency of a filter, its 'Q' parameter or its spectral envelope. We would be working in the sphere of sound transformation. For me, there are four different possible actions in the domain of transformation. The first is the 'fixed action'. This means that we will not change the parameter controlling the transformational tool while the music passes through different sound files. In this case the transformational tool acts like a sieve. Practically, it usually consists of a fixed preset of parameters with changing sound sources.

The second way to deal with transformation is what I call the 'mobile' mode. It means acting in a mobile way over the tools' parameters. The correlation, also called the mapping, between the quality of mobile acting and the final parameters' values is crucial.

The third acting mode is called 'control'. In this case, there always exists an external system like an LFO, a ramp or a square signal, which acts on the parameter without any relationship. Thus, in this case, it is also possible to hear or perceive the external system.

And the fourth acting mode, more electroacoustic, is the 'cross-synthesis', in which one parameter of one sound acts on another parameter of the same or another sound.

*TG:* Do you mean that the amplitude values of one sound can act on the frequency values of another sound?

*AVG:* Yes, this is most frequently done. But everything is far from gesture in this domain.

*TG:* Do you think that these [embodied] instruments could be used in a special musical setting? In other words, who could become a good user for these instruments?

*AVG:* They could be useful to control parameters of cycling musical parts with the gestures of my body. I could act livelier. This means that I have to have a previous idea of how the result would be, or an expectation in my imagination. This question is outside of the typical 'concrete' (musical) procedure, because in the 'concrete' method I do not know the musical result before my acting. With these controllers, and with any controller, I have to configure the parameters in relation to the bodily gesture. That means always before physical action happens.

*TG:* You have also experience in educating children. Would these instruments be useful for children interested in electronic music?

*AVG:* Yes, I see these instruments as being highly interesting for them. The most important thing to consider would be making the mapping process<sup>3</sup> in an adequate and intelligent way. However, I have an objection. When we make a play-sequence with physical sound bodies, the variety of results tends to be higher. For instance, two similar bells are never exactly the same object. If children use headphones to record play-sequences, they may hear small differences on what exactly is being played by each instrument. This is not a problem related to gesture. It mostly depends on the sound body used for playing our gestures. Thus, in the case of using controllers for education, it is crucial to define the kind of sounds. When I was asked by the direction of the music conservatory to teach 6- and 7-year-old children for a period of six years, I refused to teach *do, re, mi, fa. . .* . Instead, I invented games to record play-sequences with sound bodies. My experience was that they need to develop communicative relationships with other children during these games. One by one, or with the whole group.

After these experiences, I developed some other workshops with older children. About 10 to 14 years old. I observed that they first needed to understand the mechanism––how the technology of acousmatic music works. I took my Revox<sup>4</sup> and some tape, and built a studio in the classroom. They did the whole process of recording play-sequences, but they also liked to perform sound transformations towards building a result, a structure.

Finally, I would like to point your attention to the confusion created around the notion of 'gesture'. Bodily gesture and musical gesture are not the same. Musical gesture comes from the imagination of conducting sound along time.

<sup>3</sup>With mapping process Vande Gorne refers to the mapping sensor data to parameters in the synthesizer. 4The classic Revox tape recorder: https://en.wikipedia.org/wiki/Revox

## #10 GESTURAL AND TEXTURAL APPROACHES WITH THE EMBODIED GESTURES INSTRUMENTS

*The old man motioned me in with his right hand with a courtly gesture, saying in excellent English, but with a strange intonation. 'Welcome to my house! Enter freely and of your own free will!'*

—*Bram Stoker, Dracula*

**107**

#### **10.1 Introduction: gesture and texture**

In all types of traditional instrumental music, sound production is intertwined with the gestures and the bodily movements of the performer. Furthermore, the instrument is regarded as an extension of the body of the performer (Nijs, Lesaffre, & Leman, 2013; Schroeder, 2006; O'Modhrain, 2018). Both instrumental and vocal gestures serve as means of expression through the performer's physical motion and muscular energy. In this view, gesture is related to texture and vice versa in an inseparable way within a form-and-content relationship.

The relationships between gesture and texture can be examined under the lens of different scientific and artistic fields, including mathematics and physics, computer science, bio-arts, cognitive psychology, neuroaesthetics, neurobiology, neurophenomenology and more. The notions of gesture as forming principle and of texture as spectral behaviour are of paramount importance for the appreciation of the 'live' element in music performance (Emmerson 2007, 2001, 1994b) and of the interactions that emerge between a sounding body and a listening mind. Performative gestures and musical textures are active parts of a broader dynamic process that is related to the understanding of the musical act (Zanetti, 2019). Reybrouck (2021, p. 2) examines this process through informationprocessing models of cognition as well as operations of sense-making and models of enactivism and embodiment:

. . . that emphasize the self-organizing aspects of cognition as an ongoing process of dynamic interactivity between an organism and its environment (Schiavio et al., 2017). Musical sense-making, in this view, is not to be reduced to a conception of musical experience as a kind of abstract, decontextualized, and disembodied process as advocated by the cognitive approach to music listening and analysis. It should address, on the contrary, the actual lived experience of music, which involves more than internal cognitive processing and detached aesthetic appraisals. (Maeder & Reybrouck, 2016)1

Denis Smalley (1986, p. 83) examines the unfolding of music in time and the process of sense-making in terms of gestural and textural interdependency:

The relationship between gesture and texture is more one of collaboration than antithesis. Gesture and texture commonly share the structural workload but not always in an egalitarian spirit. We may therefore refer to structure as either *gesture-carried* or *texture-carried*, depending on which is the more dominant partner.

Thus, a sonic structure that contains intense gestures, frequent onsets and spectral transformations is perceived as gesture-driven, whereas a structure with minimal spectromorphological changes is perceived as texture-driven. However, a sonic structure always contains both gestural and textural elements in different proportions.

Smalley's description of interdependency implies that both gesture and texture carry complementary information about the source, the identity, the formation and the internal characteristics of sonic events. Although Kersten (2015, p. 196) is referring to the acoustic array and the musical invariants,<sup>2</sup> his point may support Smalley's argument that 'there seems to be a lawful causal relationship between the physical structures of sounds. . . and the stimulation of the auditory system'. For Smalley (1986, 1997), the physical structures of sounds can be decomposed into gestural and textural relationships: gesture generates spectromorphological and textural expectations and texture reflects its gestural origin. What Smalley suggests is that gesture and texture are essential elements of the acoustic array that transmits the sound and feeds the auditory perception (a soundreceiving system) with information.

However, the predominance of electronic technological tools in music creation challenged the relationship between gestural activity and spectromorphological development. Simon Emmerson (2001) argues that 'electricity and electronic technology have allowed (even encouraged) the rupture of these relationships of body to object to sound' (p. 194). This rupture becomes apparent especially in live or real-time performances where vision and optical stimuli play an important role in deciphering the gesture-field3 and the gesture-to-sound causality. As Smalley (1997) puts it,

1Further analysis is needed in order to decode the relationships between musical stimuli and the mechanisms of active reception by the listener. However, a full discussion of that issue is beyond the scope of this chapter which concentrates mainly on the spectromorphological implications of gesture and texture in composition and performance. A detailed examination of the topic can be found at Reybrouck, 2021; Kersten, 2017 and 2015; Schiavio, 2017; Emmerson, 2007, 2001, 1994a and 1994b.

2See also Balzano, 1986.

<sup>3</sup>According to the 'local/field' distinction (Emmerson, 1994, p. 31), '*Local* controls and functions seek to extend (but not to break) the perceived relation of human performer action to sound production. While *field* functions place the results of this activity within a context, a landscape or an environment'. In our case the term *field* is local and refers to the *gestural topology* of the performer, i.e the area within which the performer performs his/her gestures. Hence, gesture-field defines the space in which the performer acts.

We should not think of the gesture process only in the one direction of cause-sourcespectromorphology, but also in reverse––spectromorphology-source-cause. When we hear spectromorphologies we detect the humanity behind them by deducing gestural activity, referring back through gesture to proprioceptive and psychological experience in general... Not only do we listen to the music, but we also decode the human activity behind the spectromorphologies through which we automatically gain a wealth of psycho-physical information (pp. 113–114)

Bodily motion and causal gesture underlie all perceived spectromorphologies and relate them to their source. Whenever the inherent relationship between gesture and its resulting sound diminishes or disappears, the reference to the causality loosens up, granting its place to the realm of remote surrogacy.4 As a result, performers and audiences become increasingly alienated from purely physical sound production.

This alienation can be detected at different stages of a live or real-time performance. The 'amplification' of human gesture, often through new interfaces and disproportionate or na¨ıve mapping procedures, may create distorted and unreal sonic structures. A performer in front of a laptop producing gigantic masses of sound by merely pressing a button is a common example. 'The loss of appreciation of human agency within the sound world loses our immediate sense of the "live"' (Emmerson, 2001, p. 206). Consequently, the bond between performer, audience and sound perception is moderated, if not vanished, and the perspective of cause-source-spectromorphology is utterly blurred. It is a holy sacrifice though, an Iphigenian oblation for the winds of a *new perspective* of dislocated experiences (Emmerson, 2001, p. 204). Although Xenakis (1985) was referring to a new model of artist-conceptor, his remark suits this new perspective: '. . . an abstract approach, free from anecdotes of our senses and habits' (p. 3).

A suitable compromise is described by F. Richard Moore as *control intimacy*5, a notion that refers, for example, to minute textural differences caused by tiny alterations of embouchure position on a tube or bow pressure on a string. Grand or minimal, a gesture is a composite act with multiple impacts on the production of sound, as we will examine later.

For the moment, the old question persists: Shall we try to liberate the sound from its source? Shall we let it separate itself from its source and continue its own life in new spatial perspectives? Or, shall we hold the sound bounded to its source within the limitations dictated by the dynamic range of the performer's own gestural typology and the instrument's physicality? A great number of electroacoustic music works, whether acousmatic or with live and real-time elements, intersect in the shadow of this bifurcation. Karlheinz Stockhausen's *Kontakte* for example, '. . . presents intricate networks of relationships whereby differences between instrumental and electroacoustic practice and theory can appear simultaneously to conflict and support each other' (Dack, 1998, p. 86).

<sup>4</sup>According to Smalley (1986) remote surrogacy defines a state '. . . where links with surmised causality are progressively loosened so that physical cause cannot be deduced and we thus enter the realms of psychological interpretation alone'. (pp. 82–83).

<sup>5&#</sup>x27;For subtle musical control to be possible, an instrument must respond in consistent ways that are well matched to the psychophysiological capabilities of highly practiced performers. . . *Control intimacy* determines the match between the variety of musically desirable sounds produced and the psychophysiological capabilities of a practiced performer' (1988, p. 21).

#### **10.2 Voices**

*Voices*6 7 by Theodoros Lotis is a piece for one performer and Embodied Gestures interface, tape and electronics. Its duration is 10 minutes and 36 seconds. The piece proposes a proto-human<sup>8</sup> linguistic theatre consisting of primordial sonic elements and interpreted by voices of instinct reactions, voices that transmit impulsive expression, voices of humans, birds and frogs, and voices of fear, surprise, intimacy, intrusion, complaisance, ignorance, and caress. *Voices* uses two main categories of sonic material confronting the physical and mental states of mobility versus immobility and corporeality versus asomatous insubstantiality:

1. A collection of *vocal onomatopoeias* and articulatory *phones* and *phonemes*<sup>9</sup> related to the archetypal feelings of astonishment, fear, bewilderment, revelation, need, quandary, irony, uncertainty or ambivalence that are deeply rooted in the formation of the origins of emotional life and to innate feelings or primary affects. These onomatopoeias, phones and phonemes should not be regarded as representatives of any reality but rather as parts of a hypothetical expressive language uttered by imaginative prototypical humans. The following figure presents an attempt to transcribe the prosodic pitch and intonation of some of the phones and the phonetic segments in *Voices* into the International Phonetic Alphabet (IPA):<sup>10</sup>


#### Figure 10.1: *Prosodic characteristics of phonetic segments in* Voices

2. Sounds of *dancing bodies* submitted to gravity and trapped into their own corporeality: bodies in motion or stillness, in closed or open spaces, periodically inactive or carved by inertial forces.

<sup>6</sup>*Voices* uses as its primal sonic material the voice of the director and musician Giorgos Nikopoulos from his film 'The Ox' (recordings: Giorgos Gargalas) and dance improvisation recordings by Christina Mertzani, Enangelos Poulinas and Evangelia Randou (recordings: Theodoros Lotis). Other sounds used include voice (Agnese Banti), violin (Nikolas Anastasiou), clarinet (Esther Lamneck) and percussion (Giorgos Stavridis) (recordings: Theodoros Lotis and Demetrios Savva).

<sup>7</sup>*Voices* is commissioned by the artistic research project Embodied Gestures and had its premiere in October 2020 at the Echoes Around Me Festival in Vienna. Thanks to Thomas Gorbach and Enrique Tomas for initiating ´ the commission. A performance of the piece can be accessed from the website https://vimeo.com/561752213

8The Proto-Human (also Proto-World and Proto-Sapiens) language is the hypothetical genealogical predecessor of the current languages. Both term and concept are speculative and rather unpopular in historical linguistics. The term Proto-Human refers to a monogenetic linguistic origin of all languages, possibly during the Middle Palaeolithic era. For more information, see Ruhlen and Bengtson (1994).

9Phones and phonemes are phonetic segments. Phonemes are specific to a language while phones are not. More information on phonetic segments can be found at Port, 2008; Perreault and Mathew, 2012.

10These phones are part of the script of the film 'The Ox' by Giorgos Nikopoulos.

I consider these two sonic categories as paradigms of expressive articulation of what I might call *embryology of primary emotions* since body and voice are both vessels of all instinctive gestural behaviour. In my approach, the term embryology refers to a) the utterance of vocal/phonetic segments, and b) the sounds of corporeal movements. I consider both a) and b) as archetypal, pre-linguistic fertilizers of human communication. Beside language and music there are inarticulate sounds, groans, moans, exclamations and cries––not related to specific languages or musical styles––that can form musical elements. Beside dance there are unsystematic bodily movements that can be organized into choreographies. In that respect, the sounds of human utterances and of bodies in mobility or immobility included in *Voices*, focus the attention on the evolutionary struggle to rise, to move and to communicate. Chaitow, in his foreword to Beach (2010) describes these tasks as '. . . adaptative processes involved in our anti-gravity evolutionary struggle to rise from the floor––where sitting, squatting, crawling and wriggling are more appropriate––to the upright where standing and walking become possible' (p. viii).

Throughout the six scenes or movements of *Voices* the construction of phraseology, including composed and composite objects, rhythmical structures and leitmotifs serves the articulation of primal emotions via gestural behaviour and textural evolution.

#### **10.2.1 Typology of gestures in** *Voices*

Gesture is often regarded as a motion trajectory from a point A to a point B, an abstract vehicle that advances the textural content forward (Smalley, 1997; Hirst, 2011). In *Voices*, however, the kinetic behaviour of gestures is of minor importance. It is their narrative abilities that are substantial. In other words, it is not the teleological character of the gesture that prevails but rather its narrative appraisal. Gestures lose their property as textural chisels, thus liberating their potential to narrate and to create time as storytellers.

Gesture encompasses spectromorphological changes in texture by pushing the *musical narrative* to its logical (formal/systematic) and ontological (informal/intuitive) implications. Logical implications refer to the temporal structuring of the sound, which contains the *onset* or *attack* (how a sound starts), the *continuant* (how it continues) and the *termination* (how it ends) (Smalley, 1997, p. 115). Ontological implications refer to the grouping of onset-continuant-termination into the perceptual categories of *beginning*, *middle* and *ending*. The structural elements of onset, continuant and termination create spectral expectations. For example, an onset/attack may be soft or abrupt, sudden or gradual, expected or unanticipated. Accordingly, a continuant may have the character of statement, transition or prolongation (Smalley, 1997, p. 115). A continuant is always the outcome of an onset. A termination is always the outcome of an onset plus the continuant: the end of the story of a sound. In other words, the continuant happens because of the onset, and the termination because of the onset and the continuant. Both continuant and termination are contingent on the onset. At a higher structural level, the elements of onset, continuant and termination are shaped by a gesture. For example, when a piano key is pressed down by a high velocity gesture, the attack of the resulting note will be sudden and abrupt, followed by a prolonged continuant and a gradually decreasing termination. I refer to these interrelated structural stages as musical narratives. Throughout *Voices* gestural typologies are used in order to shape textures and utter the musical narrative.

Figure 10.2: *Graphic score of* Voices *(Photo: Theodoros Lotis, 2020, CC BY-NC 4.0)*

The gestural typology in *Voices* does not seek to divide time into small or larger linear temporal structures but rather to establish a style of floating narration. Hints of this style are given in the graphic score by the words 'cut', 'silence', 'distant', 'whisper', etc. (Figure 10.2). Thus, gestures act as timeless formative vehicles that connect the evolution of sonic morphologies with their narrative function.

#### **10.2.2 Typology of textures in** *Voices*

Most of the sound material in *Voices* consists of recordings of syllables, phones and onomatopoeias, and movements of dancers' bodies. The vocal category comprises mostly vowels and, therefore, its spectral content is often harmonic with varying intonation. The dancers' recordings are largely of noisy character with eminent attacks and diverse dynamic ranges. The textural character of the sonic material, whether grainy, noisy or harmonic, is interrelated with its spatial context. Minor or major spectral alterations are directly affected by spatial transformations and vice versa. This is especially the case in the fourth scene of *Voices*, where textural variations emerge through spatiomorphological modifications. Occasionally, texture provides the setting for gestural activity, a backdrop for the vocal and the corporeal sonic material.

#### **10.3 The C4**

'The Controller #4' (C4) (Figure 10.3) is a member of the Embodied Gestures family of interfaces. The Embodied Gestures project proposes

. . . a new paradigm of interfaces for musical expression especially designed to emphasize a performer's gestural embodiment within an instrument. For that, 'embodied gestures' explores the possibilities of shaping the physical affordances of designed digital instruments with the intention of inspiring particular forms of gesturality. Specifically, [the] objective is to study the implications of designing musical interfaces which can afford the same type of gesturality that a particular sound inspires.11

The C4 interface operates on the basis of two fundamental types of gesture: *pressure* and *rotation*. It is built around an ESP32 microprocessor which sends wireless OSC data captured by its sensors. This information may be used for parametric mapping and sound generation at host devices (i.e. a computer). The C4 is a rotation encoder with pressure sensing capabilities. It can be used by rotating its handle (generating four increments/decrements per step) and pressing it towards the centre of the instrument.12

Figure 10.3: *C4 interface prototype (Photo: Theodoros Lotis, 2020, CC BY-NC 4.0)*

The C4 affords the reinstatement of the performative physical activity as the control mechanism for spectromorphological evolution. It reestablishes the gesture-to-sound re-

<sup>11</sup>More information about the instruments can be found in the chapter *Embodied Gestures: Sculpting Sonic Expression Into Musical Artifacts* (Tomas & Gorbach) of this book. The instruments are the outcome of a ´ collaboration between the Institute of Media Studies, University of Art and Design, Linz and the Institute for Technology Assessment & Design, Vienna University of Technology funded by the Austrian Science Fund FWF, Programm zur Entwicklung und Erschließung der Kunste (PEEK AR99-G24). ¨

<sup>12</sup>Although in technical terms the C4 is an interface, it can also be described as an instrument since it is compatible with various performative actions, including different types of gestures. In that sense, the C4 can be regarded as both a performance controller that controls sonic parameters and as an instrument that hosts highly expressive performative gestures.

lationship and the tactile and visual features of the performance. As Smalley (1997) indicates,

Sound-making gesture is concerned with human, physical activity which has spectromorphological consequences: a chain of activity links a cause to a source. A human agent produces spectromorphologies via the motion of gesture, using the sense of touch or an implement to apply energy to a sounding body. A gesture is therefore an energymotion trajectory which excites the sounding body, creating spectromorphological life. From the viewpoint of both agent and watching listener, the musical gesture-process is tactile and visual as well as aural. Moreover, it is proprioceptive: that is, it is concerned with the tension and relaxation of muscles, with effort and resistance. In this way sound-making is linked to more comprehensive sensorimotor and psychological experience. (p. 111)

Although the C4 is not the source of the sound itself but rather the tool for its gestural articulation, it affords stimulation and control for sensorimotor integration in performance.

Apart from reassembling '. . . some of the cause/effects chains which have been broken by recording and computer technology' (Emmerson, 1994, p. 31) by addressing the issue of the stationary 'live' sound in performance, the C4 re-establishes the proprioceptive energy of the performer (tension and relaxation of muscles, effort and resistance) and the awareness of physical presence and motion.

Figure 10.4: *Gestural curves for pressure and rotation with the interface (Tomas & ´ Gorbach, 2021) (Photo: Theodoros Lotis, 2020, CC BY-NC 4.0)*

#### **10.3.1 Typology of gestures**

The C4 renders two main models of gestures: *pressure* and *rotation* (including variations such as swing and rebound). Pressure is a round-trip action model for exerting force between two poles: from a point of equilibrium to a point of maximum pressure and back (Vande Gorne, 2017, p. 19). It deals with the evolution and transformation of both gestural behaviour and spectral content. Its attributes include velocity, direction, acceleration and deceleration. Rotation is an archetypal model due to its cyclic and repetitive character (Vande Gorne, 2017, p. 20). It involves both a motion and a function. As a function, rotation can be applied to other types of gestures including pressure (rotational pressure, which equals the pressure-release phenomenon). Figure 10.4 demonstrates six curves in shape-space corresponding to the evolution of gestures with the C4 in the gesture-field absolute space of the performer. Gestural improvisations on the C4 were recorded with a camera. The most frequently occurring gestures of the improvisations were examined and outlined roughly in the sketches below.

The sketches in Figure 10.4 describe some of the gesture typologies within the gesturefield. Such paradigms include the *cochlea* (1, 2), the *linear* (3), the *butterfly* (4) the *free* (5) and the *square* (6) typologies as well as the *centrifugal* and the *centripetal* tendencies of the rotational gestures. The square typology (6) represents discrete, sequential and unidirectional gestures. The three axes represent the directions of pressure (P), the left direction (RL) and the right direction of rotation (RR). All the gestures of pressure and rotation start from the point 0, which indicates the position of balance of the C4.

The exact timing of gestural activity during performance is outlined in the *action score* of *Voices* (Figure 10.5).

Figure 10.5: *Page 2 of the action score for scenes 1, 2 and 3*

As indicated in the score, the volume is controlled by pressure and the panoramic by rotation. The performative gestures are divided into the following categories:

Long gestures with low velocity / fluid. These gestures concern both the pressure and the rotation of the handle. They are mainly preoccupied with the control of the overall volume and the panoramic.


#### **10.3.2 Anatomy of a gesture with C4**

Figure 10.6: *A gestural paradigm for* Voices

In Figure 10.6 we can observe the representation of the temporal evolution of a hypothetical gesture produced by pressing and rotating the handle of the C4. The gesture is divided into four discrete parts which are also indicated in Figure 10.6:


#### **10.4 Mapping network**

The mapping of mechanical performative gestures––such as the ones produced by the C4 interface––to sonic attributes raises some important questions: 1) Which and how many sonic attributes will be influenced by a gesture? 2) With what percentage or *weight* will these attributes be affected? These questions point out a fundamental issue of the mapping, which may be called *justification of mapping*. That is, to what extent can the produced sound be justified by the instrument's gesture and the predetermined mapping?

Control operations can be complex and must be analysed prior to any mapping. An act of control, such as the movement of the C4's handle, is determined by several parameters, such as the absolute position of the handle, its velocity and inclination, the degree of pressure exerted, etc. A prior to mapping analysis can demonstrate which of these parameters play a major role and which have minimal or no effect on the sonic attributes.

#### **10.4.1 Overdetermination**

These observations point towards the phenomenon of *overdetermination*, <sup>13</sup> whereby an event is certified by multiple causes, any of which would be sufficient to account for it. Consider the following example: A single movement of the arm that moves the bow on a string comprises various components, including pressure, velocity, direction, acceleration, deceleration and inclination. That means that for every alteration in the sound (pitch, spectral content etc.) multiple parameters join forces in collaboration. This rule is considered as absolute. Although only one of these parameters (e.g. pressure) can be used––and usually is in one-to-one mapping strategies––for the alteration of a sonic attribute (e.g. pitch), this alteration cannot be fully justified by that use. For example, the channelled energy (kinetic, mechanical, automatic, robotic or physical) that moves a potentiometer is often reduced into a single value that reflects the position of the potentiometer, ignoring, at the same time, all the other components of its motion such as the speed, the acceleration, etc. This omission of cooperative components may be referred to as *underdetermination*. In reality, however, no change can be made without the holistic synergy of multiple parameters. In mapping, as with the bow, a number of parameters

<sup>13</sup>The term overdetermination (Uberdeterminierung) is used by Sigmund Freud in The Interpretation of Dreams ¨ as a key feature that explains the presence of multiple causes in a dream.

that cause even the smallest change in sound should be taken into account. This is done by defining weights for each parameter.

Overdetermination in *Voices* is treated by an intermediate application called *Mapping Network*<sup>14</sup> that determines the rate of each parameter in the production and evolution of sonic events. Ellinas (2020) describes the software as follows: 'Mapping Network is an application for making complex MIDI controller-to-software mappings. Aiming to mimic the overlapping one-to-many and many-to-one gesture-to-sound mappings found on acoustical musical instruments. The interface is designed after the pin matrix popularized by hardware synths15, with the addition of specifying percentages (or weights) to each mapping, rather than choosing just whether two parameters are mapped or not. Mapping Network has also built in a rate-of-change calculation feature for control parameters, enabling the use of motion as an excitation gesture for sound'.

#### **10.4.2 The C4 and weighting distribution in** *Voices*

Figure 10.7 illustrates the processing method of triggered sounds in *Voices* and the weight distribution in mapping.

Initially, the C4's rotation and pressure lever sends its values to the Mapping Network application (2), where they are weighted via a Max patch (1).<sup>16</sup> Part of the sonic material in *Voices* is triggered by a simple Markov model. The Markov model is a Max patch which works with a weighted transition table of probabilities. When active, the pressure lever triggers a series of random numerical values, which enter the Markov patch and cause the calculation of weighting values and transition probabilities. According to these probability values, different sonic grains with durations between 100 and 300 milliseconds are triggered from a buffer of audio files. Subsequently, the sonic grains are processed by a pitch shifter that randomizes their pitch and by four delay lines in a Max4Live device (3).

Let us analyse the example in Figure 10.7. Mapping Network (2) is divided into columns (inputs) and lines (outputs). The first two columns accept values from the pressure and rotation lever of the C4. The third column (nul) is a bogus input for all residual weights.<sup>17</sup> Each line represents a parameter mapped onto the Max4Live device. In Mapping Network each parameter is correlated with a percentage or weight. In the given

<sup>14</sup>Mapping Network is an Open Source software developed by Demetrios Aatos Ellinas as part of his bachelor's thesis in 2020 at Ionian University. The software is written in JavaScript and its code can be accessed at https://github.com/dimitriaatos/mapping-network

<sup>15</sup>Pin matrices were used for patching audio and control signals in synthesizers such as the EMS VCS3, the Synthi 100 and the ARP 2500.

<sup>16</sup>Since the C4 is an OSC controller, its communication with the Mapping Network software introduces latency and data loss due to the OSC-to-MIDI conversion. This issue is partly addressed by filtering/smoothing the OSC data before reaching the Mapping Network software. In my personal experience (after several performances of the piece) the amounts of latency and data loss do not constitute a major drawback, and they do not notably affect the relationship between the gestures applied to the body of the C4, the audio processing and the resulting sound. In cases where the C4 is used wirelessly, the latency between the interface and the computer is even greater.

<sup>17</sup>A close examination of Figure 10.7 demonstrates the function of the last column, or bogus input. Consider the output line of amplitude 1 (Amp 1). It receives a weight of 0.0 (or 0%) from the C4 pressure column and a weight of 0.594 (or 59%) from the C4 rotation column (see line 5 at the bottom right corner of Figure 10.7). The residual weight of 0.406 (or 41%) remains unused in the third bogus column (nul).

Figure 10.7: *An example of weight distribution in* Voices *(Photo: Theodoros Lotis, 2021, CC BY-NC 4.0)*

example, the overall volume in the Max4Live device and a minimum and a maximum time in the Markov model (i.e. how often a probability is calculated or how often a sonic grain is triggered) are controlled by pressure and supplied with weights of 100% (1.0), 79% (0.79) and 100% (1.0) respectively (green arrows). Likewise, the delay times 1 to 4 are controlled by rotation and supplied with weights of 78% (0.78), 67% (0.67), 52% (0.52) and 23% (0.23). The delay amplitudes 1 to 4 are also controlled by rotation and supplied with weights of 59% (0.59), 38% (0.38), 31% (0.31) and 19% (0.19) (red arrows).

The numbers given above are determined by a trial-and-error intuitive approach, which mimics the 'bow-on-a-string' paradigm: by increasing or decreasing the pressure one can affect various sonic parameters to different degrees. Each of the six scenes of *Voices* has its own preset of weightings. Different performers of the piece can choose and apply different sets of weights for each scene. This method of mapping defines and justifies the resulting sound by acknowledging parametric nuances and level sensitivity. In addition, it allows for a certain degree of *mapping indeterminacy* as the synergy of multiple parameters with different weights does not lead to completely predictable results. As in the case of the bow, although we know how a *sul ponticello*––that combines pressure, inclination, etc.––will be heard, we can only predict the resulting sound to a certain extent.

#### **10.5 Codetta**

Liveness in performance is directly related to the distribution of the performer's energy through the instrument. Hybrid instruments such as the C4 and the Embodied Gestures family of interfaces in general attempt to restore the importance of gestural activity and the vitality of the performer's energy-field. For this purpose, any mapping strategy in a live or real-time performance should consider the importance of parametric weighting and the fact that certain sonic attributes are conditioned by multiple parameters, i.e. the texture of a violin note is conditioned by the pressure of the bow, its velocity and inclination, etc.

In that respect, taking into account the different rates of influence that each parameter imposes on the sound (overdetermination), we allow for a level of control intimacy by restoring the body-to-object-to-sound relationship. Emmerson (1994b) concluded his paper on the typology of local/field thus:

The mapping of performer gesture to control function: expression is multidimensional hence individual parameter choice and scale may need to be the result of a cluster of parameter controls each following a different law: hence the creation of global control functions which 'decide' more detailed values. (p. 34)

#### **REFERENCES**


## #11 EXPLORING POLYPHONY IN SPATIAL PATTERNS IN ACOUSMATIC MUSIC

#### **11.1 Introduction**

The multiplicity of musical practices in Western art music that arose from the technological developments involving the use of electricity applied to music were particularly expressive after the Second World War. Their associated terminologies are varied: *musique concrete, Elektronische Musik, acousmatique, cin ` ema pour l'oreille, sonic art ´* , among others. For the purpose of this text, their differences will not be tackled, but some of their underlying common characteristics, particularly the use of spatial concepts in *acousmatic listening* will be discussed. The decision to focus on acousmatic music concepts derives from the fact that most of the mentioned literature and musical concepts are grounded in musical traditions where space is a key element to understand this practice.

The focus on many of Annette Vande Gorne's conceptualizations is not merely for the allusion to her work in the present volume, but rather for her centrality to and legacy within literature and music creation in the acousmatic music world.

This text is organized as follows: A theoretical short introduction to the use of space as a parameter in acousmatic music and some of the main perceptive features involved in such practice, followed by a brief description of acousmatic music techniques, such as those presented by Annette Vande Gorne, and their relations to gestures. The final section describes how I have been able in my own music compositions to enhance spatial polyphony through various techniques.

**123**

#### **11.2 Space**

The study of spatial features in sound in general is a very complex task, but it can be handled by perspectives that are focused on *perception* and *reception*, framed within distinct sciences and associated methodologies. The study of Western art music may often involve understanding an increasingly differentiated socio-communicative system generated by a multiplicity of included musical practices within its frame. Here, rupture and novelty take on an important role, often creating a gap between what one can perceive, pay attention to or focus on when encountering such musical practices, and what the *music-makers* would think that should have been perceived (Reis, 2015).

The exploration and development of techniques that tackle spatialization in electroacoustic music has been a part of the associated musical practices since its origins with Pierre Schaeffer, Karlheinz Stockhausen, Varese and other pioneers. However, the liter- ` ature greatly excels the fields around the practices of Western art music history.

A brief review of some of this literature will be mentioned here, accompanied by some personal considerations concerning my own perception and musical practice.

The study of spatial sound features has been addressed in the frame of different scientific fields such as physics, acoustics, psychoacoustics, psychology, biological basis, technological approaches (Blauert, 2001; Opstal, 2016; Rumsey, 2001; Suzuki et al., 2009; Wollner, 2018), and in other perspectives. While the importance of such stand- ¨ points is of substantial significance to the study of spatial sound features in music, other perspectives based on creativity are nonetheless key to providing new paths for research that are based on artistic practices.

When trying to understand the conceptualizations and practices of the use of spatial features in music, one can find many specialized texts, articles, journal issues, etc. that describe a personal practice in general or in a specific work that may include technical, conceptual and methodological considerations, often deriving from personal research and artistic experiences.

Many of the personal perspectives that derive from a personal compositional practice, such as in the works of Chowning (1971) or Stockhausen (1988) among others, can often be extrapolated into other practices. Other texts can be considered from the beginning as more comprehensive perspectives:


1Terms such as soundscape, acoustic ecology and bioacoustics seem to have convergent meanings. This practice alludes to the study of the effects of the acoustic environment on the physical responses or behavioural characteristics of those living within it (Truax, 1978).

2Such as vector base amplitude panning (VBAP), distance-based amplitude panning (DBAP), Wave-Field Synthesis (WFS).

cations (2017); terminologies, technologies and compositional approaches within spatialization techniques (Lynch & Sazdov, 2017); the use of peculiar technologies such as the ultra-directional sound that one can obtain with a parametric loudspeaker array (Reis, 2016); immersive explorations of virtual performance spaces (Wollner, ¨ 2018, among others).

Many specialized journals have dedicated issues to related issues, gathering perspectives from a significant number of authors, such as Barriere & Bennett (1997), Chouvel ` & Solomos (1998), Dhomont (2008), Keislar (2016), (2017), Szendy (1994), and Vande Gorne (2011) among others, where one can find seminal texts from authors such as Anderson, Barrett, Barlow, Bayle, Boulez, Chion, Couprie, Dhomont, Dufour, Emmerson, Ferreyra, Justel, Lotis, Mandolini, Mary, Menezes, Nattiez, Normandeau, Parmegiani, Risset, Roads, Schaeffer, Smalley, Teruggi, Truax, Vaggione, Vande Gorne, and so many others who have largely contributed to the discussions of the use of *space* in acousmatic music.

Within the large topics that arose in literature, very often the topic of spatial conceptions that are focused on ideas of tridimensionality, or conceiving beyond the horizontal plane (Kendall, 2010), and other terminologies that have been systematized (Lynch & Sazdov, 2017). Groundbreaking works and research can be read about the specificities of terms such as dome, cupola, spherical, semi-spherical, immersive, 3D, among others, such as Barrett (2016), Brummer (2017), Kupper (2008), Normandeau (2009), and ¨ Stockhausen (1971). I find this literature to be particularly important when it comes to the domain of *spatial polyphony*, for the new possibilities that are opened up by such systems in comparison to systems that are based in the horizontal plane only.

#### **11.3 Spatial polyphony**

The development of acousmatic music practice often relates ideas of musical gesture to the perception of movements in space. But where can one draw the line between a mass of sounds and movements that are perceived as a texture, and the independence of voices and spatial patterns?

The answer to that question may rely on a large number of studies previously mentioned here. However, for the present text, I intend to explain how I view the concept of spatial polyphony as presented in the literature connected to acousmatic music, before attempting to respond to the question myself based on my own perception and current experience as a composer.

#### **11.3.1 As a concept and in practice**

Even in the realms of acousmatic music, the concept of *spatial polyphony* has been used with slightly different meanings according to the context in which they were being used.

Mary explains the importance of polyphonic materials and space referring to presenting a precise role in electroacoustic orchestration for each element (timbre partial) and therefore arriving at a polyphonic conception of space that is coherent with the polyphonic conception of the music materials and their inner connections to the position/ movement/ panoramic and to the volume/ mass/ depth (2006).

Menard discusses his ideas of a *(super) polyphony* of space through complex diffusion systems designed by the composer (2008).

Ascione (2008) describes his first experiences in 1985 with the creations of 16 track realized in G.R.M. mainly confronting differences between conceiving a spatial work for stereo or for larger multi-channel pieces. He mentions what is enabled by what he dubbed *polyphonic spatial composition*: more effective adaptation of the space to the musical subject thanks to the permanent control of the affectation of sounds in the aerial sphere (when compared to the *real time spatial diffusion* of a stereo piece); the possibility of a better evaluation of the plasticity of the work; the ability to make spatial paths within the composition, to give more perspectives, to better distribute the masses and forms which oppose and respond to each other, and to specify the locations for the listener.

Merlier (2011) distinguishes the ideas of *spatial polyphony* and *depth of the [sound] field*, mentioning that both characterize the superposition or spatial obstruction of several objects. The depth of the [sound] field is linked to the geometric occupation of space (in the sense of depth): a single sound object with a large spatial mass or several distinct sound objects distributed in space. As spatial polyphony underlies something more conceptual––such as the simultaneous perception of several spaces, or of several spatialities, or of several discourses of space––we can see that the two terms overlap in part, but are not synonymous.

I here refer to the latest concept of *spatial polyphony* as presented by Merlier. Having my ear and personal perspective as a reference, sound spatialization acts as a central musical parameter, where the distribution of sound within the loudspeakers is related not only to different kinds of movements and spatial shapes and paths, but related and associated to *energy models*; or, to put it in a different way, associated to spectral changes, different speeds and other parameters that make the spatial movements (paths) more or less recognizable––and even *un*recognizable in given moments. My overall goal in *spatial polyphony*, is to enable focus in order to distinguish and be able to describe each particular sound spatial movement, in the sense of a recognizable pattern or distinct feature that one would group as a single entity, inspired in the way described by the *Gestalt* theorists and the perspectives of perceptual psychology as Deutsch elucidates (1980).

#### **11.3.2 Limits in the perception of spatial polyphony**

One central question when creating and listening to works involving spatial polyphony is: How many independence *spatial patterns* can one listen to simultaneously?

Ludger Brummer (2017) establishes a proper link to polyphony in a common sense, ¨ referring to the examples of listening to a fugue, where up to and including three or four voices it is possible to listen to and follow each voice, whereas five or six voices can scarcely be perceived in detail. He does, however, mention that polyphonic listening is a skill that can be improved through training. Emmanuel Nunes' (1994) work *Lichtung I* features up to six simultaneous spatial paths distributed totally independently between eight loudspeakers. Karlheinz Stockhausen would often talk about the importance of perception, referring to *Cosmic Pulses*, from *Klang*, as the first of his works where he couldn't follow each of the individual (also spatial) lines during the whole piece (Siano, 2013). While in many of Annette Vande Gorne's writings (2008), in which she alludes to other composers perspectives, and to her lectures3, she professes her perceptual experience at the time as being limited to four movements or four differentiated geometric spaces. Over the years, personal conversations with Brummer, Nunes, Stockhausen, ¨ Vande Gorne and many other composers, scientists and audiences inspired me to consider what my personal limit might be for the maximum number of simultaneous spatial paths, and furthermore, to contemplate how I could construct a first musical work that would directly address this issue. I realized that if I wish to surpass the three / four distinguishable *spaces*, I would need an immersive space in a dome shape, since I believe that the addition of the dimension of height could provide a more accurate perception in the multiplicity of *spaces*. The work *Magistri Mei - Bruckner* (2020)4 intended to have up to seven simultaneous perceivable spaces. The perceptive result has yet to be tested in more than one studio or concert; nor has it either been discussed with colleagues and audiences, or subjected to further testing from other perspectives.

#### **11.4 Spatial polyphony in my music**

I have previously materialized my interest in aspects of spatial polyphony as a composer in previous acousmatic works such as:


<sup>3</sup>On several occasions, in particular while organizing concerts and related activities in Portugal (2015 and 2019) and while we were giving lectures together in places such as Kyiv (2018), Ohain/Brussels (2019) and Vienna (2019).

<sup>4</sup>Work commissioned by the Embodied Gestures artistic research project, funded by the Austrian PEEK programme.

I recently composed a new acousmatic work that features a strong personal interdependence between *gesture* and *spatial polyphony*. Inspired by Anton Bruckner's methodology in conveying traditional polyphony, I have recently been developing ideas of spatial polyphony that I try to explore not only through traditional polyphonic development, but mainly through sound spatialization connected to musical gestures expressed in spatial patterns that travel through a dome-shaped sound system, namely in the piece *Magistri Mei – Bruckner* (2020), for 16 channels in a dome distribution, composed within the frame of the project *Embodied Gestures*, using new musical interfaces<sup>5</sup> developed within this research project that were used in the conception and creation of this piece, alongside algorithms, regular patterns and gestures that were materialized in sound objects as a counterpoint to spatial polyphony.

Bruckner's sovereign mastery of counterpoint can be observed both in the predominantly polyphonic textures of the first three movements and in the massive fugue with chorale which forms the bulk of interrelatedness of the breakthrough provided by the Finale of his *Fifth Symphony* (1876), where the chorale theme 'breaks through' at the end of the exposition space (Hawkshaw & Jackson, 2001; MacDonald, 2010). The idea of *space polyphonic breakthroughs* in the organization of the layers was decisive in achieving the desired result of multiple layers that are here briefly describe in eight procedures that were conceived in connection to specific *energy models*6, leading to what I consider to be audibly distinct spatial patterns:


5Created by Enrique Tomas and Thomas Gorbach. ´

6I use the term *energy model* as taken from Vande Gorne's work, which in turn draws from the work of Pierre Schaeffer (descriptive vocabulary of listening), Franc¸ois Bayle (certain concepts defining acousmatic sound), and Guy Reibel (*play-sequence* and the importance of gesture), and that connects to a specific musical universe, usually a physical model, working as an *archetype* (a fundamental concept in conducting acousmatic listening) consisting of creating a sequence by applying a musical idea in relationship with the model (Vande Gorne, 2018).

7It is an analogy to the homonymous term that refers to the connection one can make in the frequency range giving the illusion of multiple voices when listening to a single melodic instrument, such as in a Telemann fantasia. Here the term is used in the sense of having a recognizable spatial pattern with a sub-pattern that can either create a localized action or a construct of, for instance, a sub-pattern of a distinct geometry that enables us to hear both the original pattern as well as the new one.

Although the perceptual features are less accurate when it comes to the discrimination of a sound source within height, when compared to our acute sense of space in a horizontal plane, particularly in front of us, the possibility of having sound patterns that travel above the audience allows an important feature of distinction between patterns.

When the polyphonic density is increased, the enhancement of a layer is usually achieved by a *gesture*. Almost all the sound material was made by the new musical interfaces developed for the aforementioned *Embodied Gestures* project. I mainly took short samples from works by Anton Bruckner and thereby created hundreds of *play-sequences* having in mind *energy-movements* using the new instruments and thinking about them as *sound bodies*8. I subsequently ruled out9 from the work the majority of the created *play-sequences*, as is usual for me when composing However, one of the most notable perceptive features that allowed me to create distinguishable layers was the contrast between such *play-sequences* made with these *sound bodies*, in contrast to others created by algorithms or using other simple programable interfaces.10 This contrast made *by the hand* created peculiar gestural spaces11, to use Smalley's (2007) terminology, that would emerge back on the surface and play a role in the memory, connecting a layer that was previously presented, which had started to dive into textural sound masses, but would again rise up and be more easily distinguishable to the ear in comparison with the other *play-sequences*. The importance of the physicality of the gesture in the sound result and its perception has also been tackled by Brummer (2017) and Vande Gorne (2018) in the ¨ context of acousmatic music.

This latest compositional experience allowed me to give light to two personal questions:


The answer for both questions when considering this work is *yes*. Although there are many ways to convey space polyphony, the composition of *Magistri Mei - Bruckner* allowed me to test new personal limits regarding the perception of the simultaneity of spatial patterns in conjunction with new interfaces that acted as *sound bodies* for the creation of my own *play-sequences* and their interweaving connection to their associated *energy models*, with their idiosyncratic perceptual characteristics that allowed for the nourishment of a rich space polyphony.

<sup>8</sup>To better understand the concepts play-sequences, energy-movements and sound bodies, I highly recommend reading Thomas Gorbach's interview with Annette Vande Gorne, as well as reading her trailblazing work (Vande Gorne, 2018).

<sup>9</sup>In the sense that I created many more sounds than the ones used in the final work.

<sup>10</sup>Many of the algorithms were created in SuperCollider, both in ways that allowed a sound sequence to be triggered by command lines, or in lines that were, for instance, controlled by the MouseX / MouseY.

<sup>11</sup>*Gestural space*––The intimate or personal, source-bonded zone, produced by the energy of causal gesture moving through space, as with performer and instrument, or agent and sound-making apparatus (Smalley, 2007).

#### **REFERENCES**


## #12 USER-CENTRED DESIGN AS A MODEL-BASED CO-CREATION PROCESS

#### **12.1 Introduction**

Due to digitization in almost every aspect of our lives, we are dealing more and more with contradicting requirements on technology-based services which are provided by companies, education institutions, or governmental bodies. Design with users, design for users, and voice of the customer techniques (involving the customer in the definition of the products or services) become very important in industry, especially user acceptance is key. Methodologically, everything is possible, but not everything is successful. We need the right approach for a user-centred development of innovative products. To avoid the gap between the use and design of systems, sociotechnology can be utilized as a guiding approach (Emery & Trist, 1960). Based upon the principles of participation at all stages of development processes, user-centred methods have proved themselves as very useful means of facilitating open and cooperative settings. Co-creation and mutual understanding among users and designers make it possible to design and develop successful systems that are acceptable to their users, both in their shape and look-and-feel, and in their functionality. This is a promising way to create sustainable systems for real use.

First of all, we have to understand why we should focus on users in design processes (Ritter et al., 2014). The answer is very simple: we want to create a system or technology that is intended for human use, no matter how much artificial intelligence is helping to carry out certain tasks automatically or semi-automatically, usually non-transparently, in the background. We want to design and develop an effective, safe, efficient, scalable, and––the most important among all requirements––enjoyable and usable system for people, in which users can experience what they know well because they have experienced

**133**

*https:/doi.org/10.34727/2022/isbn.978-3-85448-047-1 12.* This work is licensed under the Creative Commons Attribution-ShareAlike License 4.0 International (CC BY-SA 4.0).

similar things so far in their lives, based on that what they can remember. That is why we need to understand them, their characteristics, skills, experiences, commonalities, differences, and their use context, in which they will perform certain tasks by using the system or technology we provide. Understanding users entails specific methodological knowledge and skills on behalf of designers, focusing on ways of getting to know the potential users of an intended system, and furthermore, of involving them intelligently in the design process as much as possible. The process of understanding in order to inform the design for users involves the following actions (Ritter et al., 2014, p. 4):


All these actions require certain knowledge and skills on behalf of designers to establish the appropriate methodology, research, and design setting at the right time. User-centred design (UCD) helps to achieve not only a better understanding of users but also involving them throughout the whole design process. The consideration of human characteristics and capabilities as central factors in the design process (Preece et al., 2015) facilitates the creation of better accepted and sustainable systems, which are moreover, used.

Successful systems are the ones that go beyond individuals' requirements and capabilities by also explicitly considering the social interactions and environments of their users. Here, sociotechnology is the necessary framework to base the design on. To address all these aspects in a design process is not easy. We need to know what we have to ask when in the course of design projects. For instance, we have to find out who is going to use the system or technology and why. What are the goals of users? Are users willing to put effort into learning how to use the system or technology? How often are they going to use the system or technology? Will they use it alone or together with others? Besides the 'who' question, we have to ask why, how, and when the system or technology will be used. These questions are central, especially during the evaluation and experimentation stage with potential users of the system or technology. UCD methods like brainstorming, storyboarding, creating cultural probes, use scenarios and personas, mockups, lowand high-fidelity prototypes, and then later when design process progresses to user tests, thinking aloud evaluation sessions, focus groups, etc. are all very useful concepts to apply when answering these questions.

In the next section, we will summarize firstly the principles of sociotechnology that provide the base for a user-centred design process, e.g., to create embodied interactions to increase user experience while interacting with the systems provided. Secondly, we will introduce the user-centred design approach by showing their characteristics, especially in innovative design processes. In this section, we will explore the role of different kinds of models to facilitate Design Thinking methods in user-centred design processes. Finally, we conclude with some discussion points.

#### **12.2 From sociotechnology as a principle to embodied interaction**

Emery and Trist (1960) introduced the term sociotechnical systems to describe the complex interaction between humans, machines, and the environment aspects of the work system. The goal is to consider people, machines, and context when designing and developing such systems. Bedham et al. (2000) described sociotechnical systems as having five main characteristics:


Baxter and Sommerville (2011) introduced the term of sociotechnical system engineering to address the need to deliver the expected support for the real work in organizations. With sociotechnical system engineering they mean 'the systematic and constructive use of sociotechnical principles and methods in the procurement, specification, design, testing, evaluation, operation and evolution of complex systems' (p. 4). It is still a common problem that systems often meet their technical requirements but are seen by their users as failures because they do not deliver the expected support for the real use. To avoid producing failure in system engineering, sociotechnical principles and methods should be used in design and engineering processes. This can be facilitated by applying user experience design methods while designing new systems or interactions, especially when it comes to offering engaging and enjoyable interaction for users. 'Ubiquitous computing environments need to be responsive to people's needs, but also need to provide engaging and aesthetic experiences' (Banyon, 2019). Besides focusing on usability to achieve the best functionality and effective usage of systems, designers must think about maximizing users' pleasure while interacting with the systems they design, which can also be improved further to facilitate a certain (desirable) lifestyle imposed within the design of an object or interaction provided. A successful user experience requires the consideration of all senses in the interaction with systems aiming for high usability and acceptance by their users. To achieve this, we need embodiment in interaction mechanisms, at least to a certain degree, if not completely. Embodiment in this sense focuses purely on interaction with the objects themselves; as Dourish (2004) explains '. . . we take activity and interaction with the real phenomena of experience to be central, rather than focus on internal or purely cognitive interpretations.' This shows us that embodied interaction does not need any translation (van Rheden & Hengeveld, 2016, p. 349). Its specificity embeds meaningful input and output for users. Users control what is relevant for the activity. They decide what is needed for the interaction, not the object or the system itself. Of course, one of the goals of designers should be to achieve a high degree of seamless embodiment in the interaction, which again requires a precise mapping of bodily expression to the expression of the device's output. At best, this results in an arranged or coordinated way of acting that is smooth, gentle, and natural.

The question at the heart of this paper is how to ensure a successfully realized user experience in a new design. In other words, how to proceed in a design project in a way that understands the target users and their context, including their past experiences, and to consider this insight in the design of artifacts and interactions provided as part of the new design. My answer to these questions is to apply modeling in all phases of the design process by creating models of all findings gathered after studying the target users and their past and current contexts, as well as by preparing and accompanying the design process as a reflective and self-critical practice. In the next section, we will show which models are needed to facilitate a co-creation process in a UCD project, by putting users at the centre of attention.

Figure 12.1: *The iterative process of user-centred design (Adapted from ©DIN EN ISO 9241-210, 2019, p.21)*

#### **12.3 User-Centred Design as a Model-Based Co-Creation Process**

User-centred thinking is about creating a direct link between the current and future users (Baek et al., 2007; Wallach & Scholz, 2012). Gould and Lewis (1985) defined three principles for a UCD process: early focus on users and tasks, to gather knowledge about the social, cultural, personal, and all other types of characteristics of users; empirical measurement, gained by capturing and analyzing user feedback; and iterative design, based on iterations after each user feedback. The iterative process of UCD allows for approaching a final product step by step, by reducing development risks and avoiding dismissing big parts of the achieved components or results (Figure 12.1).

Besides involving users in design processes, we believe that Design Thinking (Cross et al., 1992; Eastman et al., 2001) is a very helpful approach in designing sociotechnical systems. 'Design thinking is a human-centred approach to innovation that draws from the designer's toolkit to integrate the needs of people, the possibilities of technology, and the requirements for business success' (Tim Brown, IDEO). Design Thinking as a framework provides a set of methods that are used in UCD processes. If we take Design Thinking as an approach seriously and apply (all) its methods thoroughly throughout the whole design process, we can easily follow the goal of understanding everyday practice and its actors. This would furthermore lead us to the design of systems that consider the context of use, user experiences, and the necessary technology support as a substantial part of the sociotechnological approach. Our objective in designing systems is being innovative and improving user experience. We think this can only be done by understanding the actors, their actions, their use context, and, of course, by including them as experts in the design process.

Exploring the Design Thinking methods that are necessary to set up and carry out a UCD process, we end up creating artifacts in each step of the design process (Tellioglu, ˘ 2016, p. 24). These artifacts are both enablers and hosts of the evolving design ideas. In the course of design processes, especially if they are user-centric, several models are created (Figure 12.2).

Figure 12.2: *User-centred design in relation to use, system, and interaction models, and assigned methods*

In a UCD process, contextual inquiry, the capturing of ludic experiences, and use context definition are needed to describe and understand the use scenarios, and furthermore to specify the users' requirements based on their abilities and conditions. Later on, designing and evaluating solutions for users requires the design of an interactive usercentred system and the following product design. The following models help to support these processes:


Our claim is that if designers do not create such models, they will fail their design purpose. The above-mentioned models offer a holistic view of the design objects and processes by helping both the designers and the users. They help address and fulfil the requirements for designing (embodied) interactions for better and improved user experience. *Use models* are applied to study and understand the target users, their habits, wishes, mental and physical conditions, settings of use, and desires. Furthermore, they enable experimentation with the users to find out how to engage them and how to motivate them to remain active in the application of the design object. *System and interaction models* continue with designing usable artifacts as parts of the whole system under development, based on the knowledge and experience gathered and provided by use models. They facilitate design corrections by means of co-creation processes carried out together with target users by ensuring the achievement of maximum usability, clarity, pleasure, and satisfaction of users while interacting with the system. Embodiment can be an important part of the interaction mechanism designed. Finally, all meaningful input and output actions and interactions are realized during the product design by considering additional aspects of usage, such as control and customization of the system by its users.

#### **12.4 Example of a model-based co-creation process**

In this section, we present an example to illustrate how this methodology and the abovementioned models can be applied in a real design project. ReHABITAT-ImmoCHECK+, a research project funded by BMK,1 has developed the conceptual basis of a gender and age sensitive set of instruments for illustrating the development potential of vacant or not fully occupied single family detached houses. This potential could be generated by redensification and by fostering innovative forms of living together. Furthermore, this set of instruments permitted an assessment of the houses. On the one hand, it aimed to support persons (the users of the designed product) in a phase of reorientation regarding their living and housing situation; on the other, it provided banks with decision guidance in the granting of credits. UCD was applied throughout this project, which resulted in a satisfactory and successful solution, both for the residents and for surrounding stakeholders. Figure 12.3 shows the intermediary results (use as well as system and interaction models) created in several design workshops with all involved persons in the project: a) 2D bricolage of the house; b) a brick presentation of the house; c) emotional and behavioural expectations from the house; d) the ground-plan of the house as a Lego construction; e) design workshops with users to plan the 'new' house; f) the use perception of the house and its areas; g) the representation of the 'new' house with different use aspects; h) the use context mapping based on the usage of the house; i) the visualization of the usage of the house from the other inhabitants' perspective, which varied in most cases significantly among the inhabitants of the same house; j) the scenario to implement in the technological solution; k) the sketch of an entry point interface; l) a wireframe and low-fi mock-up of the usage representation of the house; m) the prototype of an interactive solution to create the house plan based on usage; n) the final product landing page to enable multiple users to enter data into the system; o) the interface to build a simple representation of the house to facilitate a common understanding between all stakeholders involved. This example shows how the single models build the base for the next step in the process, and how the design evolves over time through the active participation of the stakeholders. Most importantly, it shows the process by demonstrating the different types of models created.

1Federal Ministry Republic of Austria Climate Action, Environment, Energy, Mobility, Innovation and Technology, https://www.bmk.gv.at/en.html

Figure 12.3: *Artifacts created as models during the User-Centered Design process in the project ReHABITAT-ImmoCHECK+*

#### **12.5 Conclusions**

In this paper, we introduced user-centred design as a dynamic multidimensional process utilized by several Design Thinking methods and UCD artifacts. The whole design process is an iterative circle of intertwined factors, namely of people (users, designers, other stakeholders), particular design phases, and artifacts as intermediaries or final results to represent certain design aspects and parameters. The iteration of a UCD process is accompanied with user studies for design and for evaluation, which need different methodological approaches in each design phase. In UCD projects, usability studies have to be seen as integral parts of design processes. This fact makes usability studies to important activities which enable the creation of the products and shape their future use in real settings (Bødker, 2000).

We presented how a UCD process can be established and how a design process can evolve from the very beginning until the definition and presentation of the product design. A careful combination of by now well-established Design Thinking methods makes it possible to design systems for improved user experience. It remains crucial, however, to create the right intermediaries while focusing on users during the design process. The generated design artifacts host implicit knowledge about the target users, their contexts, and all other factors that are relevant for the design process, mainly to make the right design decisions throughout an entire project. Use, system, and interaction models are very powerful artifacts that help to achieve this goal when applied correctly at the right time while designing. There is no strict rule stating that all the methods contributing to create the models presented in Figure 12.2 should be used in any type of design projects. Each project is unique, and designers have to select the most suitable methods for their particular project. This paper only helps to show the possible and useful ways of doing user-centred design.

#### **REFERENCES**


This is a book about musical gestures: multiple ways to design instruments, compose musical performances, analyze sound objects and represent sonic ideas through the central notion of 'gesture'.

The writers share knowledge on major research projects, musical compositions and methodological tools developed among different disciplines, such as sound art, embodied music cognition, human-computer interaction, performative studies and artifcial intelligence. They visualize how similar and compatible are the notions of embodied music cognition and the artistic discourses proposed by musicians working with 'gesture' as their compositional material.

The authors and editors hope to contribute to the ongoing discussion around creative technologies and music, expressive musical interface design, the debate around the use of AI technology in music practice, as well as presenting a new way of thinking about musical instruments, composing and performing with them.

Tomás / Gorbach / Tellioğlu / Kaltenbrunner (Eds.)

**EMBODIED GESTURES**

The artistic research project 'Embodied Gestures' is coordinated by the Tangible Music Lab of the University of Art and Design Linz, and the Artifact-based Computing & User Research unit of the TU Wien. Publishing this book was possible thanks to the funding received from the Austrian Programme for Artsbased Research (FWF PEEK).