## Beyond Media Borders, Volume 2 Intermedial Relations among Multimodal Media

*Edited by* Lars Elleström

### Beyond Media Borders, Volume 2

Lars Elleström Editor

# Beyond Media Borders, Volume 2

Intermedial Relations among Multimodal Media

*Editor* Lars Elleström Department of Film and Literature Linnaeus University Växjö, Sweden

#### ISBN 978-3-030-49682-1 ISBN 978-3-030-49683-8 (eBook) https://doi.org/10.1007/978-3-030-49683-8

© The Editor(s) (if applicable) and The Author(s) 2021. This book is an open access publication.

**Open Access** This book is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence and indicate if changes were made.

The images or other third party material in this book are included in the book's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the book's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specifc statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affliations.

Cover illustration: Jay's photo/gettyimages Cover design: eStudio Calamar

This Palgrave Macmillan imprint is published by the registered company Springer Nature Switzerland AG.

The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

## Foreword: Mediations of Method

As the subtitle of the two volumes of *Beyond Media Borders: Intermedial Relations among Multimodal Media* makes clear, these refections on media have the mission to begin where medium-specifcity or, what I call slightly irreverently, medium-essentialism ends. The media under discussion here, considered from a great variety of perspectives, are all 'multimodal', set in more than one semiotic mode. The most readily understandable example we have rehearsed for so long would be, of course, cinema or television, the study of which in monodisciplinary departments seems to take for granted that they *are* media, whereas the inevitable combination of words and images, colour, sound, narrativity and technological effects clearly demonstrates that no single disciplinary framework will do. As I am also a maker of flms and video, I feel I am in a good place to say this. But as the essays in these volumes make clear, practically all media deploy more than one modality.

The point is not so much, however, that 'multi-' aspect, although that, too, is important, since it advocates an anti-purist view of the media products—Lars Elleström's term for 'texts' and 'images', 'sounds' and 'words', and what have you, that it is the Humanities' mission to study. What catches my eye is primarily that word 'relations', in combination with the preposition 'inter-', which is particularly dear to me, as I have explained more times than I care to remember. Briefy, 'inter-' stands for, or *is*, relation, rather than accumulation. It is to be distinguished in crucial ways from that currently over-used preposition 'trans-', which denotes a passage through, without impact from, another domain. With his consistent interest in media *as* intermedial and his prolifc publication record, many edited volumes, and as director of the Linnaeus University Centre for Intermedial and Multimodal Studies (IMS), Elleström has become a primary authority in that domain that is best characterized as one that doesn't ft any of the traditional disciplinary concepts, yet is probably the largest, most frequently practised mode of communication among humans, indispensable for human life. Elleström's ongoing focus on—his intellectual loyalty to—the idea of the *semiotic*, a concept and feld that on its own already indicates the need for the 'beyond' in the books' main title, demonstrates a resistance to ephemeral academic fashion and a consistency of thought without dogmatism which I consider characteristic of the semiotic perspective. Briefy again, a semiotic perspective asks how we make meaning. The interest of these volumes lies in the importance of communication in general, without which no human society is possible.

Media, as the editor explains, are always-already 'inter-', as the centuryold debates about inter-arts clearly demonstrates. The preposition is a bridge, and the articles brought together here explore what the bridge bridges. This requires refection on the concept of media itself. One cannot understand intermediality without a sense of what a medium is; even if, as such, in its purity, it doesn't exist. With exemplary clarity, Elleström begins his substantial opening and synthesizing article with fve tendencies he fnds damaging for intellectual achievement in (inter)media studies. Anyone interested in this feld of study will recognize these tendencies and agree with the editor's critique of them. But then, the challenge is how to remedy these problems. This is where Elleström earns his authority: he proceeds to announce how these tendencies will be countered, or overcome, in the present volumes. If only all academics would take the time and bother to lay out what they are up against and then redress it: academic bliss would ensue. In other words, this is real progress in the collective thinking of cultural analysis. Felicitously refraining from short defnitions, he embeds the relevant concepts in what he calls a 'model', but what those of us with a mild case of 'model-phobia'—the fear of a certain scientistic demand of rigour before all else—may also see as a theoretical frame. Felicitously, he calls his activity 'circumscribing' rather than defning. His approach alone, then, already demonstrates in the frst pages of his long introductory text an academic position that integrates instead of separating creativity and rigour and thus not only helps us understand the general principles of communication but, through detailed analysis, makes us 'communicationally intelligent', if I may follow discursively the example of psychoanalyst Christopher Bollas who, in his 1992 book *Being*  *a Character*, sensitizes us to the complexities and *thereby*, clarity, of how people are able, and by the media products, enabled, to communicate effectively, with nuance.

There is not a term or concept here that is not both circumscribed and relativized and put to convincing use. The length of the introductory essay is, in this sense, simply a demonstration of generosity. For example, the central concept of 'transfer' that we can hardly do without when talking about communication is neither defned in a simplistic way, as a postal service that goes in one direction only, nor theorized into incomprehensibility. The idea of transferring means that a message goes from a sender to a receiver; we were told in the early days of semiotic theory. Of course, in order to discuss communication, we must consider the idea that a message is indeed transferred from a sender to a receiver; without it, we are foundering. In this, Elleström is realistic; he doesn't reinvent the wheel. Yet, the implicit (but not explicit) notion that the content of a message, as well as its form, go wholesale from sender to receiver, as endorsed in traditional semiotic theory, is clearly untenable. For, the sender's message, with the sender always already 'in' communication, will always be infuenced, or coloured, by what the sender expects, and has reasons to expect, the receiver will wish, grasp, appreciate.

What do we do, then? Instead of casually rejecting the idea, concept or notion, Elleström and his colleagues in these volumes recalibrate and nuance what we consider a message to be, with the help of the relationality that the preposition 'inter-' implies. This makes the sender-messagereceiver process an interaction, mutually responsive, hence, communicative in the true sense. The change from 'sender' to 'producer' intimates that the former sender has *made* something. The former 'receiver' has shed her passivity by becoming a 'perceiver', a term that adds the activity performed at the other end of the process. And when the term 'meaning' is hurt by a long history of rigid semantics, as is the case of many of the concepts we use as if they were just ordinary words, they come up with alternatives, but not without bringing these in 'discussion' with the simpler but problematic predecessors. The need for a concept that cannot be reduced to dictionary defnitions compels the authors, guided by the experienced and ingenious editor, to come up with richer terms that are able to encompass all those nuances that were always a bit bothersome and that we liked to discard or ignore. Thus, 'cognitive import' cannot be reduced to 'meaning', and neither can it be confned to language. That would make the substitution of a well-known term by a new one pointless. Instead, the new term necessarily includes the embodied aspect of communication. This eliminates the mind-body dichotomy to which we are so tenaciously attached; not because we believe in it, but because, until these volumes, we had no alternative vision.

The word 'dichotomy', here, is perhaps the most central opponent in these volumes' discourse. And as with 'inter-' as implying relationality, I feel very close and committed to an approach that does not take binary opposition as its 'normal', standard mode of thinking. And once we are willing to give up on dichotomies such as mind/body, it becomes possible to complicate all those dichotomies that structure what we have taken for granted and should let go in order to recognize the richness of mental life—mental in a way that does not discard the body but endorses it, along with materiality, as integrally participating in the thinking that communication stimulates, helps along and substantiates. Both the partners in communication, who can be singular and, at the same time, plural, and the site of communication, are necessarily material or bound to materiality. Moreover, the sense-based nature of communication makes the abstract ideas surrounding communication theory, not only untenable, but futile, meaningless. Getting rid of, or at the very least, bracketing, binary opposition as a way of thinking is for me the primary merit of the approach presented here.

So, the frst thing these books achieve is to complicate things, in order to get rid of cliché simplicity, and then, right after that, to clarify those complicated ideas, concepts and the models that encompass them. This is perhaps the most important merit of these volumes. They complicate what we thought we knew and clarify what we thought is diffcult. With that move as their starting point, the enormous variety of topics of the chapters become, thanks to the many cross-references from one article to another, a polyphony. Rather than a cacophony of loud divergent voices, this polyphony constitutes a symphony that, as a whole, maps the enormously large feld of the indispensable communication that is human culture, without pedantically demanding that every reader be an expert in all those felds. I don't think anyone can master all the areas presented and examined in the contributions, but the taste of it we get makes us at the very least genuinely interested. This is not a dictionary or an encyclopaedia but a beautifully crafted patchwork of thoughts.

The conceptual travels are stimulating, never off-putting, because they are completely without the plodding idiosyncrasies one so often encounters when new concepts are proposed. And devoid of the polemical discussions with other terminologies, well explained and labelled meaningfully, the conceptual network towards which these books move flls itself as we read along and thus ends up offering a ground for cultural analysis that I am eager to put my feet on. Solid, reliable and, still, exciting. What more would we wish from in-depth academic work? This collective, collaborative work is based on a deep understanding of what scholarly work should be: an act of communication between producers and perceivers, as the view presented here would have it, one that makes its readers feel involved. This is the only way they can learn something new.

University of Amsterdam Amsterdam, The Netherlands Mieke Bal

## Preface

In 2010, Palgrave Macmillan published a volume entitled *Media Borders, Multimodality and Intermediality*, which I had the pleasure to edit. It included my own rather extensive introductory article, 'The Modalities of Media: A Model for Understanding Intermedial Relations', which has since then attracted some attention in intermediality studies. It is my most quoted publication, and scholars and students still use the book and my introductory article in research and education. For my own part, I apply the core concepts of 'The Modalities of Media' as a basis for all my research, including in the two Palgrave Pivot books *Media Transformation* (2014) and *Transmedial Narration* (2019). Over the last decade, however, I have also deepened, developed and slightly modifed the original ideas because I think some of them were formulated prematurely. I have also noted that people sometimes misunderstand certain parts of the article because of my somewhat inadequate and occasionally confusing ways of explaining some of the concepts. Therefore, I decided to rewrite 'The Modalities of Media: A Model for Understanding Intermedial Relations'.

However, the reworking became more substantial than I had expected, resulting in a text that is not only modifed and updated but also signifcantly expanded, incorporating ideas that I have presented in other publications during the last decade. Therefore, I have called it 'The Modalities of Media II: An Expanded Model for Understanding Intermedial Relations'. The new version more clearly frames mediality and intermediality in the context of inter-human communication and defnes the central concept of media product as the intermediate entity that makes communication among human minds possible. It retains, but slightly modifes and expounds, the central idea of characterizing media products in terms of four media modalities, four kinds of media traits. For instance, the discussions now include not only virtual (represented) time and space but also virtual (represented) materialities and sensory perceptions. Providing a fuller picture of representing and represented media traits, as well as adding discussions of cross-modal cognitive capacities of the human mind, makes it possible to offer a much-developed understanding of the concepts of media types and media borders and what it means to cross media borders. As a result, the new article hopefully better explains the intricacies of media integration and media transformation. Overall, most of the concepts have been fne-tuned, leading to a more consistent and developed framework. However, attentive readers will note that I have not mentioned a few ideas that I briefy discussed in the original article. This does not necessarily mean that I have abandoned them; rather, I have decided to develop them further in other publications instead of trying to squeeze even more into an already extensive article. Nevertheless, 'The Modalities of Media II' is supposed to replace rather than complement the original article*.*

This means that the two-volume *Beyond Media Borders: Intermedial Relations among Multimodal Media* is effectively a completely new publication. All of its other contributions are entirely novel compared to *Media Borders, Multimodality and Intermediality* (2010) and are written by authors that (with only one exception) are not the same as those in the earlier book. The main idea of the new publication is not only to launch an updated version of 'The Modalities of Media' but also to present it together with a collection of fresh articles written by scholars from a broad variety of subject areas, united by their references to the concepts originating from 'The Modalities of Media'.

Besides being highly original pieces of scholarship in themselves, the accompanying articles practically illustrate, exemplify and clarify how the concepts developed in 'The Modalities of Media II' can be used for methodical investigation, explanation and interpretation of media traits and media interrelations in a broad selection of old and new media types. To provide space for analysis of such a wide range of dissimilar media types, without reducing the complexity of the arguments, two volumes are required. Their title, *Beyond Media Borders: Intermedial Relations among Multimodal Media*, refects the underlying idea that all media types are more or less multimodal and that comparing media types requires that these multimodal traits being analysed and compared in various ways. As different basic media types have diverging but also partly overlapping modes (for instance, several dissimilar media types have visuality or temporality in common), and because humans have cognitive capacities to partly overbridge modal differences (between, for instance, space and time or vision and hearing), media borders are not defnite; in that sense, one must move 'beyond' media borders.

Overall, the two volumes form a collection with strong internal coherence and abundant cross-references among its contributions (not only to 'The Modalities of Media II'). Simultaneously, they cover and interconnect a comprehensive range of very different media types that scholars have traditionally investigated through more limited, media-specifc concepts. Hence, the two volumes should preferably be read together as a unifed, polyphonic and interdisciplinary contribution to the study of media interrelations.

Växjö, Sweden Lars Elleström

## Acknowledgements

The support and help of my colleagues at the Linnaeus University Centre for Intermedial and Multimodal Studies (IMS) has been invaluable for my work. I am also in debt to all the contributors to these two volumes, including Mieke Bal, who is currently a guest professor at IMS and kindly agreed to write the foreword. Moreover, I am grateful to those esteemed colleagues who acted as peer reviewers: Kamilla Elliott, Anne Gjelsvik, Pentti Haddington, Carey Jewitt, Christina Ljungberg, Jens Schröter, Crispin Thurlow and Jarkko Toikkanen. Finally, I would like to acknowledge the fnancial support from the Åke Wiberg Foundation and IMS, which made it possible to make the publication open access.

## About the Book

Although all of the contributions can be read as separate articles, the two volumes of *Beyond Media Borders* form a whole. Because the contributions are written in concert and include some dialogues, reading the publication in its entirety adds substantial value. Part I in Volume 1, 'The Model', contains the extensive theoretical framework presented in 'The Modalities of Media II: An Expanded Model for Understanding Intermedial relations'. Part II in Volume 2, 'The Model Applied', offers a brief summary and some elaborations that end the two volumes. Between these two opening and closing parts, one fnds Part II in Volume 1, 'Media Integration', and Part I in Volume 2, 'Media Transformation', which contain the majority of contributions. As explained in 'The Modalities of Media II', media integration and media transformation are not absolute properties of media and their interrelations, but rather analytical perspectives. Hence, the division of articles into two parts only refects dominant analytical viewpoints in the various contributions; a closer look at them reveals that they all, to some extent, apply an integrational as well as a transformational perspective.

This is the second volume of *Beyond Media Borders*. The complete table of contents for both volumes is as follows:

#### *Volume 1*

#### **Part I The Model**

1. The Modalities of Media II: An Expanded Model for Understanding Intermedial Relations *Lars Elleström*

#### **Part II Media Integration**


*Volume 2*

#### **Part I Media Transformation**


#### **Part II The Model Applied**

8. Summary and Elaborations *Lars Elleström*

## Contents




## Notes on Contributors

**Jørgen Bruhn** is Professor of Comparative Literature at Linnaeus University, Sweden. His latest books are *The Intermediality of Narrative Literature: Medialities Matter* (Palgrave Macmillan, 2016) and, with Anne Gjelsvik, *Cinema Between Media. An Intermediality Approach* (2018). He is a member of Linnaeus University Centre for Intermedial and Multimodal Studies.

**Ana Cláudia Munari Domingos** holds a PhD in Literary Theory. She is an associate professor at the University of Santa Cruz, Brazil, teaching comparative literature. She leads the working team Intermediality: Literatures, Arts and Media at Anpoll (Associação Nacional de Pós-Graduação e Pesquisa em Letras e Lingüística) and has written *Hiperleitura e escrileitura: Convergência digital, Harry Potter, cultura de fã* (2015).

**Øyvind Eide** works at the Department of Digital Humanities, University of Cologne, Germany. His research interests include modelling the relationships between texts and maps as media of communication, which was also the focus of his monograph entitled *Media Boundaries and Conceptual Modelling: Between Texts and Maps* (Palgrave Macmillan, 2015).

**Lars Elleström** is Professor of Comparative Literature at Linnæus University, Sweden. He presides over the Linnaeus University Centre for Intermedial and Multimodal Studies and chairs the board of the International Society for Intermedial Studies. Elleström has written and edited several books, including *Divine Madness: On Interpreting Literature, Music, and the Visual Arts Ironically* (2002), *Media Borders, Multimodality*  *and Intermediality* (Palgrave Macmillan, 2010), *Media Transformation: The Transfer of Media Characteristics Among Media* (Palgrave Macmillan, 2014), *Transmedial Narration: Narratives and Stories in Different Media* (Palgrave Macmillan, 2019), and *Transmediations: Communication Across Media Borders* (2020). He has also published numerous articles on poetry, intermediality, semiotics, gender, irony, and communication. Elleström's recent publications, starting with the article 'The Modalities of Media: A Model for Understanding Intermedial Relations' (2010), have explored and developed basic semiotic, multimodal, and intermedial concepts aiming at a theoretical model for understanding and analysing interrelations among dissimilar media.

**Liviu Lutas** is Professor of French Literature at Linnaeus University, Sweden. His research interests include metalepsis in different media, narrative questions in flm and literature, intermedial theory, and French literature from the Caribbean. He has published articles such as 'Storyworlds and paradoxical narration' (2015), 'Sur la syllepse narrative—Un concept théorique négligé' (*Poétique,* 2012), and 'Narrative metalepsis in detective fction' (2011). He is a member of the Linnaeus University Centre for Intermedial and Multimodal Studies.

**Kate Newell** is the dean of the School of Liberal Arts at the Savannah College of Art and Design, USA. Her areas of research include adaptation and intersections of literature and visual culture. She is the author of *Expanding Adaptation Networks: From Illustration to Novelization* (Palgrave Macmillan, 2017) and has published essays on the topics of adaptation and illustration as adaptation.

**José Arlei Rodrigues Cardoso** is a journalist and PhD student of linguistics, literature, and media at the University of Santa Cruz, Brazil. He is a comics artist and works professionally on digital design and illustration.

**Zoe Schubert** works at the Department of Digital Humanities, University of Cologne, Germany. She is writing her dissertation about 'Virtual reality as a transformative technology in the humanities—Theatre in VR'. Her research interests include virtual and augmented reality, modelling, media transformations, media culture, theatre, visualizations, and web technologies.

**Mary Simonson** is Associate Professor of Film and Media Studies and Women's Studies at Colgate University, USA. Her scholarship explores performance across media in the nineteenth and early twentieth centuries, and she has published in edited volumes and journals including *Journal of the American Musicological Society*, *Journal of the Society for American Music*, *Screening the Past*, and *Oxford Handbook of Dance and the Popular Screen.* She has also published a book entitled *Body Knowledge: Performance, Intermediality, and American Entertainment, 1907–1917* (2013). Simonson is writing a book on music and voice in American silent flm presentation in the 1920s.

**Miriam Vieira** is a professor at the Department of Letters, Arts and Culture at Universidade Federal de São João del Rei, Brazil. She is a member of the International Society for Intermedial Studies, International Association of Word and Image Studies, and Grupo Intermídia*.* Her recent edited volumes include *Escrita, Som, Imagem: Perspectivas contempora*̑*neas* (2019), with Márcia Arbex and Thaïs Diniz, and a forthcoming second volume. Her main interests include the relations between literature and architecture, ekphrasis, and (teaching) intermediality.

## List of Figures



## Media Transformation

## Finding Meaning in Intermedial Gaps

#### *Mary Simonson*

#### **Contents**


#### 1.1 Introduction

In recent decades, a furry of excellent scholarship has described media and processes of mediation in increasingly expansive terms. Media, much of this work posits, is not only the specifc technologies by which information is disseminated, but also any communicative conduit that conveys ideas or meaning between one place or person and another. As Lars Elleström described in his original "Modalities of media" article and further elucidates in his essay in this volume, for example, media products phenomenon, objects, bodies, and/or extensions of bodies—transfer "cognitive imports" between the "producer's mind," where they

L. Elleström (ed.), *Beyond Media Borders, Volume 2*, https://doi.org/10.1007/978-3-030-49683-8\_1

M. Simonson (\*)

Colgate University, Hamilton, NY, USA e-mail: msimonson@colgate.edu

<sup>©</sup> The Author(s) 2021 3

originated, consciously or unconsciously, and the "perceiver's mind," where they are received bearing some resemblance to their original state (Elleström 2020: 12–13). Elleström's and other scholarship then elaborate and concretize these models with examples: a dancer develops a concept, generates choreography, and uses their body as a medium of display upon and through which audience members see, hear, and feel some version of that original concept. A photographer sees a scene, decides how to frame and compose it, and captures it on flm; the digital photo or print enables viewers to glimpse a version of that original scene. For obvious reasons, these examples almost always involve successful communication: the viewer gets a sense of what is pictured, the photographer deems their captured image acceptable, the audience is able to make sense of the dance, and the choreographer and dancer both feel as if they have shared something with those present.

However, media products—and particularly intermedial products—are not always able to transmit meanings and ideas so smoothly. And at times, media products are envisioned and created precisely so that they will likely not transmit meanings and ideas in a straightforward way. In this essay, I explore a series of historical and contemporary media objects and performances that do not necessarily facilitate a "successful" transfer of meaning, in part because of their intermedial confgurations: a silent flm about (and featuring repeated performances by) a ventriloquist, another that revolves around about an opera singer's stage debut, radio broadcasts of flm programs in the 1920s, and fnally, a modern American theater production that relays a narrative through a series of media practices rooted in historical moments, geographies, and languages distant from most audience members' experiences. Each of these media objects and performances mix, combine, and transpose multiple media. In doing so, each conceals as well as reveals: something is lost or obscured in translation.1 At times, these "losses" are purely accidental, the result of an overly ambitious agenda: aesthetics and/or content associated with one medium are transposed to another that cannot quite contain them. At other times, these losses are the natural byproduct of experimentation and a still-developing understanding of a medium's modalities, possibilities, and limits: a creator represents a media object in a technical medium of display defned by strikingly different modalities to see how both media will be reimagined, and their relationship to one another reconfgured. At still other times, these losses are entirely purposeful: an aesthetic, social, or cultural intervention designed to yield specifc effects. Yet in all cases, these acts of concealment are—or become—meaningful and productive.

That a media product is likely better suited to one technical medium of display than another is hardly a novel concept. Nor is it news that some media products are more easily representable in certain media, nor that media require contextual frameworks and experiences. Indeed, as Elleström notes in his original "Modalities of media" essay, while technical media are generally able to mediate a range of different media, a television show will more "completely" and "successfully" mediate a flm than a painting could, because its modal capacities are far more like those of flm (Elleström 2010: 31). In this essay, I am interested in expanding our understanding of what constitutes "successful" mediation, particularly in the context of intermediality. Numerous scholars have argued that intermediality is defned in part by its resistance to completeness and unity. Irina Rajewsky, for example, describes intermedial references as "as-if" situations: though one medium is treated as if it is another, it never actually becomes another. While that medium may "generate an illusion of another medium's specifc practice," there remains an "intermedial gap" between the attempted enactment or reproduction of the medium and the medium itself (Rajewsky 2005: 55). These gaps and "as ifs" are often made readily visible; indeed, as Peter Boenisch writes, the broadcasting of these "detours, inconsistencies, and contradictions" refuses our easy immersion into a narrative or world and instead draws our attention to the (inter)media at work (Boenisch 2006: 115).2 By creating space for and highlighting these gaps—these moments that withhold as much as they communicate, and that communicate withholding—artists and performers generate a wealth of new expressive possibilities. These new modes of expression, in turn, promise new affective and perceptual experiences for performers and audiences alike. And at times, the creation of moments in which media do not or are not able to communicate smoothly also prompts new aesthetic and technical innovations, reshaping genres and media, and drawing attention in our cultural imagination to media's plasticity and possibility.

#### 1.2 Ventriloquism with No Voice

*The Unholy Three*, a 1925 silent flm directed by Tod Browning, is a suspenseful thriller about an elaborate criminal ruse hatched by a former sideshow ventriloquist, Prof. Echo (Lon Chaney), and two of his colleagues, a little person named Tweedledee (Harry Earles) and a strongman named Hercules (Victor McLaglen).3 Using his vocal mimicry skills and a clever costume, Echo disguises himself as Mrs. O'Grady, the proprietress of a pet shop specializing in talking parrots, and the elderly grandmother of his pickpocket girlfriend, Rosie O'Grady (Mae Busch). "Mrs. O'Grady" uses ventriloquism to convince wealthy customers that the birds in her shop can speak on command. The sales are part of a larger ruse, though. When customers call to complain that their new pets will not talk, Mrs. O'Grady makes house calls; while she convinces the customers that the birds can actually speak, her accomplice Tweedledee, disguised as a napping baby in a pram, is left alone to case the house so the trio can later return and rob it.

The conundrum bound up in depicting this narrative on the silent screen is immediately obvious. Ventriloquism is predicated on aural illusion. The ventriloquist speaks with no movement or other visible sign, throwing their voice in order to vocally animate that which does not speak. Without audible voices, there is no magic: the ventriloquist need not speak with still lips at all, much less throw their voice or disguise their vocal timbre and speech patterns. How to depict ventriloquism and vocal impersonation in the absence of precisely the modalities that defne them? Or, to borrow from Kate Newell's essay in this volume, how to "reframe" ventriloquism to be seen and heard as it moves from stage to the silent screen? (Newell 2020). Previews for the flm puzzled over the question. As a writer for *Picture-Play Magazine* wrote, "when the parrots in his shop can't talk Chaney is supposed to imitate them with his ventriloquial talents, and they are probably having a monkey-and-parrot time of it trying to register this on the screen" (Hollywood Highlights 1925: 114).

The flm's creators employed a variety of narrative, dialogic, and visual cues to represent ventriloquism and vocal disguise. From its opening moments, the flm establishes Echo's identity as a ventriloquist: we meet him performing with a dummy at a sideshow. Though he quickly begins ventriloquizing birds instead, the dummy remains a conspicuous prop throughout the flm, constantly reminding us of his skill. Additionally, the flm deploys physical markers and intertitle dialects to visually convey Echo's ability to successfully alter his voice. When disguised as Mrs. O'Grady, Echo's bent posture, relatively still lips, gently tilted head, and relaxed cheeks and chin suggest a quiet, gentle voice; her dialogue that appears in intertitles is correspondingly mannered and mild. When Echo breaks his Mrs. O'Grady character or appears out of disguise, however, his face becomes far more animated, his eyebrows raising and lowering for emphasis as his speech emanates from the corner of his slightly distorted mouth. These physical hints are registered in the intertitle dialogue with coarser expressions, slang, and contractions. Frustrated at Rosie's firting with the naïve and besotted pet store clerk, Hector (Matt Moore), for example, Echo rips off his Mrs. O'Grady wig, juts his head forward, and wags his fnger in her face, disgustedly exclaiming via an intertitle, "You're makin' a play for that guy!" The flm also draws our attention to the sound of voices through offhanded diegetic comments. In one of the early pet store scenes, for example, Mrs. O'Grady tries to shoo Rosie away from Hector by remarking on the effect of Rosie's voice on "baby Willie" (Tweedledee in his disguise), who fusses on the foor nearby. "Little Willie darling heard your voice," the older woman tells the younger. "He wants you to take him bye-bye." Neither these explicit references to character's voices, nor the careful visualization of speech through intertitles and facial expression are accidental: both tacitly encourage audiences to imagine the sounds of these different voices.

As the reviewer quoted above predicted, some of the most striking attempts to visualize Echo's ventriloquism utilize the parrots. During the very frst scene in the pet store, Hector comments on Mrs. O'Grady's special ability to make the birds talk. When a customer enters and begins looking at the birds, Hector immediately runs to summon Mrs. O'Grady, telling her (via intertitle), "I think you'd better show the parrots. They talk so well for you." As Mrs. O'Grady obliges, the action is interrupted by an intertitle reminding the audience of Echo's ventriloquial profciencies and likening the parrots to Echo's dummy: "Echo, the ventriloquist, could make wooden dummies talk—now watch the jolly parrots!" The flm immediately cuts to a shot of Mrs. O'Grady, surrounded by caged and loose parrots; as each bird "speaks," speech bubbles appear over their heads containing intertitles depicting the bird's words: "Good morning, Aunty!" "Hello Polly!" "Pretty Lady!" (see Fig. 1.1). The shot cuts to Hector, who again draws attention to Mrs. O'Grady's magical ability to make the birds talk when he raves to Rosie, "Isn't it wonderful how the parrots always talk to your Grandma?"

Echo's ventriloquism is a crucial plot device—the mechanism by which the trio's crimes are facilitated. However, it is also central to the flm's climactic scene, and its narrative resolution. Increasingly uncomfortable with the trio's activities, Rosie falls in love with Hector. When the trio botches a robbery, Echo frames Hector and then kidnaps Rosie in hopes of reigniting their former passion. But he cannot bear to see Rosie so miserable, and sneaks into Hector's trial in hopes of exonerating him through a fnal ventriloquism performance. Echo manages to slip Hector a note instructing him to take the stand and silently repeat the Lord's Prayer over and over, moving his lips without actually making a sound. As Hector

**Fig. 1.1** Mrs. O'Grady and her "talking" parrots in *The Unholy Three* (dir. Tod Browning, 1925). All rights reserved

follows the note's instructions, the flm's audience is treated to yet another visual strategy for representing ventriloquism on the silent screen: the camera oscillates between close-ups of Echo in the gallery, his throat and jaw moving slightly as he envoices Hector; frst person intertitles exonerating Hector; and shots of Hector obediently moving his lips. Seconds later, Hector draws attention to Echo's ventriloquism when he declares, "That wasn't me talking. I didn't say a word." In this moment, ventriloquism is portrayed by revealing—and commenting on—its mechanics.

While the scene punctures the ventriloquial illusion for the flm's audience, the performance goes off without a hitch within the flm's diegesis. Neither the judge nor either of the lawyers acknowledge Hector's claim that he has just been envoiced—an act that presumably amounts to perjury. Nor do any of them seem to doubt the contents of his ventriloquized testimony. However, even this diegetically "successful" ventriloquial performance is not enough to clear Hector's name. The judge immediately instructs the jury to disregard Hector's testimony on the grounds that there are no witnesses to confrm this new story, and sends them out to deliberate. Narrative closure comes only when Echo intervenes and makes a full confession. In the end, ventriloquism can only get so far on the silent screen.

As was common practice, a talkie remake of *The Unholy Three* (directed by Jack Conway) was released by MGM in 1930, shortly after the widespread conversion to sound flm. The remake retained the silent flm's narrative conceit and characters, and both Lon Chaney and Harry Earles reprised their roles alongside an otherwise new cast. It also retained ventriloquism as a drawing card, central plot device, and Echo's defning talent. Advertisements and previews for the new *Unholy Three* touted the imminent logic of making a sound version of this particular flm. A writer for *Film Daily* noted that the story was "a bigger draw in sound, for so many of the dramatic incidents depend on audible effects, such as the ventriloquist's dummy," and advised exhibitors to highlight Chaney's "varied speaking voices" in their pitches to audiences (The Unholy Three 1930b: 10). With sound, not only could audiences marvel at Chaney's vocal impersonations, they could also hear his voice magically emerge from his dummy's fapping mouth as his own lips remained remarkably still.

However, though sound flm includes an aural channel, it is in some ways no more appropriate for the display of ventriloquial acts than silent flm: it does not replicate the modalities that defne ventriloquism and vocal impersonation, either. Ironically, it is because sound flm is itself a ventriloquial medium that it cannot convincingly represent ventriloquism.4 Watching sound flms, audiences are invited—indeed, compelled—to pretend that the sound they hear issues directly from the image that they see on screen. In reality, though, these synchronized images and sounds are usually the result of careful construction, re-recording, dubbing, and extensive editing. As flm sound scholar Michel Chion notes, the materiality of flm requires "an incision or cut between body and voice" that is then covered over enabling us "to believe, implicitly or explicitly, that the body and voice cohere in some self-evident, natural way" (Chion 1999 [1982]: 125–126). By 1930, MGM was certainly using such techniques; and thanks in part to decades of cinematic "attractions" that openly displayed all sorts of editing tricks, many audience members attending *The Unholy Three* were savvy enough to understand that Lon Chaney's ventriloquial stunts and vocal disguises could easily have been generated through technical means.

Readily anticipating audience skepticism, MGM staged a publicity stunt to reassure audiences that it was actually Chaney's voice they were hearing, and not a cast of vocal extras dubbed in: Chaney issued a signed affdavit that read, in part, "The ventriloquist's, the old woman's, the dummy's, the parrot's, and the girl's are actual reproductions of my own voice and in no place in said photoplay or in any of the various characters portrayed by me in said photoplay was a 'double' or substitute used for my voice." (Inside stuff—Pictures 1930: 49; The Unholy Three advertisement 1930: 26). The affdavit was widely reported in flm trade press, fan magazines, and daily newspapers, and was reproduced in advertisements for the flm. Notably, however, the affdavit—and the press surrounding it—blatantly sidesteps larger questions about the flm's representations of ventriloquism. The affdavit pointedly does not claim that Chaney's vocal impersonations were synchronously recorded with the flm's image track, nor that he ventriloquized his dummy or the parrots live on set. While his vocal impersonation skills were validated, then, little was done to convince audiences of the validity of his ventriloquial performances.

Strikingly, the most convincing display of vocal trickery in the talkie version of *The Unholy Three* is achieved through failure. One of the few moments in the talkie that signifcantly departs from the silent version comes in the fnal courtroom scene. Instead of ventriloquizing Hector from the audience, Echo actually takes the witness stand in his Mrs. O'Grady disguise. As he testifes, his vocal disguise slips: Mrs. O'Grady's voice descends a bit, then a bit more, and eventually is so low that the defense attorney approaches her and rips off her wig, revealing the whole ruse. Unmasked, Echo confesses and is found guilty and sentenced. While the silent version ends with the failure of ventriloquism, the sound version culminates in actual ventriloquial failure. The authenticity of Echo's vocal trickery is confrmed by showcasing its dissolve.

The silent and sound versions of *The Unholy Three* make a counterintuitive proposition: to represent ventriloquism using a medium—flm—that at best obscures and at worst undermines the illusion that defnes the art. While these flms develop strategies to do so, the fact remains that silent and sound flms both lack the modal confguration to "completely" mediate a ventriloquial performance—or, to put it differently, ventriloquism exceeds the modal capacities of the flm projector and screen. Despite this modal mismatch, though, the two *Unholy Three* flms are hardly failures. Lon Chaney was famed for the range of characters he convincingly embodied: reviewers and fans marveled at his transformation from one flm to the next. Playing a ventriloquist offered Chaney the opportunity to embody radically different characters within the same flm, and at times, within the same scene. Reviews and commentary on the silent flm highlight Chaney's marked visual transformations as he shifted from Echo to Mrs. O'Grady, as well as his expressive abilities within each of the roles (The Shadow Stage 1925; The Unholy Three 1925a: 45; The Unholy Three 1925b: 845). Describing the flm's fnal moments, a writer for *Exhibitor's Trade Review* noted that though Echo lightly bid Rosie goodbye "with a laugh that is akin to mockery" after his confession, within seconds of her departure, "his seamed face is limned with the searing lines of heartbreak" (The Unholy Three 1925a: 45). Similarly, the sound version enabled audiences to hear Chaney's voice for the frst time and treated them to Chaney's vocal impersonation abilities. Long nicknamed "the man of one thousand faces," the sound version of *The Unholy Three* led the press to embrace a new moniker: "the man of one thousand voices" (Nelson 1930: 32; Moak 1930: 44). While Chaney's transformations had in the past hinged on costumes, makeup, and his acting abilities, *The Unholy Three* flms frst implied and then revealed that his disguises and transformations were audible, as well: he could fake convincing laughter, shift his vocal timbre, and alter his speech patterns. Here, Chaney dazzled with his ability to shift in a moment from one emotion to the next, one voice to another, one person to a different person.

Shoehorned into the flm's narrative, ventriloquism also prompted innovation within the cinematic medium and the flm industry more broadly. The creators of the silent *Unholy Three* explored how the cinematic medium might visualize vocal production and illusion, as well as the acts of hearing and mishearing; in doing so, they reconfgured and reimagined silent cinema's sensorial modalities. Some of the strategies used to represent ventriloquism—intertitles, moving lips and expressive faces, close-ups and short/reverse shot sequences—align with silent flm's usual techniques for conveying speech. Other strategies might themselves be considered ventriloquial: speech bubble text and oscillating throats speak "for" and replace these time-tested techniques.5 In the sound remake of *The Unholy Three*, too, Chaney's ventriloquist character prompted the flm's creators, MGM, and even the press to negotiate and consider how best to acknowledge and address audience awareness of sound cinema's mechanical and technical underpinnings—precisely in order to enable audiences to suspend their disbelief or even disavow flm's status as a highly constructed medium. The intermediality of both the silent and sound versions of *The Unholy Three*, then, generates successful medial experimentation, recombinations, and fuidity, beginning to reimagine, challenge, and construct the defning modalities and medial limits of silent and sound flm.

#### 1.3 Silent Film with No Image

In the frst years of the 1920s, radio, which had been limited to military use during World War I, emerged as a highly popular and much debated technology. Amateur radio operators and new commercial radio experiments fooded the airwaves, broadcasting a range of different program formats, content, and musical styles. Among those who embraced and helped to shape the emergent medium were flm studios and exhibitors: radio was viewed as an ideal venue for advertising new pictures, a sort of live fan magazine by which movie stars could directly address their adoring audiences, and a potential source for flm accompaniment—if not a means of making flms "talk."

While many in the industry eagerly explored radio's possible applications, Samuel "Roxy" Rothafel, then the manager of the Capitol Theater in New York City, was particularly visionary. In 1922, he began broadcasting concerts by the Capitol Theater orchestra over WEAF, and hosting shows that promoted the theater's flm programs and featured performances by a rotating cast of orchestra members and the other performers who appeared in theater presentations before and between flms. By April of 1923, Roxy's radio broadcasts were synchronized with his evening theater program—or rather, as media scholar Ross Melnick has argued, the Capitol's evening theater program was dictated by the broadcast schedule (Melnick 2012: 212).6 Period articles in *Radio Digest* and other trade publications reported that at 7:20 pm, Roxy would introduce the evening's broadcast from near the Capitol Theater's stage (but out of ear- and eye-shot of the audience in the theater), sharing details about the orchestra's musical selections and other things they would hear in the two hours ahead (Lanzius 1923: 5). Then, at 7:30 pm, the radio audience would be invited to listen along with the theater audience to the orchestral overture that traditionally opened silent flm presentations in movie palaces. But radio listeners were not just allowed to listen in on the theater's musical performances. As the newsreel and other short flms appeared on the Capitol's screen following the overture, the radio audience heard Roxy's careful descriptions of each flm's narrative and visual imagery over the accompanying music. Then, as live performers appeared on the theater's stage between the short subjects and the evening's feature, Roxy introduced each and, when necessary, described precisely what the theater's audience was seeing, offering audiences mental images to compliment the musical performances and other acts that they heard through the ether. Shortly after 8:00 pm, as the feature flm began, Roxy and his "Gang" of performers moved to the theater's broadcasting studio and continued their broadcast with further conversation and additional musical performances until 9:00 pm.

At frst glance, these radio broadcasts, and particularly the segments broadcasted live from the theater's stage, seem as counterintuitive as a silent flm about ventriloquism. Here too, media objects—silent flms and live stage performances—were being mediated by a technical medium of display—radio—with signifcantly different modal characteristics. It is tempting to read these broadcasts as a case of remediation gone wrong: the nascent radio industry's attempt to take up and incorporate the "older" medium of silent flm in order to draw legitimacy from and ultimately displace it.7 And certainly, the broadcasts followed the remedial logic of claiming (and benefting from) the status of silent flm and the artists who work in it, much as silent flm did with opera, as is discussed in the next section (Bolter and Grusin 1999: 69). Broadcasts of silent flm presentations borrowed and represented silent flm's familiar entertainment formats, compelling ready-made narratives, and well-prepared musical and stage performances. They also attempted to invoke, if not create, the atmosphere of the silent flm theater in listeners' living rooms. As *Radio Digest* writer Evelyn Lanzius reported in 1923, "One evening recently, previous to the rendering of 'Love's Old Sweet Song' by 'Betsy,' Mr. Rothafel requested his Radio audience to lower their lights, thus bringing his own lighting effects into their homes. Many wrote in to say that they had followed this suggestion with telling effect" (Lanzius 1923: 5).

Even as these radio broadcasts remediated silent flm, they also distinguished themselves from it. On air, Roxy sought to create an intimacy that was impossible at flm screenings in his enormous Capitol Theater. His patter was familiar, his descriptions accessible and casual. He presented himself as a personal friend of his listeners, and encouraged the musicians and performers who joined him in the studio to do the same. He regularly introduced and referred to his musical guests by frst names and nicknames: conductor William Axt was called Billy on air, singer Betsy Ayres was simply Betsy, and violinist and concertmaster Eugene Ormandy was often presented as "The Blue Blonde." The featured performers chatted casually with the listening public, generating both public personae and imagined relationships that were nearly impossible in a darkened flm theater. As Lanzius wrote in *Radio Digest*, "Roxie [*sic*] introduces each artist most informally, and encourages them to talk in little personalities to the invisible audience, these being mostly in humorous vein, and there is no doubt that there is an unusual sympathy and bond of interest between the artists of the Capitol broadcasting studio and the Radio listeners" (Lanzius 1923: 5). While radio lacked the visual dimension so central to silent flm and the live performances presented alongside, its modalities and the social and cultural practices that surrounded it enabled radio to exceed silent flm in meaningful ways.

It is important to remember, however, that Rothafel's broadcasts were driven as much by Rothafel himself—a fxture within silent flm exhibition—as they were by those in the radio industry. Even if those at WEAF and its parent company hoped that the venture might legitimize and allow radio to compete within the entertainment landscape, and some within the flm industry engaged in a fair amount of hand-wringing over radio's potential to decrease flmgoing, Rothafel and other exhibitors who broadcast from their theaters clearly thought of the practice as a means of renewing and expanding silent flm's cultural relevance. Period audiences, too, seem to have understood that listening to silent flm via radio was profoundly different than attending a flm presentation at the theater; that is, the technical medium of display profoundly changed the media product (Elleström 2020: 33–40). Indeed, audiences in the 1920s likely used the broadcasts as a means to imagine the original media objects and performances.8

Rothafel's broadcasts hardly shied away from radio's inability to display the very images that constituted the flm—this intermedial gap. Instead, they highlighted the gap by describing the images cast onscreen in great detail. As he narrated the flm shorts and stage action, Rothafel was telling audiences exactly what they were missing, exactly how their experience differed from that of the audiences in his theater, exactly what they were being denied by their radio. As Evelyn Lanzius reported, Rothafel was well aware that flm's "chief form of entertainment, the picture, cannot be transmitted through the air—yet—and if people who hear the Radio concert like them in Radio form they must actually attend the theater to get the picture and their interest is greatly stimulated by the Radio presentation" (Lanzius 1923: 5). Nor was it only the flm images that were leveraged to help audiences visualize the show, and to draw them into the theater. After audiences began writing to the Capitol asking for photos of "Roxy and his Gang," postcards featuring portraits of various performers in the broadcast studio were produced and mailed to fans; by June of 1923, it was reported that thousands of requests were in the process of being flled (Lanzius 1923: 5). Similarly, magazine writers were invited to visit the studio during broadcasts, resulting in articles that Melnick argues offered fans yet another "visually corollary to the disembodied voices audiences had grown accustomed to" (Melnick 2012: 219). I would go a step further: the postcard portraits and colorful descriptions that the writers penned following their visits displayed the broadcasts in other media photographs and magazine articles—with yet other sets of modalities to compliment the radio broadcast's intimate aural characteristics. Together, these media enabled audiences to better imagine the sights, smells, feel, and energy that accompanied the voices they heard over the air. This intermedial "migration" of stage and screen performance to radio, then, not only bolstered the stage and screen performances themselves, but also prompted additional intermedial layers and movement. The resultant intermedial web of still and moving images, print journalism and fan letters, bodies on stage, and voices over the air traversed and defned not only media but time and space as well.

#### 1.4 Opera with No Song

In the fall of 1920, Universal released the silent flm *Once to Every Woman*, directed by Allen Holubar and starring Dorothy Phillips as Aurora Meredith, a blacksmith's daughter who also happens to possess a beautiful singing voice.9 The flm joined a host of other silent flms that showcased opera. While many of these flms adapted operatic narratives for the screen—there were silent versions of Bizet's *Carmen*, Mascagni's *Cavalleria Rusticana*, and a host of others—others actually wove opera into the plot.10 *Once to Every Woman* is lost, but period reviews and advertisements for the flm and its presentation at various theaters make clear that Holubar's production did both. Early in the flm, a wealthy patron discovers Aurora singing in her village church's choir and sends her to Europe for training, where she makes a successful operatic debut. Eventually, Aurora returns to the U.S. to appear as Marguerite in Charles Gounod's opera *Faust*, and in hopes of distancing herself from an unwanted Italian admirer. As Aurora appears onstage in the opera (in the flm), however, it is revealed that her Italian admirer has also crossed the Atlantic, and is seated in the theater's balcony with a gun. As she sings, he shoots her and then kills himself, interrupting the opera and Aurora's American debut, and throwing the theater into chaos. Eventually, Aurora recovers physically, but the trauma of the violence leaves her unable to sing. Deserted by her European friends and fans, her career all but over, she returns to her parents' home alone. Even here, she fnds no solace: her mother becomes ill and dies shortly after her homecoming. In a melodramatic twist, however, this second trauma restores her singing voice: in her mourning, Aurora begins to sing again.

Numerous scholars have theorized the popularity of flmic adaptations of opera in the silent era: opera plots were familiar, compelling, and easily adaptable to the screen, and opera carried the sort of artistic value and cultural prestige that the early flm industry sought. Moreover, operatic adaptations relieved theater musicians and music directors of the timeconsuming work of composing a musical score or compiling cue sheets from an endless selection of existing music. Instead, they could cobble together instrumental versions of familiar arias, choruses, and incidental music from a single opera score. For audience members familiar with the adapted operas, the resultant medleys were enjoyable, even when rendered solely by theater pianists, organists, or instrumental ensembles. Yet these sorts of flms—particularly those like *Once to Every Woman*, which actually showcased opera singer characters and their performances—presented a fascinating challenge, too. How to represent opera *singing* without audible voices? Like the Lon Chaney's silent and sound screen ventriloquism and Roxy's Capitol Theater broadcasts, *Once to Every Woman* offered audiences a representation of one medium in another medium that lacked the necessary modal qualities.

Exhibitors were clearly sensitive to this modal dissonance, and some developed strategies to remedy it, including the introduction of live performers into the flm's presentations. When *Once to Every Woman* was screened at the Stratford Theater in Chicago in December of 1920, for example, the flm projection was paused during the scene in which Aurora appears onstage as Marguerite, and a soprano dressed in a costume modeled on Aurora's stepped into the spotlight on the Stratford stage in front of the screen and sang Marguerite's "Jewel Song" from *Faust*. As she concluded, the flm projection resumed and the soprano disappeared into the darkness. One flm trade journal reported that the effect was seamless: "the impression conveyed to the audience was that Miss Phillips had stepped out of the screen to sing the aria" (Chicago Theatre's presentation elaborate 1921: 367). It was also undoubtedly dramatic: while the flmic medium could offer audiences Aurora's beautiful voice only as a mythical, unheard object, the soprano's appearance onstage transformed the silenced into the sounding, the mythic into the realized.

The Stratford staff developed an equally compelling strategy for restoring Aurora's voice in the theater when her mother's death restores it onscreen. As Aurora opens her mouth to express her grief in the flm, the soprano, hidden offstage, began singing the popular nineteenth-century song "Ben Bolt," which remembers a woman who is now buried in a church cemetery. Once again, the mythical voice became manifest: that which audiences believed to be unavailable frst due to the modalities of the silent flm, and then due to the flm's narrative, became audible. Additionally, this (re-)envoicing of Aurora aligned the flm's story and its presentation: those in the theater experienced precisely the vocal loss and return that was depicted onscreen.11

For flm audiences today, who rarely experience flms accompanied by live musicians or bookended with stage performances, these interludes might seem like temporal and medial interruptions. However, the review quoted above suggests that audiences in 1921 probably did not perceive these moments as ruptures at all. Rather, the soprano's frst performances were understood as part and parcel of the flm's presentation—indeed, as part and parcel of the flm itself. Or, to apply Andy Lavender's notion of performer as medium from this volume, the soprano, "herself a complex technical medium," operates within the larger media product of the silent flm (Lavender 2020: 117). *Once to Every Woman*'s intermedial play with opera was, like the examples above, an "as-if" situation. Though momentarily masquerading as opera, the silent flm never actually became opera; in fact, even experiencing it "as" opera required audiences to engage in a signifcant suspension of disbelief. Yet together with Stratford Theater staff and performers, those audience members who were willing to embrace the "as-if" reimagined and remade *Once to Every Woman* to allow for audible voices. In doing so, they also temporarily reimagined and remade the technical medium of display of silent flm, enlarging the audience's conception of its modal boundaries and qualities to more fully accommodate the media product.

#### 1.5 Speaking in Another Language

In the historical examples discussed above, media producers developed compelling reasons and strategies to represent or mediate performances using technical media of display that would inevitably withhold, remove, truncate, and radically reconfgure those performances. My fnal example is a bit different. *Club Diamond* is a theater piece in which creators Saori Tsukada and Nikki Appino (with composer Tim Fain) consciously incorporate—indeed, thematize—the sort of medial withholding, truncation, gaps, and "as ifs" that the historical examples stumble into. A deeply complex work, *Club Diamond* uses intermedial layers and combinations to purposefully "other" the audience, confronting them with the unfamiliar (and at times, that which seems unintelligible) narratively, medially, and structurally. By doing so, the work invites audience members to explore the affective experience of its protagonist(s), calling upon them to traverse gaps, piece things together, and compensate, just as the work's protagonist(s) must.

The piece opens unassumingly. Tsukada stands center stage dressed in modern clothes, a large flm screen behind her and a podium and clothes rack to her left, and cues the lights and an audio recorded monologue. Almost immediately, identities begin to blur. Is the frst-person monologue about a childhood love of Disney and drawing Tsukada's autobiography or that of a character she is embodying—and why is it recorded? Tim Fain enters with his violin, only deepening the uncertainty: is he accompanist or character? Is this prologue or performance? As the recorded monologue ends, the pleasant haze seems to clear, momentarily: Fain begins tuning and Tsukada puts on a costume, declaring, "My friend Tim and I are going to tell you a story about how I came to the United States." Within seconds, though, her voice echoes as the recorded monologue resumes, reintroducing her, or rather, reintroducing her in her newly costumed role: that of a prominent Japanese silent flm narrator, or *benshi*, in 1927.12 Tsukada-as-*benshi* takes her place behind the podium as the flm screen fickers alive and Fain begins to accompany the images on screen, but again, the performers and their roles are quickly complicated. Fain lowers his violin mid-phrase, revealing that he is playing (is he playing?) over and with prerecorded layers of music, and onscreen, the flm's title appears: it too is called *Club Diamond*, and it too credits Nikki Appino as director (in the theater production, she is credited as a creator), Saori Tsukada as the principal performer (here, she is credited as "Heroine," "New Girl," and "Bartender," while in the theater production program identifes her as "performer" and creator), and Tim Fain as composer, along with a feet of others.13 Is the screen simply a mirror of the stage, or has *Club Diamond* migrated from one medium to another? (See Fig. 1.2.)

As the flm begins to roll, it is immediately revealed that it is neither. Onstage, Tsukada-as-*benshi* describes the flm images as they fash, drawing attention to particular elements, providing sound effects to deepen moods and clarify actions, and offering extensive commentary on the various characters and their activities—including, of course, those of Tsukadaas-Heroine and her other screen roles. Yet he does so almost exclusively in Japanese, despite the fact that the work premiered in New York City for a predominantly English-speaking language, and has thus far been performed for audiences mainly of English speakers. Recognizable place names and English expressions occasionally creep into the *benshi*'s narration, but Japanese dominates, forcing audiences into a space of disorientation, even incomprehension. English-speaking audience members must draw clues from the images onscreen, the context in which the *benshi*

**Fig. 1.2** *Benshi* (Saori Tsukada) and Musician (Tim Fain) in *Club Diamond*. (Photo by Kathryn Raines, courtesy of Nikki Appino and Saori Tsukada)

speaks, and his expressive face and gestures. As he dramatizes the images that fash on the screen in front of us, the *benshi* (to borrow from Andrea Virginás 2020: 152–159) dramatizes and prolongs the act of perceiving and interpreting the flm, guiding audience reactions before they fully understand what they are seeing.14

Shot in black and white, the flm's palette, pace, and aesthetics convincingly pass as a 1920s production. The flm includes the sort of experimental elements, visual trickery, and cinematic "attractions" common in silent cinema. Structured in ten chapters, the flm opens in Japan, where the Heroine is working as a flm actress. Almost immediately, she is rejected in favor of a nearly identical replacement. Gutted, she is crying on a bench when a small bird—a symbol of opportunity in Japanese culture, an intertitle teaches us, and a recurrent symbol in the piece—lands near her, inspiring her to strike out for a new life in the Hollywood. After a long boat journey flled with fashbacks to the Heroine's rejection, she arrives in New York City. A series of montages, shadowy shots, and rapid cuts suggest the disorientation of the newly arrived immigrant: the Heroine's purse is stolen and the police pay her no mind, the streets and subway become a maze, crowds push and jostle. Through a series of intertitles, New Yorkers instruct and reprimand her: "What do you want, lady?"; "I DON'T SPEAK CHINESE!"; "Come on, just get on the bus!" The English titles do little to quell the anxious, overwhelming mood created by the flm's quick cuts, its fast, noisy musical accompaniment, and the *benshi*'s rapid, audible breaths and quavering, high-pitched stuttering as he inhabits the Heroine. As the narrative unfolds, one wonders: is this a version of Tsukada's own coming-to-America story, the story she promised to tell? Is Tsukada-as-Heroine an avatar for Tsukada herself? Or is such an assumption both impatient and essentialist?

By Chapter Five, the Heroine has found her way to a hotel, but it offers neither relief nor a sense of safety. It is dark and as she traverses the hallway, hand-drawn spider webs appear along the walls. She cries herself to sleep and awakens to the sound of the bird's wings as it alights atop a copy of *Variety*, where she fnds a wanted ad seeking "Oriental Girls" to work at Club Diamond. At the club, she is met by Tsukada-as-Bartender, bossa nova-inspired music, and a male boss who takes advantage of her in her dressing room between performances. The flm quickly turns nightmarish: the *benshi*'s narration becomes increasingly frantic and Fain's ascending melodies become louder and stormier as our Heroine dreams a montage of images and moments: her boss's cigar, his hand on her leg, toasting glasses, the doorway to her hotel, static and glitches, large black birds, dancing parasols, more glitches, then a black screen. "Chapter Nine: Revolution" fashes onscreen, but the flm immediately begins to decay, frames and sprocketed edges rushing past and a flm leader's countdown numbers obscuring shots of the Heroine before the image disappears completely and projection ceases. The fnal chapter promised in the flm's opening titles is excised: any semblance of a storyline explodes, as does the silent flm industry and the *benshi*'s cultural status and career, as "talking flms" revolutionize the global entertainment scene. Just as the recorded monologue that began the work was punctured by Tsukada's declaration that she and Fain would tell us a story, and this statement was punctured by Tsukada's sudden transformation into the *benshi* character, now the *benshi*'s performance and the medium of silent flm are dramatically interrupted, leaving a bewildering, gaping fssure.

As Tsukada-as-Heroine vanishes from view, Tsukada's recorded monologue resumes. The last time the audience heard this voice, it was introducing the *benshi* to us. Here, it continues narrating his story, describing the death of a career and lifelong love: "You want to ride it out. You try to ride it out. But suddenly, you become […] ancient." Tsukada-as-*benshi* removes his stage outft slowly, inhabiting the rhetorical "you" of the monologue more fully with each movement. But the monologue pivots, and Tsukada's voice describes the desperation that drove her to answer a newspaper ad titled "looking for girls," recounting an experience working as an escort that parallels Tsukada-as-Heroine's and further collapses the two characters into one, despite the century gap.

Meanwhile, in the corner of the stage, Tsukada has transformed from *benshi* into *kamishibaiya*, a type of itinerant street performer taken up by many *benshi* after the introduction of talkies, who entertained children with candy and paper plays in postwar occupied Japan.15 In white coveralls, a hat, and goggles, Tsukada-as-*kamishibaiya* mounts a bicycle with a large wooden box (*butai*) on the back, and rides unsteadily to center stage, radio blaring. He parks and sets his stage, opening the *butai* to reveal a candy display and a set of illustrated storyboards displayed in a frame, and then begins his performance. An informal street act from the back of a bicycle for Japanese and American expat children, the *kamishibaiya*'s performance lacks the polish and ceremony of the *benshi*'s. Yet the audience is encouraged to understand the ragtag street performer as the *benshi* reinvented. Tsukada-as-*kamishibaiya* retells the same story that Tsukada-as*benshi* did earlier, narrating the heartbroken Japanese flm star's trip to New York, her frantic navigation of the unfamiliar city, and the rundown hotel and club where she fnds herself living and working. Though the *kamishibaiya* uses a few more English words and phrases than he had as *benshi*, the narration remains largely in Japanese, and is performed with the same dreaminess and deep emotional investment. Even the images featured on the storyboards harken back to the silent flm: several are exact replicas of illustrations from the flm. This mirroring of *kamishibaiya* and *benshi* hints at the continuity of another pair that, though separated by time, are increasingly diffcult to tease apart: our flm heroine and the Tsukada character who emerges in the recorded monologue.

The paper play ends as abruptly, and as traumatically, as the flm—and at exactly the same point in the narrative. The Heroine's work at the club is introduced with a bright red card with swoops of yellow that resemble stage curtains and the same bossa nova tune that accompanied the club scenes in the flm, and suddenly, the *kamishibaiya* abandons his narration to move about the stage, stomping his feet and beating a shaker against his palm percussively. As narrator becomes musician, musician becomes dancer: Fain moves increasingly extravagantly to his own violin music and the *kamishibaiya*'s rhythms. It is a musical duet and *pas de deux* in one, the violinist seemingly drawn into the *kamishibaiya*'s postwar Japanese world, into his paper play, into Club Diamond. The musician and *kamishibaiya* whirl in synchronization, crouching and then unfurling their bodies to stand upright only to bend again, more quickly, more frantically, as the music grows increasingly disjoined and then suddenly, stops. The two pull apart and the *kamishibaiya* resumes his narration, quiet and still. Then, a fourish: he tosses a handful of confetti and, as Fain begins a pentatonic *pizzicato* passage, removes the red storyboard to reveal a green card covered with white confetti. Instead of quickly hiding the previous board, though, he gracefully fips it overhead, revealing a muted image of the rising sun fag before depositing the board behind the others. It is a feeting but potent symbol that hints at the *kamishibaiya*'s worldview and orients the entire scene in time and space, differentiating this telling of the Club Diamond tale from the earlier silent flm. Tension grows as the *kamishibaiya* continues on, mimicking the heroine's panicked high voice and beating a drum attached to his bicycle with increasing power until he bends from the waist, gasping and crying—an embodied rendition of the heroine's bad dream. He straightens, covers the storyboards, and abruptly dismisses the crowd: "The end. Everybody go home."

Unexpectedly, though, the *kamishibaiya* does not exit. Instead, accompanied by a haunting ascending violin melody, he wheels his bicycle to the right side of the darkened stage, where he pulls open the reverse side of the *butai* to reveal a built-in flm projector. Squatting next to it, he extends the projector's lens forward, fips on the lamp, and begins rotating the bicycle's pedals to power the projector. An intertitle identical to those in the *Club Diamond* silent flm fashes onscreen, introducing the fnal, unseen segment of the flm: "Chapter Ten." A Universal Studios International News Reel titled "Oriental Cutie makes it big in Hollywood!" begins to roll, presenting Tsukada-as-Heroine blowing kisses to her adoring fans. The newsreel is a talkie, with a jaunty orchestral march accompanying the opening action and then voiceover narration regaling us with tales of this "new born" flm star: the fnal chapter has fnally arrived, and it has arrived in sound. The flm captures a fnal shot of Tsukada-as-Heroine outdoors, dressed in white; the camera lingers on her face, and then a "The End" title fashes onscreen, with a subtitle added a few seconds later: "Made in the USA." Tsukada-as-*kamishibaiya* slows the bike pedals as the image fickers out. He rises, removes his goggles and hat; suddenly, it is Tsukada who stands before us blinking, then licks her lips and begins a live monologue about the drawings that she would make and fax to her parents in Japan to show them her new New York life, and her father's advice that she keep talking, even imperfectly, because Americans do not like silence. Delivered quietly and haltingly, the monologue stitches together the childhood reminiscences from the frst recorded monologue, her fraught adjustment to New York City detailed in the second recorded monologue, the *kamishibai* drawings, the fate of the *benshi* in our culture of sound and talk, and the linguistic and social gulfs between Japanese and U.S. culture that the *benshi*, Tsukada-as-Heroine, the *kamishibaiya* entertaining postwar children, and Tsukada herself all face. In this fnal, subdued moment, Tsukada's various characters collapse into one another even more profoundly, across time and space, gender and career, identity. Onstage, Tsukada points upward, cueing the lights.

Brilliantly coherent as it moves from stage to screen to paper play and back, *Club Diamond* nonetheless generates the same sorts of medial and narrative gaps, fssures, and disorientation as my earlier examples. In addition to the overlapping, intertwined characters that lead audiences to constantly question who is speaking, for whom, and whose stories we are witnessing, *Club Diamond* positions English-speaking audiences as linguistic "others," thematizing the disorientation and partial understanding so familiar to immigrants and travelers.16 Creating the piece, Appino and Tsukada meticulously constructed the *benshi* and *kamishibaiya*'s speeches such that one word, perfectly hit, might carry an entire paragraph, and generated visual cues to highlight the importance of visuality and looking when navigating linguistically inaccessible spaces. Nonetheless, these sections withhold as much as they reveal, prompting viewers to constantly question what and whether they understand, and to settle uneasily into a space of not quite knowing.

For most audience members, the fgures of the *benshi* and *kamishibaiya* are as unfamiliar as the language of their narration. Indeed, the *benshi* tradition itself is a uniquely Japanese practice of which relatively few Americans are aware, and even fewer are familiar. For that matter, though silent flm has its enthusiasts, it likely seems quaint and distant to modern audiences. *Kamishibai* performances are even more mysterious in an American context. Despite the ubiquity of these performers in postwar Japan, the medium has no real American corollary. While the *kamishibaiya* and *benshi* may offer historical, geographical, and political clues to some audience members, then, many more are likely to experience them as lacking contextual qualifcation, to borrow a useful term from Elleström (2020: 60–64): removed from their historical, social, and cultural contexts and represented in a contemporary work of the theater, the *benshi* and *kamishibaiya* lose a bit of their intelligibility. Yet the use of contextually unqualifed media is precisely what enables *Club Diamond* to generate gaps in which audience members must founder, compensate, piece together, and reorient.

The gaps and uncertainty built into *Club Diamond* come with costs. Linguistic unfamiliarity makes it easy to miss narrative action. Audiences less familiar with twentieth-century politics, media history, and performance practices can easily fnd misunderstand when or where various sections of the work are set. Important details can slip by without registering, particularly while audience members work to piece other details together in their minds. Even as a music and media scholar who studies silent flm and performance, I overlooked a number of compelling moments in my frst several interactions with the work—and none of them were particularly nuanced or veiled. In a Skype conversation with me (17 July 2019), Tsukada noted that younger audience members, in particular, seem to miss visual and historical cues, in large part because they do not have the historical framework through which to "read" the work's characters and action. As a result, she and Appino continue to consider if and how they should make certain elements—such as the political and cultural context in which the *kamishibaiya* performs, and the political overtones that make their way into his performance when, for example, he displays the rising sun on the reverse of a storyboard—more legible to audiences who lack the historical understanding to situate him and his actions.

The pleasure—and ultimately the legibility—of *Club Diamond* depends in large part on the intermedial web it constructs. Tsukada's immigration story is told in a series of live and recorded monologues; a second, slightly abstracted version of that story is told on flm and in a paper play. Each time the narrative is cut off, a medial transposition is forced; each medial transposition offers audiences a new set of modalities through which to experience and interpret this story. Tsukada's second monologue, for instance, translates moving images into spoken words, adding precision and detail. Similarly, the simple images of Tsukada-as-*kamishibaiya*'s paper play highlight and distill themes easily lost in the silent flm's elaborate moving images or the pace of the monologues. With each medial transposition, too, layers of signifcance accrue around the work's recurrent symbols, and the audience is resituated in space and time, moving from the 1920s and 1930s to the late 1940s and 1950s and on to the present, and between Japan and New York.

*Club Diamond* showcases the continuity and synergies not only of the narratives told within these various forms and genres, but also of the forms and genres themselves. The piece narrates the evolution of silent flm into sound flm, and more subtly suggests the emergence of *kamishibai* from the ashes of the silent flm industry. By virtue of casting and narrative, the *benshi* and *kamishibaiya* are also inseparable from the silent flm starlet, the contemporary immigrant actress and Tsukada herself. Moreover, aesthetics and modal qualities associated with one medium are frequently deployed or at least invoked in other media in the piece. With the repeated interruptions and truncation of the narrative, for example, Appino and Tsukada invoke a characteristic of kamishibai paper plays: kamishibai artists regularly omitted the fnal scenes of their stories in order to encourage repeat "customers." The technique is equally effective here: the refusal of narrative closure builds anticipation such that the *kamishibaiya*'s quiet, private viewing of "Chapter 10" becomes a strikingly beautiful and profound moment, a fulfllment of the work's unspoken promise to its audience. Similarly, for English-speaking audiences, the *benshi* and *kamishibaiya*'s Japanese narrations come to function much like silent flm musical accompaniment (including Fain's within the work): they establish mood, guide emotional registrar, and offer hints about action more than they legibly narrate the action of the flm or paper play. Indeed, it is frequently diffcult to tell which sound effects are generated by the *benshi* and *kamishibaiya*, and which come from Fain's violin and prerecorded material; when Fain and Tsukada-as-*kamishibaiya* begin to move and make music in synchronization, their sonic, dancing bodies transcend simple boundaries between music, dance, and narrative. The creators of *Club Diamond* use intermediality to generate in audiences precisely the feeling of disorientation that Tsukada and her alter egos describe, enact, and embody. Yet the gaps, fssures, and sensations that result also reveal much more: the limits and possibilities of performance, the intimate connections between seemingly disparate media, and the aesthetic and narrative promise of medial transposition and borrowing.

#### 1.6 Conclusion

*Club Diamond*, Roxy's broadcasts, *Once to Every Woman*, and *The Unholy Three* do not convey meanings and ideas as expected. As such, they remind us that mediation is not only about seamless and easy transmission of sensations and ideas across space, time, and bodies. Mediation can also prompt experimentation within a media product and/or a technical medium of display, as *The Unholy Three* does, or make tangible the unique possibilities and intermedial potential of multiple media, as in Roxy's Capitol Theater broadcasts. Mediation can push producers and perceivers to reimagine and differently experience medial boundaries, as in the Stratford's presentation of *Once to Every Woman*. Finally, mediation can purposefully and productively refuse smooth transmission in order to generate particular perceptual effects and bodily affects, as in *Club Diamond*. In each of the examples described above, and innumerable others, intermediality foregrounds various media's modal limits and mismatches, their expressive incompatibilities and shortcomings. At the same time, intermediality opens up new expressive felds and modes of representation, and stages unexpected perceptual possibilities. In his essay on media and theater, Peter Boenisch notes that intermediality is not just about transmission of messages, but also about activating audiences, inviting them into complex imagined worlds; it is as much a perceptual effect generated in performance as an inherent quality (Boenisch 2006: 114–115). Perhaps, then, intermediality is most potently generated in performances that challenge—and at times confound—the audience's expectations and understandings of media.

#### Notes


16. To be clear, I am not suggesting that media require language to communicate; I agree with Elleström's identifcation of the problematic nature of such assumptions in his essay in this volume (Elleström 2020: 6). Rather, I am pointing out that it is striking that *Club Diamond* incorporates language as much to obscure as to transmit. Indeed, *Club Diamond* displays precisely media's ability to communicate despite language, as well as through language.

#### References


———. 2020. The Modalities of Media II: An Expanded Model for Understanding Intermedial Relations. In *Beyond Media Borders: Intermedial Relations among Multimodal Media, Volume 1*, ed. Lars Elleström, 3–91. Basingstoke: Palgrave Macmillan.

Gerow, Aaron. 2010. *Visions of Japanese Modernity: Articulations of Cinema, Nation, and Spectatorship, 1895–1925*. Berkeley: University of California Press.

Gomery, Douglas. 2005. *The Coming of Sound: A History*. New York: Routledge.


Inside stuff—Pictures. 1930. *Variety*, July 9: 49.


*The Unholy Three* Advertisement. 1930. *Film Daily*, 19 July: 26.

Universal Arranges Song Tie-Up. 1920. *Motion Picture News*, 11 December: 4446.

Virginás, Andrea. 2020. Electronic Screens in Film Diegesis: Modality Modes and Qualifying Aspects of a Formation Enhanced by the Post-Digital Era. In *Beyond Media Borders: Intermedial Relations among Multimodal Media, Volume 1*, ed. Lars Elleström, 141–173. Basingstoke: Palgrave Macmillan.

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/ by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## Transferring Handmaids: Iconography, Adaptation, and Intermediality

*Kate Newell*

#### **Contents**


#### 2.1 Introduction

All around us, depictions of red-cloaked, white-bonneted female silhouettes proliferate. An image of the forty red topiaries that lined the east colonnade of the White House in the 2018 Christmas season, edited to appear topped with white "Handmaid" bonnets, went viral on Twitter in November 2018. A June 9, 2019 Associated Press article entitled

K. Newell (\*)

© The Author(s) 2021 33

L. Elleström (ed.), *Beyond Media Borders, Volume 2*, https://doi.org/10.1007/978-3-030-49683-8\_2

Savannah College of Art and Design, Savannah, GA, USA e-mail: knewell@scad.edu

"Scientists feel chill of crackdown on research" is accompanied by a May 23, 2017 photograph of women donning red cloaks and white bonnets at a protest against Texas SB8, a bill which bans second trimester abortion, and requires that fetal remains be buried or cremated (Neergaard et al. 2019). A lingerie website stocked a "Brave Red Maiden" costume "featuring a red mini dress, a matching cloak with attached hood, and a white bonnet headpiece" (Gupta 2018).1

While many viewers recognize this iconography from Margaret Atwood's novel, *The Handmaid's Tale* (1985) or the Hulu series (2017–) of the same name, a recognition of the source does not explain how the red cloak and white bonnet have come to signify in contexts as diverse as a topiary display, threats to scientifc research, and lingerie. In the thirtyfve years since its publication, *The Handmaid's Tale* has been adapted numerous times in a range of media. Reasons for its longevity are many: the novel's popularity and status as a classic work in dystopian literature make it ripe for adaptation, as does the fact that it has already been adapted, as works that have been adapted one or more times seem to be prime candidates for future adaptation. Another, perhaps more telling reason behind the continued adaptation is a general feeling that the novel's themes refect the political dis-ease of a particular moment. Conversations related to any adaptation of *The Handmaid's Tale*, regardless of the decade, are likely to mention the contemporary political climate in the United States, and elsewhere.2 This sense of sustained relevance means that *The Handmaid's Tale* is almost always read through and against broad contextual lenses at the same time that the novel becomes the point of reference for that context.

This essay examines the intermedial transfer of Handmaid iconography across platforms and contexts, and the mechanisms that facilitate such movement. I begin with a consideration of the intermedial network established within Atwood's novel to show that, even prior to adaptation, the Handmaid is understood as a product of intermedial transfer. I next survey the movement of Handmaid iconography in a variety of seemingly static print-based and dynamic motion-based media, and then turn to articulations of Handmaid iconography in more generalized spheres. The image of the Handmaid transfers through processes of adaptation that interpret visual markers in distinct modalities, each of which emphasizes particular traits or characteristics over others. The emphasis or disclosure that characterizes each iteration is accompanied by concealment; that is, as an adaptation foregrounds a particular modality, it simultaneously represses another. In the context of *The Handmaid's Tale*, the tension between disclosing and concealing, between visibility and invisibility operates thematically and in terms of its foregrounded media.

#### 2.2 Intermediality, Iconography, and Transfer

There are many ways to think about intermediality, iconography, and transfer. My own view of intermediality aligns with that described by Irina O. Rajewsky in her contribution to the essay collection *Media Borders, Multimodality, and Intermodality*: "Generally speaking, and according to common understanding, 'intermediality' refers to relations between media, to medial interactions and interferences. Hence, 'intermediality' can be said to serve frst and foremost as a fexible generic term 'that can be applied, in a broad sense, to *any* phenomenon involving more than one medium' and thus any phenomenon that—as indicated by the prefx *inter*—in some way takes place *between* media. Accordingly, the crossing of media borders has been defned as a founding category of intermediality" (Rajewsky 2010: 51–52).3 Rajewsky proposes viewing borders as sites of possibility, "as enabling structures, as spaces in which we can test and experiment with a plethora of different strategies" (65). The broadness of this conceptualization, which I fnd welcome, may frustrate others who approach intermediality more narrowly or who would prefer a more circumscribed set of terms that delineate media borders and avenues of "crossing." Yet, as Rajewsky, Lars Elleström, and other contributors to this volume illuminate, such borders, while useful theoretically, are always constructed and perceptual. That is, no material 'border" exists between, say, the animated and live-action segments of a particular flm, yet audiences perceive aesthetic differences, and articulate that difference in terms of juncture and border crossing.

Throughout this essay, I refer to the movement or transfer of iconography from one media product to another. Here, I use "iconography" to refer to a dynamic and fuid collection of visual symbolic information that signifes to perceivers a distinct "message"—be that message a particular cultural object or other form of communication. By "transfer," I mean the generalized perception that messaging associated with one place "moves" to another.4 That which is transferred may be iconography or theme or tone or narrative or character or compositional principle, or any number of identifable aspects. The perception that transfer has occurred does not mean that the thing no longer exists in the previous location (or previous locations before that). My concept of transfer is informed by a model of communication put forward by Elleström in "The modalities of media II: An expanded model for understanding intermedial relations" (2020). Elleström identifes "three indispensable and interconnected entities" necessary for communication to occur between a "producer's mind" and a "perceiver's mind" (2020: 10, 12): "1. Something being transferred"; "2. Two separate places between which the transfer occurs"; "3. An intermediate stage that makes the transfer possible" (2020: 10).

All media products are communicated by way of their materiality, and, likewise, the perception of transfer manifests in materiality. Elleström's model identifes "four kinds of basic traits," or "media modalities"; these include the material, spatiotemporal, sensorial, and semiotic modalities. All media products are realized through some materiality (e.g., fat, raised, solid), and experienced spatiotemporally (e.g., two-dimensional, threedimensional, sequential, non-sequential) and sensorially (e.g., audial, visual). The semiotic modality is "derived from" the other three modalities. That is, the perceiver takes in material, spatiotemporal, and sensorial information and interprets that information as signifying (2020: 20). Important to the process of communication is the manner by which perceivers understand their particular experience of a given message as compared to and within the context of previous perceptions and communications—what Charles S. Peirce called "collateral experience" (Elleström 2020: 26–27). In the course of examining the movement of Handmaid iconography, I show how each adaptation foregrounds certain modalities to lead perceivers toward particular interpretations of the communication transfer.

#### 2.3 Mediation in *The Handmaid's Tale* (1985)

"What's fascinating about the legacy of *The Handmaid's Tale*," Natalie Zutter writes, "is how it's spread to almost every medium: reimagined on stage and screen, buzzing on the airwaves and between your ears, inked earnestly onto skin and snarkily onto protest signs, embodied in real bodies through viral marketing and political action. […] you can see Offred's story from tech conferences to the Senate foor" (2017). Zutter's verb choices—"reimagined," "buzzing," "inked," "embodied"—identify multiple avenues of mediation that point both to communication transfer (e.g., how particular messages are produced, transferred, and perceived) and medial transfer (e.g., how *The Handmaid's Tale* is adapted from novel to television, radio, and commercial and political arenas). This aspect of *The Handmaid's Tale*'s legacy is not as surprising as it might seem at frst glance. Atwood incorporates into the novel numerous medial references, both within the "tale" and as an organizing tool for the narrative conceit. Such moments indicate a keen awareness of how media and mediation function in concert to construct and convey meaning, as well as how shifts in media impact perception. The "Historical Notes" section, which appears at the end of the "tale" proper, purports to provide "a partial transcript" of a "live" event—the 2195 conference proceedings of the Twelfth Symposium on Gileadean Studies, in which Professor James Darcy Pieixoto conveys orally the context of the transcribed "The Handmaid's Tale" in a talk entitled, "Problems of Authentication in Reference to *The Handmaid's Tale*" (Atwood 1985: 312). In this turn, readers realize the conceit: the text that they thought they had been reading as a fctional novel is presented as both an excerpt from conference proceedings as well as the supplement on which those proceedings are based.

Professor Pieixoto's comments alert his audience to the elaborate network of transmedial transfer that has informed his presentation. The Handmaid's "tale" had been invisible, recorded onto some thirty cassette tapes over existent musical recordings. At the time of the discovery, cassette tape technology had long been obsolete. However, Professor Pieixoto and his colleague, Professor Wade, have "with the aid of [their] excellent resident antiquarian technician, reconstructed a machine capable of playing such tapes" (313). For Pieixoto and Wade, the concern in transferring an oral account to audio recording and then to transcript and then to oral presentation rests in the authenticity of the medial relationships and crossings. Pieixoto states that "we were assured by the experts who examined [the tapes] that the physical objects themselves are genuine. Certainly, the recording itself, that is, the superimposition of voice upon music tape, could not have been done within the past hundred and ffty years" (315). For these scholars, authenticity of medium and authenticity of narrative are yoked—the narrative's authenticity comes into being through the processes of adaptation and transmediation that allow the narrative to be communicated. Visibility and invisibility factor into their establishment and assessment of authenticity in that the superimposition of one media type (oral speech) on another (music) results in the necessary concealment of one message to reveal another.

Atwood prepares her readers for such challenges of mediation within the "tale" proper. We know, for example, that the narrator's job "in the time before," prior to the instatement of the government of Gilead, was in a library, "transferring books to computer discs, to cut down on storage space and replacement costs, they said." The narrator relates that, "After the books were transferred they were supposed to go to the shredder, but sometimes I took them home with me. I liked the feel of them, and the look" (182). Here, the act of transfer renders one technology (print, in this case) obsolete; once content is transferred, the form is destroyed. The narrator, however, appears less interested in the content or semiotic value of the media product, and, instead, fetishizes its sensorial and material qualities—she "like[s] the feel of them, and the look." The narrator serves as the intermediate stage in the communication process—she facilitates the movement of communication from one place to another, and can also redirect that communication. In bringing a book home, she initiates a new communication and expands the intermedial network. The hypothetical book now exists in the library's records of material objects, as content transferred to disc, and as an aesthetic object in the narrator's home. The book has not abandoned any of these communication points, but exists in all simultaneously.

While the verb "transfer" suggests a seamless activity, a "move from one place to another," distinctions in defnitions of transfer highlight a process that is considerably more fraught. Transfer means to "copy (data, music, etc.) from one medium or device to another." Transfer can also mean, "make over the possession of (property, a right, or a responsibility) to someone else" (Transfer 2019). Each of these defnitions signifes differently in *The Handmaid's Tale*, particularly as the narrator herself experiences "transfer"—from freedom to bondage, from the Rachel and Leah Re-education Center (Red Center) to an assignment as a Handmaid, from the home of one Commander to that of another, from knowing to unknowing. Each of these defnitions signifes differently in *The Handmaid's Tale*'s adaptation history, as well. Much in the way that the transfer of speech to cassette and of recording to paper involves shifts in medial interface and reframing of what is seen and emphasized, transfer of, say, a novel from its physical, dimensional book materiality to a physical, dimensional disc materiality involves a reframing—a reframing of how that novel is read (paper vs. screen, page turn vs. scroll) and understood as a sensory object (smell, touch)—and a tacit understanding that all materialities are prone to obsolescence.

#### 2.4 The Handmaid: Establishing a Network of Iconography

Early in the novel, Atwood's narrator describes the features of her Handmaid uniform:

I get up out of the chair, advance my feet into the sunlight, in their red shoes, fat-heeled to save the spine and not for dancing. The red gloves are lying on the bed. I pick them up, pull them onto my hands, fnger by fnger. Everything except the wings around my face is red: the colour of blood, which defnes us. The skirt is ankle-length, full, gathered to a fat yoke that extends over the breasts, the sleeves are full. The white wings too are prescribed issue; they are to keep us from seeing, but also from being seen. (1985: 18)5

This description demonstrates to readers the boundaries between "Handmaid" and "not Handmaid," as well as that between "individual" and "Handmaid" through statements that establish both the narrator's awareness of her subject position (i.e., "I") and the object position of Handmaid (e.g., "their red shoes," "the breasts"). The specifcs of the uniform cannot be separated from its task of creating distanciation between self and role, and of rendering Handmaids both highly visible and invisible. As the narrator, called Offred within Gilead, learns during her reeducation process at the Red Center, "[m]odesty is invisibility […].To be seen—to be *seen*—is to be […] penetrated" (39). The paradox in this statement is that the Handmaid's job is to be penetrated, monthly, by a Gilead Commander, in the context of the government-ordained Ceremony. The danger is in being "seen" as an individual, in a perceiver acknowledging the boundary between Handmaid and individual. "Modesty," as defned within Gilead, protects that boundary. The uniform renders all Handmaids more or less anonymous visual equivalents (as reinforced in the narrator's blurring of plural/singular usage in describing her movements with a fellow Handmaid: "Doubled, I walk the street" [33]). The red uniform transforms the women who wear it into symbols, both of lifebringing possibilities, and of sexual threat and promiscuity. In explaining a character's coldness toward her, the narrator explains, "the frown isn't personal: it's the red dress she disapproves of, and what it stands for" (19). There are certain exceptions to the Handmaids' anonymity, of course. Pregnant Handmaids are made visible, individualized by their swollen bellies, which distend the front of the red dress, altering its shape. Handmaids are also visible to each other: they see each other as distinct women wearing the uniform, rather than as "Handmaid." Such instances of recognition draw attention to the constructed nature of these individual and thematic borders.

Atwood's descriptions of the uniform are not quite ekphrastic, but offer enough detail to convey to readers a clear image of the uniform, while remaining general enough to invite fexibility in interpretation. Fred Marcellino's jacket illustration for the frst American edition of *The Handmaid's Tale*, the 1986 Jonathan Cape edition, marks the frst transfer of the iconic Handmaid uniform from written, verbal description to a twodimensional visual illustration, materialized through color, shape, line, typography, and other attributes of visual design. Marcellino's illustration depicts two Handmaids walking along a tall, gray brick wall. The emphasis is on the vivid red cloak and distinctive winged bonnet. The image's "bird's eye" perspective and juxtaposition of the Handmaids against the insurmountable wall evoke the dystopic society of the city of Gilead imagined in Atwood's novel, as does the anonymity of the fgures. Even here, though, the image explores the boundary between the individual and the collective. The Handmaids are slightly staggered in the illustration, one moving away from the other, who has turned to look behind her—a gesture that imbues the image with a sense of expectation, and suggests subtly differences in motivations of individual Handmaids. The particular moment singled out by Marcellino is not based on a specifc moment from Atwood's novel, but appears to be a generalized summation of the novel's numerous scenes of walking and looking. The particular shape and color saturation of the Handmaids' cloaks, as well as the upturn of the wings on the white headdress align with the narrator's description, but are also the product of Marcellino's transfer of visual verbal description into visual iconic signifers.

Marcellino's interpretation initiates a method of visually articulating the Handmaid in book jacket design. A tour of *The Handmaid's Tale* book covers shows Marcellino's iconography transferring from edition to edition through a repetition of basic thematic and visual coding. For example, a 1998 McClelland and Stewart edition likewise features a tall gray brick wall, curving at roughly the same arc as that in Marcellino's image, and adopts also a bird's eye perspective, but depicts the Handmaid abstractly. The center of the image features a piece of red cloth blowing in the wind, bent into the shape of a Handmaid's cloak. The effect, then, is that of an escape, of the Handmaid fying up and over the wall. Other examples include the 1995 Virago Press cover, which features two Handmaids walking along a low wall and row of tall, spike-formed topiaries. The women are "doubled," to borrow Atwood's narrator's description, and walk with heads bent and baskets clasped.

As the iconography of the Handmaid transfers from edition to edition, the effort to imbue it with secondary symbolism becomes evident, particularly symbolism that underscores an aspect of the Handmaids' oppression (such as an inability to speak, as is emphasized in the cover of the 1986 Seal Books edition, which depicts a Handmaid with her lips bound by metal rings) or that suggests freedom. For example, the impressionistic quality of a 1996 Vintage, alternative paperback edition cover obscures the Handmaid's identity, while the saturation of red heightens the fgure's visibility. The warm color palette coupled with the modeling technique suggest a melting of the Handmaid's face—alluding both to her anonymity (one Handmaid "melts" into the next), as well as the violence suffered by Handmaids at the will of the Aunts and others. A Vintage cover for a 2007 edition depicts the Handmaid in silhouette, kneeling before a fruit tree, staring upward beseechingly. Her gloved hands contort into a shape reminiscent of shadow-cast birds, and two white birds fy from her torso visual extensions of the white bonnet. This cover incorporates into its design a crescent moon shape associated with the Red Center as well as the biohazard symbol, thus alluding to both the Handmaids' role in propagation and the punishment for women who "fail" at that role. The tree is heavy with pome fruit, underscoring the Handmaid's reproductive potential, though three pieces have fallen to the ground, suggesting unrealized opportunities.

While several covers focus on the singular Handmaid, others strive to represent graphically the uniformity of the Handmaids. The 2010 Vintage Classics edition, for example, presents a row of faceless, identical screenprinted Handmaids, reminiscent of mannequins or robots, that rare books' specialist Rebecca Romney compares to *Doctor Who's* Cybermen (2017). The 2017 Vintage edition designed by Suzanne Dean, featuring an illustration by Noma Bar, represents the Handmaid graphically on the front cover, signaling the Handmaid through the outline of a full red dress and white bonnet, and, again, on the back cover, through a keyhole-shaped representation of a Handmaid, which is then repeated in a pattern in the edition's front and end papers. A 2015 translated Georgian edition likewise utilizes the familiar graphic signifcation of the Handmaid via the silhouette of the red dress and white bonnet, and incorporates doubled fgures against a gray wall. However, the doubling is of a Handmaid and a Wife, rather than two Handmaids. The silhouette of the wife appears behind the Handmaid, signifed by blue dress and short veil. She holds a baby, coded red, that remains attached to the Handmaid via the umbilical cord, also red. The brick wall is incorporated as the shadow cast by the Handmaid, and her fgure is visually segmented, divided at the chest and waist and mid-skirt. Through visual echoing, this image highlights the distinct functions of the Handmaids and the wives, while simultaneously alluding to the uniformity of those roles.

If we think of a book cover as offering a visual shorthand of a book's most essential information, the continued repetition and reinterpretation of the red cloak and white bonnet reinforces these features as essential signifers of both the fgure of Handmaid and *The Handmaid's Tale*. As I have discussed elsewhere, the repetition of specifc iconography through successive adaptations across multiple media and platforms prioritizes that iconography as essential to and as a visual equivalent of the media object being adapted (Newell 2017). Rather than grant the Handmaid a discernable identity, the majority of book covers reinforce the anonymity of the Handmaid and, in so doing, establish in visual terms the condition of the uniform described by Atwood's narrator—to render the Handmaid both unseen and highly visible. As the image of the Handmaid transfers from edition to edition, the iconography of the costume gains semiotic weight. The visual iconic characteristics that emerge from the book covers initiate and refect characteristics central to adaptations of *The Handmaid's Tale* in other media, including other print-based formats such as illustrated novels and graphic novels.

#### 2.5 Print-Based Movement: Illustration and Graphic Novel

Book covers, book illustrations, and graphic novels share similar medial properties in that both are heavily semiotic and apprehended as twodimensional, fat surfaces and they can share similar compositional and technical features and concerns, though they differ in degrees of iconicity and spatial-temporality. Book cover designs, as we have seen, are highly iconic and non-sequential—that is, they attempt to distill a book's themes and tone into a single image, or may function as a "teaser," a hint to spark readers' curiosity about a book's contents. Book illustrations, on the other hand, typically adapt specifc moments from a work, and invite readers to consider their semiotic import more directly in concert with the prose. We might think of the majority of book illustrations, particularly those published by major publishing houses and geared toward a mainstream market, as offering a limited sequentiality, in that most depict a succession of scenes from a novel that, when pieced together, convey an intentional trajectory or other system of cohesion. Graphic novels are fully sequential in that they communicate content visually through a series of intentionally linked visuals and frames. I now turn to consider the transfer of Handmaid iconography in Anna and Elena Balbusso's illustrations for the 2012 Folio Society edition of *The Handmaid's Tale* and Renée Nault's 2019 graphic novel adaptation, *The Handmaid's Tale*.

The Balbussos utilize the established iconography of the red cloak and white bonnet, and explore, through composition and pictorial emphasis, the various signifcations of the uniform. The Balbussos transfer the color coding and primacy of the Handmaid from previous iterations through, they explain, the stylistic lenses of "Futurism, Russian Constructivism and fascist-period design"—styles that add a layer of emotional detachment and impersonalization to the Handmaid iconography and subtly underscore the novel's social and political tensions (*The Handmaid's Tale* 2019). Much like the 2010 Vintage cover design that presents a row of identical screen-printed Handmaid uniforms, the Balbusso cover focuses on the uniform: the Handmaid fgure is depersonalized, rendered as a mannequin. Four hands are positioned on the body of the mannequin as well as entering into the seemingly hollow lower-half of the torso, which symbolizes the "empty vessel" to which Handmaids are expected to aspire. The putty-colored rectangle framing the fgure is suggestive of a book cover— "the Handmaid's Tale" within *The Handmaid's Tale*—but its canting likewise evokes the imbalance of the Handmaid within her environment. In keeping with the novel's themes and those of other visual interpretations, this suite of six illustrations and a frontispiece explores tensions between Handmaids' invisibility and heightened visibility. The Balbussos render fgures within Gilead uniformly, in a manner that reinforces the depersonalization of this stratifed society. All Handmaids, Marthas, Wives, daughters, Guardians, and so on look alike and are rarely distinguished visually. Yet the moments that the Balbussos select to adapt trend toward the personal or interpersonal. In this way, they draw attention to the oppressiveness of Gileadean frameworks and explore tensions between concealing and revealing characters' "true" selves in opposition to that façade.

Illustrations have the ability to direct vision, to render visible aspects of a narrative that might be downplayed by a writer or concealed intentionally. Handmaids are fetishized throughout Atwood's novel for all they represent—fertility, sexuality, hegemonic power structures, and so on. The Folio Society edition includes an illustration of the narrator's visit to the doctor's offce, titled "I take off my clothes, behind the screen, and leave them folded on the chair." This image depicts the narrator behind a red accordion screen in the doctor's offce, unclothed save for her white bonnet and red stockings. This image is challenging in its positioning of the viewer and its messaging regarding the Offred. The novel makes clear that Handmaids remain clothed at all times, and that, even during the Ceremony, the body is still largely covered to help dissociate desire and other emotions from the act, to keep the Handmaid from being "penetrated." Seen in the context of the suite's emphasis on uniformity, this image may provide a rare glimpse of individuality—the person distinct from the uniform. As the Balbusso Twins explain in an interview with Porter Anderson for *Publishing Perspectives*, "The Handmaids seem clones. Their faces are anonymous, they're women emptied of their personalities, they've lost their identity […]. When the Handmaid takes off her red uniform, the body re-emerges with its carnality and identity" (Anderson 2017). Yet the image also spectacularizes Offred through positioning and framing: the viewer is invited to see behind the privacy screen and to observe Offred in a private moment of adjusting her stockings—a gesture fetishized in visual culture.

As is the case with many illustrated novels, the Folio Society's illustrations are juxtaposed against prose different from their ostensible subject a circumstance that invites and facilitates signifcations and thematic refections otherwise unavailable. Within the Folio edition, "I take off my clothes" is juxtaposed not against the scene in the doctor's offce, but against that in which the narrator is given a bath to prepare for the Ceremony: "I take off the rest of the clothes, the overdress, the white shift and petticoat, the red stockings, the loose cotton pantaloons." The narrator comments that, "My nakedness is strange to me already. My body seems outdated. Did I really wear bathing suits, at the beach? I did, without thought, among men, without caring that my legs, my arms, my thighs and back were on display, could be seen. *Shameful*, *immodest*" (Atwood 2012: 62). These two moments of undressing—for the doctor and for the bath—evoke in Offred feelings of alienation and distanciation, of heightened visibility and objectifcation as they precede instances of physical violation—that of the doctor's exam (and proposition of sex) and that of the Ceremony. The narrator states: "I avoid looking down at my body, not so much because it's shameful or immodest but because I don't want to see it. I don't want to look at something that determines me so completely" (63). The Balbusso image, then, invites the reader to refect on an aspect of Offred that she, herself, avoids. That the body visible in this image is so distinct from the uniformed body presented in other illustrations may invite readers to also feel the strangeness of Offred's nakedness, and to refect on which—uniform or individual—is the more determining or determined body. At a basic level, however, the image invites the reader to objectify Offred, to view her as a body. The viewer's vision is thus aligned with that of Gilead's dominant power structure, but differently—she is presented to us in posture that is coded openly as private and sexualized, whereas, within Gilead, she is coded as public and not sexualized.

In 2019, *The Handmaid's Tale* was adapted into a graphic novel by Renée Nault. Though the materiality of illustrated novels and graphic novels is similar, differences in compositional arrangement and in the proportion of word and images cause them to be perceived as distinct media. Commonly, the ratio of words to images in illustrated novels favors words, whereas graphic novels emphasize images, and tend to use the combination of words and images to convey meaning. Nault utilizes the iconography of the Handmaid uniform to explore thematic tensions between Handmaids' visibility and invisibility, often to dramatic effect through her play with proportion, color saturation, and other graphic devices. The compositional conventions of the graphic novel allow the uniform to appear as "in motion"—moving through frames—and this movement increases the range of signifcations, as, for example, in the scene of the Particicution, Gilead's communal execution event, which unfolds in a series of increasingly sharp and skewed panes in which the crowd of Handmaids blurs into a mob and the red of their cloaks becomes indistinguishable from the red used to convey their rage and the blood of the victim. Nault uses the uniform symbolically, depicting it as blending into or standing out from the background to establish or reinforce themes or tensions. Nault's interpretation draws from the color the stratifcation of Gilead society, and she describes Gilead itself as a "sort of a designer's dream because everything is colour-coded already and you just sort of have to guide the viewer with the symbols that already exist" (Dundas 2019). In approaching Atwood's novel, Nault explains to *Shondaland*'s Madeleine Deliee that "the comic format complements both the narrator's perspective—which refects jumps in time, memory, and thought and the story's frequent overlapping of reality and fantasy" (Deliee 2019). Nault hand-painted each page using watercolor, and selected distinct palettes to convey setting and time: fashbacks to "the time before" are depicted through a full range of color that "evok[e] what we see every day, and a more normal time," whereas scenes in Gilead are depicted through a juxtaposition of muted backgrounds and high saturation, such as the vivid red, green, and blue of the Handmaids', wives', and Marthas' uniforms (Dundas 2019).

Nault employs compositional arrangements familiar from previous visual iterations of *The Handmaid's Tale*, as is evident in the striking twopage spread of the narrator and her shopping partner walking along the Wall, reminiscent of Marcellino's cover illustration. As is the case in Marcellino's design, the wall dominates both the composition and the Handmaids. The two women are indistinguishable, and identifable only by their white bonnets, red uniforms, and shopping baskets. In Marcellino, the haunting quality of the composition is heightened by the Handmaid looking behind her at an unknown beyond the frame, whereas Nault's depicts both Handmaids moving forward toward an explicit horror: six fgures hanging from the gallows—the images of fetuses on their neck placards identifying them as doctors.

Given the nature of the graphic novel, one could say that the iconography of the Handmaid's uniform is reinforced in each panel that features a Handmaid. Nault, however, devotes several panels to Offred donning the Handmaid uniform at the novel's outset—slipping into the shoes, pulling on the gloves, adjusting the white wings, holding the shopping basket. As is the case in other adaptations, she extends this iconography to additional signifcations. Nault organizes the frst page of "II: Shopping" into two horizontally oriented panels both of which depict the narrator's red shoes poised at the edge of the red circular rug of her bedroom in the Commander's house from two distinct perspectives. The top panel offers a bird's eye view of much of the bedroom, which is rendered in grayscale save for the saturated red of the shoes and rug. The reader is able to see a window seat with a pillow embroidered with "faith," the footboard of the bed, and a partial view of a door. The lower panel offers a close-up perspective of the red shoes, which are centered in the composition. The text accompanying the lower panel is that in which the narrator considers her name, "I have another name […]. I keep the knowledge of this name like something hidden, some treasure I'll come back to dig up, one day. I think of this name as buried" (Atwood and Nault 2019: II Shopping, n.p.). In Nault's portrayal, the rug resembles a target, or, given the placement of the shoes at the edge, a hole to jump or fall into, or, further, given the juxtaposed text, a future excavation site.

The Handmaid's shoes are a continued motif throughout the adaptation, and provide the focus of several panels—the narrator walking doubled with her shopping partner, under the stall door in the Red Center bathroom, dressing in her bedroom, walking through Serena's garden, shoes like hers dangling from the gallows. The rug also comes to symbolize the narrator and her personal safety. The concluding scene, in which the Eyes come for Offred, includes a panel in which a guard's boot steps onto the round red rug; yet, a few pages later, this potential threat is revealed as a potential opportunity. The novel's opening image is bookended by that at the "tale's" close—a full, white page, the top half of which depicts the narrator's billowing skirt and single shoe-shod foot as she runs toward the future, "And so I step up, into the darkness within; or else the light" (Atwood and Nault 2019: XV Night, n.p.). The absence of the rug, here, reinforces the idea that Offred has "stepped up," or "stepped into."

As in the case of other adaptations, the graphic novel explores the tension between the Handmaids' uniformity and individuality, both thematically and compositionally. Nault's achievement is perhaps best seen in a series of panels devoted to the narrator's daily meetings with her shopping partner, Ofglen. Atwood's frst description of this meeting reinforces the manner by which Gilead codes Handmaids as identical and shows, also, that they remain individual to each other: "A shape, red with white wings around the face, a shape like mine, a nondescript woman in red carrying a basket, comes along the brick sidewalk towards me. She reaches me and we peer at each other's faces, looking down the white tunnels of cloth that enclose us. She is the right one" (Atwood 1985: 28–29). Nault adapts these meetings in several half-page panels spaced through the novel, each of which includes an inset half circle, which is divided into four rows, each depicting a distinct stage in the meeting: one Handmaid waiting alone, another's approach and their exchange of greeting, the two standing together, two walking off. The background behind the half circle depicts the houses of two Commanders. The state of being the "right one" comes to mean later when the narrator realizes that she is meeting the "wrong" one. Toward the end of the novel, the narrator waits at the corner for the partner she has come to know as Ofglen. Nault renders this meeting compositionally in the same manner as the previous two meetings. The accompanying text reads: "I notice nothing at frst. Then, as she comes nearer, I think that there must be something wrong with her. She looks wrong" (Atwood and Nault 2019: XIV Salvaging, n.p.). At this moment, the narrator realizes that the woman she has come to understand as Ofglen is no longer Ofglen. This name has come to signify a new body, one that Atwood's narrator comes to understand through differential features: "She is altered in some indefnable way […]. She isn't Ofglen. She's the same height, but thinner, and her face is beige, not pink" (Atwood 1985: 294). Nault establishes Offred's realization through visual markers that likewise become evident through comparison. Whereas in the previous scenes, the fgure approaching from the left of the frame is slightly shorter than the fgure waiting toward the center of the frame, in this last meeting the approaching fgure is slightly taller than the fgure waiting. In visually presenting this meeting as the same as those before it, Nault creates a telling rift between the visual and worded cues and again reinforces that tension between what is visible or invisible, what is observed, and what is understood.

#### 2.6 Motion-Based Transfer

As we have seen, the transfer of iconography in print-based visual adaptations of *The Handmaid's Tale* tends to reinforce the Handmaid uniform as a shorthand signifer for a range of meanings, and does so in ways that foreground unique medial and modal aspects. I will now turn to an examination of audiovisual, motion-based adaptations on stage and flm. The transfer of iconography from a verbal, written media product to an audiovisual, motion-based one involves activities similar to those utilized in print-based visual adaptation in that worded descriptions and narrative voice are conceptualized in new ways, but audiovisual, motion-based adaptations likewise bring in additional considerations, among them gesture, movement, three-dimensionality versus two-dimensionality, live or recorded performance, live or recorded speech and music. For example, *The Handmaid's Tale* as conveyed through ballet poses a medial experience unique from an operatic version, though both communicate broadly through set, sound, movement, and gesture. Similarly, a one-woman theatrical performance of *The Handmaid's Tale* and a television series both transfer theme, iconography, and other elements from Atwood's novel and other adaptations, yet each poses unique challenges, among them pacing, character empathy, dialogue construction, and set design. My goal in what follows is not to provide an exhaustive examination of each modality, but, rather, to highlight the manner by which balletic, operatic, theatrical, flmic, and televisual adaptations of *The Handmaid's Tale* interpret iconography of the Handmaid uniform, and, in so doing, demonstrate the fexibility of the iconography and the manner by which it might be bent to align with a specifc modality.

As has been the case for print-based adaptations, the transfer of iconography from one media product to another in audiovisual, motion-based adaptations has come to rest most visibly in the uniform. As the uniform transfers from motion-based adaptation to motion-based adaptation, it is adapted to meet the requirements of different modalities. Because the uniform described by Atwood is too restrictive for dance, in adapting *The Handmaid's Tale* to ballet, costumer designer Liz Vandal reimagined the Handmaid uniform for choreographer Lila York's "dance drama," which premiered with Canada's Royal Winnipeg Ballet in October 2013, as a red tunic with a triangle-shaped yoke that covers the head, and is open at the sides and ftted over a gray leotard. The bonnet and white wings are reimagined as a structured, pyramid-shaped headpiece ftted to the back of the dancer's head. While a technical necessity for this modality, the increased mobility and unrestricted vision of this uniform create a version of the Handmaid distinct from that offered by Atwood, and many other adaptations. Within Gilead's stratifed society, mobility is linked to power, and each group's uniform allows a degree of mobility and visibility appropriate to their station. As Atwood's narrator explains, the Handmaid's uniform limits perception and vision: "Given our wings, our blinkers, it's hard to look up, hard to get the full view, of the sky, of anything. But we can do it, a little at a time, a quick move of the head, up and down, to the side and back. We have learned to see the world in gasps" (1985: 40). In tokenizing a fundamental feature of the costume, the ballet impacts the audience's sense of how the iconography functions within the society.

The conditions of live performance may have informed also the decision of the Boston Lyric Opera adaptation (2019), directed by Anne Bogart, and conducted by David Angus, to opt for the white bonnet, but not the wings.6 The wings would likely impact the performers' hearing and ability to communicate with each other on stage. Yet, as with the balletic adaptation, the visibility of the Handmaids' faces to audiences both within and outside the diegesis, and Handmaids' own unobstructed vision, impacts audiences' understanding of the role of the Handmaid uniform within Gileadean social structures. In other ways, though, the opera overtly aligns itself with established iconography of the uniform by including in the program a sketch for the Handmaids' costume by James Schuette, who also designed the sets, juxtaposed against an image of Marcellino's cover design for the Jonathan Cape edition.7

The one-woman *The Handmaid's Tale* stage adaptation written by playwright Joe Stollenwerk's script, directed by Brian Isaac Phillips, and performed in 2015 at the Know Theatre of Cincinnati, featuring Corinne Mohlenhoff as Offred, likewise omits the white wings, though, as the play unfolds within a single, private setting, the absence is not as pronounced. Additionally, given that the performance (including communication of character, story, narrative tension, and confict) rests on the nuances of Mohlenhoff's delivery, the costume decision is understandable. In Noelle Wedig's costume design, the bonnet fts over a cowl-like garment that conceals the Handmaid's hair and head. This layering of garments coupled with Andrew J. Hungerford's strategic lighting design, which utilizes high- and low-contrast effects to refect and craft emotional and environmental mood, creates a Handmaid who, though literally visible to the audience, comes through as metaphorically veiled. Zack Hatfeld, reviewing the play for *The News Record*, likewise comments on the relationship between costuming and lighting in this production, calling Wedig's interpretation of the Handmaid costume, "a visual marvel, encapsulating the appropriate shades of paranoia and violence," and commenting that Hungerford's lighting "creates a tonal dissonance vital to understanding the complex psyche of Mohlenhoff's role" (Hatfeld 2015).

Implications for performance and reception likewise inform how flm and television adaptations interpret the Handmaid uniform. Colleen Atwood's take on the iconic uniform for the 1990 flm adaptation directed by Volker Schlöndorff and starring Natasha Richardson is much more streamlined than other iterations. Handmaids in this version wear only a sheer, red head scarf in public, and occasionally a red veil. Handmaids' faces are visible, and they remain distinguishable from one another, as a result. The decision to avoid a more obtrusive head covering may have come from a fear of blocking the audience's view of actors' faces. If the poster advertising the flm, which features lead Natasha Richardson, ostensibly unclothed save for the red garment she holds against her body, is any indication, this flm seems much less concerned with highlighting the Handmaids' invisibility within Gilead than in highlighting Richardson's star qualities. By contrast, for the Hulu series created by Bruce Miller, which aired on April 26, 2017, costume designer Ane Crabtree, understanding the aesthetic and symbolic impact of the full, white wings, persuaded the show's developer to use them instead of less obtrusive headscarves (Chuba 2018). In the third season, in particular, the series begins to exploit the headpiece as a device of both invisibility and visibility. Episode 2, for example, opens with a scene of June/Ofjoseph (formerly Offred) waiting for her shopping partner, and watching other Handmaids pass her on the street. The camera adopts a subjective perspective from "behind" the white wings, allowing viewers to see the scene as she does, as if through a tunnel. Once within the grocery, however, June/Ofjoseph uses the privacy provided by the wings to exchange information with Alma/Ofrobert, a Handmaid involved in helping to organize the resistance.

The motion-based adaptations reiterate the signifcance of the iconography of the red cloak and white bonnet in ways that highlight features of their specifc modalities. Thus, the highly stylized uniform adopted by the ballet supports both the aesthetic characteristics and physical necessities of the medium—the uniform is visible, but downplayed to foreground the physicality of dance. In comparison, the Hulu series can utilize more restrictive uniforms because the handheld cameras can compensate for actors' restricted mobility.

#### 2.7 Transfer and Networks

The popularity of the Hulu series and the popular media's depictions of the series as a mirror to the political climate have contributed, no doubt, to the "real world" upswing in Handmaid iconography and what we might think of as "embodied" adaptation. Since spring 2017, individuals have donned Handmaid cloaks and white bonnets at numerous protests globally, mostly related to women's reproductive rights and real and threatened erosions of other freedoms, among them the NARAL Pro-Choice protests in Texas, protests against the abortion ban in Ireland, and against Brett Kavanaugh's 2018 United States Supreme Court nomination. As *Guardian* writer Arielle Bernstein notes, "The image of the red robed and white bonneted women […] has become as celebrated as Rosie the Riveter" (2018). Alex Yarde, writing for *The Good Men Project*, compares "the unmistakable iconography of the red bonnet as a universal symbol of oppression," to "the white worn in the Suffrage Movement at the turn of the last century" (2019). This pattern of reading the Handmaid "in context" brings us to an important question: what *does* the Handmaid uniform symbolize?

To answer this question, I return to the examples with which this essay began and the questions they raise: How did a line of red trees come to make a political statement? How did a group of women in red cloaks come to illustrate a threat to scientifc research? How did a red cloak and white bonnet become a site of sexual pleasure? Each of these examples comes to "mean" at the point of transfer, or, borrowing from Elleström's communication model, in the "intermediate stage" between one place and another, "that makes the transfer possible" (2020: 10). That is, each comes to mean at the point when the perceiver recognizes the "thing" as not the thing. For example, the trees in the colonnade are simply topiaries, three-dimensional static structures composed of faux cranberry branches. Set within the larger cultural moment, the trees are still seen as trees but also, satirically, as not trees, as Handmaids. In transferring the iconography of the Handmaid from the Hulu series to the holiday display, the edited photograph discloses a semiotic message that conceals the materiality of the topiary. The materiality continues to exist (i.e., the topiary continues to exist as a holiday decoration), but, within the context of the photograph, the tree's semiotic potential dominates in the transfer of the message to the perceiver. The political effcacy of the protesters dressed as Handmaids comes in the recognition that they are not Handmaids, but, rather, are evoking the status of Handmaid to prove a point. Were the women actually Handmaids, the act of wearing the uniform would not constitute protest. Likewise, in the example of the "Brave Red Maiden" costume, the iconography of the Handmaid transfers to the person wearing the costume. A pleasure in the costume is recognizing the lover as both Handmaid and not Handmaid. In each example, the transfer of iconography exposes the border between the thing and not the thing. The precision of the semiotic sign is not as important as the sign's broader connotations, which are understood through the perceiver's recourse to collateral experience.

Perceivers' collateral experience, which, in the case of these examples, takes shape from a larger adaptation network, might include familiarity with Handmaid iconography through experiences with one or more of the numerous print- and motion-based adaptations, visual publicity for the Hulu series, or accounts of protesters in the news. The experience allows perceivers to perceive the message in the transfer, thus enabling them to decode the representation—as a political or erotic, for example—within the context of *The Handmaid's Tale*. Perceivers' experiences with varied messaging allow them to decode the line of red trees, or the group of women in red cloaks, or the costume denotatively, and perceivers' recourse to the intermedial adaptation network allows them similarly to decode these examples connotatively.

What can *The Handmaid's Tale*'s adaptation network tell us about intermediality in general? As Rajewsky, Elleström, and others have demonstrated, the so-called borders between media and modalities are largely contrived and perceptual. Recognizing such borders as constructed, however, is not in confict with a recognition of their usefulness or a curiosity related to their areas of porosity—the shifting points of contact generated through processes of adaptation. Perceivers' collateral experiences shape their decoding of icons, as does the generalized collateral experience generated by the repetition of specifc iconography through an adaptation network. Elleström's and Rajewsky's proposals that intermediality be conceptualized as a meeting of "border zones" remain a productive way of thinking about such relationships (Elleström 2010; Rajewsky 2010). As Rajewsky explains, borders can be understood as sites of possibility, "as enabling structures, as spaces in which we can test and experiment with a plethora of different strategies" (2010: 65). Adaptation is the enabling structure that makes the transfer of iconography between media products possible. Much like Atwood's narrator, who serves as an intermediary between books and discs, but who also redirects that transfer to other, unanticipated points of communication, adaptation illuminates areas of porosity between media and media products, further erodes notions of impermeable divisions between media, and enables understanding of broadening intermedial networks.

#### Notes


Constitutional rights was inconceivable. Yet in the wake of the 9/11 attacks, we saw just how natural it felt for citizens and governments to overreact in the name of national security" (Lyman 2015). Deborah Dundas, who interviewed Renée Nault for *The Toronto Star*, notes that delays in developing the graphic novel adaptation resulted in a keen alignment of the book's thematics and current politics: "its dystopian slant echoes many of the issues facing a Donald Trump America—eroding women's rights, the rise of religious fundamentalism" (Dundas 2019).


#### References


———. 2012. *The Handmaid's Tale*. Illustrated by Anna and Elena Balbusso. London: The Folio Society.


———. 2017. Adaptation and Intermediality. In *The Oxford Handbook of Adaptation Studies*, ed. Thomas Leitch, 509–526. Oxford: Oxford University Press.

———. 2020. The Modalities of Media II: An Expanded Model for Understanding Intermedial Relations. In *Beyond Media Borders: Intermedial Relations among Multimodal Media, Volume 1*, ed. Lars Elleström, 3–91. Basingstoke: Palgrave Macmillan.

Flock, Elizabeth. 2017. How Margaret Atwood Dreamed Up the Costumes in 'The Handmaid's Tale.' *PBS.org*, 25 April. https://www.pbs.org/newshour/

arts/how-margaret-atwood-dreamed-up-the-costumes-in-the-handmaids-talepremiere-hulu. Accessed 3 September 2019.


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/ by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## Building Bridges: The Modes of Architecture

#### *Miriam Vieira*

#### **Contents**


#### 3.1 Assembling Foundations

It is possible to explore the relationship between architecture and other media through a wide variety of theoretical scopes—from time-space relations, as described by Gaston Bachelard (1994 [1964]) and Stephanie Glaser (2009, 2014), to Ulf Pettersson's cognitive approach (2013), or even Jennifer Bloomer's philosophical one (1993). It is also possible to view the architect's activities or a piece of architecture from different angles. Although still not nearly as extensive as the feld of research dealing with the relationship between literature and painting, investigations of literature and architecture have nonetheless become more common over

M. Vieira (\*)

Universidade Federal de São João del Rei, São João del Rei, Brazil

<sup>©</sup> The Author(s) 2021 59

L. Elleström (ed.), *Beyond Media Borders, Volume 2*, https://doi.org/10.1007/978-3-030-49683-8\_3

recent decades. Departing from the premise that ekphrasis is meant to evoke an absent image in the mind of the spectator, or the reader, and thereby to provoke an emotional response, I have elsewhere studied such relations by using the notion of architectural ekphrasis as a theoretical operator, under the light of intermedial studies (Vieira 2016, 2020). While examining the possibilities for working with this typology, I realized that the already consolidated pictorial models' parameters were not suffcient for my needs. One of the greatest challenges I faced was the delimitation of architecture's medial traits that literature is actually able to transmediate. Lars Elleström's proposed model for studying the transfer of media characteristics was crucial to achieving consistent results (2010, 2020). The four modalities of media, along with their qualifying aspects, became the backbone of an interpretative model proposed to explore the presence of architecture in literature. The aim of this chapter is then to delve into the core of my research: the modality modes of architecture.

According to Mario Biselli, architecture is "the intersection between the hard sciences and humanism"1 (Biselli *apud* Penna 2005: np), while Gustavo Penna defnes it as "the spacial synthesis of all culture"2 (Penna 2005: np). In order to establish which kind of architecture I mean to explore, I depart from its dictionary defnition as the "art and technique of organizing spaces and creating environments to shelter the many kinds of human activities, aiming also at specifc aesthetic intentions"3 (Houaiss and Villar 2009: 186), along with the premise that, in architecture, "the work of many individuals will be involved and transformed within the unity of the fnished whole" (Crowther 2009: 182–183), where the "whole" is the building itself. Adopting a more philosophical point of view, Karsten Harries refects upon the notion of architecture as an interpretive and critical act, as well as on the possibilities of viewing architectural works as "texts" (Harries 1997: 4). According to Harries, the style of a building, when attributed to a certain code, can communicate a "way of standing in the world, a specifc *ethos*" (Harries 1997: 92). The French architect Émile Aillaud (1902–1988) has for his part argued that "architecture is nothing more than the organization of a story whose syntactical elements are constituted by a scenery that carries it". Aillaud explains that he himself has built "cities, and these cities are themselves operas, stories told; they are great stories in which one can dwell, an inhabitable story" (Aillaud *apud* Hamon 1992: 29, footnote 13). Based on Aillaud's reasoning, Philippe Hamon illustrates the important narrative capability of architects, "who on the one hand listen to the narrative instructions of their clients, and on the other hand produce places that are stories as well as stories that are places" (Hamon 1992: 29). As a qualifed medium, architecture may, under specifc contextual qualitative aspects, convey messages to society, such as power or status. With this idea in mind, I consider that the two-way narrative capability suggested by Hamon requires a refnement of the communicative tools used in interactions between the client, the architect, and other relevant professionals involved in the architectural process, in addition to the building's fnal users.

Considering how all media products necessarily have several media dimensions, Jørgen Bruhn uses the term medialities to discuss media as "specifed clusters of communicative forms" (Bruhn 2016: 17) that "may be briefy defned as tools of communicative action inside or outside the arts" (1). Accordingly, in his treaty on the autopoiesis of architecture, Patrik Schumacher4 argues that architecture can and should be considered as a (qualifed) medium (2010: 323–362). The author explains that "architects communicate to wider audiences via buildings and designed spaces. Buildings and spaces constitute a very specifc type of communication: they are ordering and framing communications […] between people co-present in space" (Schumacher 2011: 1). In other words, when understood as a process, architecture may be considered a medium due to its inherent potential to store communication. The concept of "process" is thus central to my previous investigation of architectural ekphrasis.5 Moreover, the communication between those involved (client, architect, engineer, technicians, construction workers, and others) in relation to the edifce's modality modes is key to the fulfllment of an architectural process.

Architecture, as a process,6 originates in a client's wish. And that wish must, before it can be consolidated into a palpable building, undergo a succession of stages carried out through sketches, technical drawings, mockups, models, and so forth. Such depictions operate in the same way as *librettos* in opera, musical scores in performance, or screenplays in flm. As such, we understand that architecture is developed through a dynamic communicative process in which every stage is accomplished through specifc qualifed submedia (graphic and technical representations) that not only "crave" for transmediation but also operate as part of a predetermined hierarchical process. For instance, the client's desire is followed by the architect's interpretation of the oral verbalization of such wishes, expressed through sketches and technical drawings. However, as the architect Steen Eiler Rasmussen states, "not everyone […] can visualize a building merely by looking at the plans" (Rasmussen 1964 [1959]: 9). Therefore, the inclusion of other submedia—such as models and technical representations—is made necessary to the process's successful culmination at the construction site, where the architectural work will in fact be executed.

According to Rasmussen, "the architect is a sort of theatrical producer, the man who plans the setting for our lives, […] but his producer job is diffcult for several reasons. First of all, the actors are quite ordinary people" (Rasmussen 1964: 10). In this sense, the submedia which structure the stages of the process "are not an end in themselves, a work of art, but simply a set of instructions, an aid to the craftsmen who construct his buildings" (Rasmussen 1964: 14). These stages can be roughly described as: (a) the verbalization of a client's desire; (b) the architect's mental interpretation of the client's intention; (c) the representation of this cognitive import through two-dimensional visual qualifed submedia;7 (d) the graphic representation of the future edifce's volume, expressed by CG renderings or 3D models;8 (e) the qualifed submedia known as a blueprint, comprised of plans, façades, and sections, technically and diagrammatically representing the architectural project;9 (f) the construction site, in which the entire process will be transmediated, to fnally culminate; (g) in the building *per se*. All of the mentioned submedia—graphic, diagrammatic, and technical representations, such as sketches, plans, blueprints, mockups, CG renderings, and 3D models—are relevant within the chain of a communicative process containing different levels of technical and architectural literacy, where a considerable amount of verbal articulation may be required if any of the stages is not decoded in detail.

Once the understanding of architecture as a medium has been established, I believe it is useful to identify its intrinsic characteristics and affordances, which are the medial traits that may be transmediated by literature, more specifcally, works of fction in which architecture plays a signifcant role so that the full architectural experience can be unfolded in the reader's mind through words. Since I have previously identifed that the already consolidated pictorial models' parameters are not enough for scrutinizing the relationship between these two media types—architecture and literature—the proposed model for the study of transfer of media characteristics among dissimilar media in this volume (Elleström 2020) was crucial for the delimitation of architecture's medial traits. Therefore, as this is a reductionist model, where there is nothing to remove, but certainly to expand from, the foundation of my analysis also counts on the notions of embodiment and perspective in order to broaden the understanding of the process in which architecture is transmediated by literature.

#### 3.2 Setting Up the Piers, or the Modes of Architecture

As proposed by Elleström (2010, 2014, 2018a, b, 2020), semiotic characteristics—which include iconic, indexical, and symbolic aspects—are combined with presemiotic ones—which speak to the fundaments of mediation by involving the material, sensorial, and spatiotemporal modalities and their respective contextual and operational qualifying aspects. Therefore, in order to investigate the four modalities of architecture, I depart from questioning the basics: (a) What is the dominant sign system in architecture? (b) What materials constitute architecture? (c) How can architecture be perceived by the senses? (d) What are the possible relationships between architecture, time, and space? (e) Under which historical, cultural, and social circumstances has the building been designed? (f) Which aesthetic movement has infuenced the architectural process?

The semiotic modality deals with the system of media signifcation and representation; the representing and represented; and more specifcally to my case study, the objects and different media products that act as signs in relation to one another. Thus, it is relevant to understand how iconicity is based on the similarities between the involved media types; indexicality on their contiguity; and symbolicity on pre-established habits and conventions (Elleström 2010, 2014, 2018a, b, 2020). The frst premise, the structural similarity—or diagrammatic iconicity, according to Charles Sanders Pierce—between the style of the architectural work and the written text, is probably the most frequently studied, as for instance in the way that nineteenth-century English Gothic fction borrowed elements from gothic medieval architecture. Hamon (1999) consistently shows examples of iconicity based on similarity in his work on the relations between architecture and nineteenth-century French literature.10 Regarding the architectural process stages that take place before the construction begins, the second premise relies on indexicality, whereas the habits and conventions suggested by the last premise will depend on the cultural background of those involved. Hence, architecture as a process is apt to prompt a communicative process by encompassing both visual and verbal signs, by means of sketches, plans, and models, as well as through the negotiations between client, architect, and all other professionals involved in the design project. Therefore, meaning is assigned through a combination of signs of equal importance, weight, and value within the architectural chain of semiosis.

An edifce is comprised of different materials, such as brick, cement, mortar, glass, and steel. However, the material modality concerns both the physical materiality and the mental relationship involved within its transmediation. Therefore, if architecture is to be understood as a process spanning several stages, materiality will also manifest itself inside the minds of those involved in said process, as discussed by cognitive studies. Architect Christian Norberg-Schulz (1980) relates the materiality of architecture to the linguistic structure employed in its procedural verbal negotiations. According to him, the structure of a place—be it a country, a landscape, or a building—includes the space itself, the characteristics of the manmade environment, and the respective technology employed in its execution. The author states that places tend to be designated by nouns, such as "island", "forest", or "street", as well as "wall", "roof", and "door", since instead of referring to "space" in our daily lives, we refer to the relationships between things which are, for example, "over", "in", or "along" one another. Space, therefore, if understood as a system of relationships, is indicated through prepositions and phrasal prepositions. The technology and the characteristics present in man-made environments are, in turn, indicated by adjectives (Norberg-Schulz 1980: 15–16). Norberg-Schulz's contribution is indeed useful in the linguistic analysis of architectural ekphrasis.

As a process, architecture comprises a series of stages between the decision of having something built to the actual usage of the edifce in which is created an interactive discourse between client(s), architect(s), and all others involved. Beyond the linguistic elements put forth by Norberg-Schulz, this communication will only reach a common goal when a common cognitive import is incited among all parties. Although the process is conceived in the client's mind, its function will be triggered by the architect. This means that the transfer of vivid visions and perceptions by means of a shared mental image, which happens to be the rhetorical principle of ekphrasis, is highly relevant to its successful accomplishment.

According to Rudolf Arnheim, "a building, […] in all its aspects is a fact of the human mind. It is an experience of the senses of sight and sound, of touch and heat and cold and muscular behavior, as well as the resultant thoughts and strivings" (Arnheim 1977: 4). Thus, when it comes to the sensorial modality, it is imperative to investigate the effects of the fve senses, along with proprioception (and even interoception), on the production and reception of an architectural site. The sensorial modality relates to the reception of the fnal product. After all, many actions must be taken in relation to an edifce's affordances,11 such as the opening/closing of doors to enter/exit spaces, or the opening of windows to let light and air in/out. Moreover, regarding touch, the impact of the relative brightness, temperature, and humidity in an edifce will all affect its users. The combination of multiple materials with the natural element—meteorological conditions—allied to the human activities happening inside the edifce, will produce aromas and odors which are likely to affect the sense of smell. One example might be the smell of food that recurrently comes from the kitchen and is permanently absorbed by the living-room carpet. Hearing is also affected by the combination of the chosen materials, the weather, and human activities: for example when the sound of rain on a tin roof hinders someone's concentration while studying. The only sense that is not likely to manifest in an architectural process is taste. However, regarding its connotative meaning, taste, as in aesthetics, should also be considered. For Paul Crowther, "the building's articulation of form, shape, and mass in enduring materials, on the basis of symmetry, proportion, and the like, offers a kind of rectifcation or *idealization* of the body's vectors of sensorimotor activity" (Crowther 2009: 181—author's emphasis). Although not as deeply scrutinized as the fve senses, along with mental introspection and interoception, the relevance of proprioception is brought about in the expanded model for understanding intermedial relations (Elleström 2020: 27, 30, 49). The kinesthetic sense, in particular the sensorimotor activities, must be included when it comes to the investigation of architecture's sensorial modes. As underscored by Arnheim, all these sensorial experiences depend on the historical, social, and even personal context. In sum, an architectural site can virtually activate all senses at once.

Given that "*spatiotemporal modality* is a category of spatiotemporal media modes" and media products "consist of physical matter", their spatiotemporal properties can "be grasped by human minds" (Elleström 2020: 48). An architectural process defnitively occupies different types of spaces which cannot be dissociated from time. In spite of being known as the art of creating space, it is perhaps more accurate to say that architecture alters and organizes space. Unsurprisingly, space and time are among the comparative aspects most frequently studied by both architectural and literary scholars. Rasmussen argues that the relationship between space and time is among the greatest diffculties faced by any architect whose "work is intended to live on into a distant future". The professional "sets the stage for a long, slowmoving performance which must be adaptable enough to accommodate unforeseen improvisations" in an edifce which "should preferably be ahead of its time when planned so that it will be in keeping with the times as long as it stands" (Rasmussen 1964: 12). For him, "one of the proofs of good architecture" lies in buildings whose space "is being utilized as the architect had planned" throughout time (Rasmussen 1964: 12).

In architecture, communication and human experience do not endure virtually, as in literature; they happen, effectively, within a materialized space. In contrast with literature and cinema, which can be preserved through technical, digital, or virtual reproduction, an edifce never reaches the status of fnal product, since its materiality will always be vulnerable to climatic and socioeconomic variations. Another noteworthy aspect of the spatiotemporal modality is the ratio between the fnal product's volume, depth, and height, as considered by Stephanie Glaser in her latest works (2014, in press).

In order to provide stability to the piers, besides the understanding of architecture's four modalities, special attention should be given to the origins, delimitations, and usage within the specifc historical, cultural, and social circumstances of both production and reception of the investigated architectural environments in relation to their aesthetic and communicative features, as in the qualifying and operational aspects suggested by Elleström's model (2020). This investigation leads to the fact that, due to the matter of form and function, including the study of ergonomics and affordances of architectural elements, the scale of an edifce is set by the human body. Thus, two more notions inherent to architecture should be taken into consideration in order to tangibly delve into its modes: embodiment and perspective.

#### 3.3 Adding Girders: Embodiment and Perspective

In section two, I discussed how the modes of architecture both operate throughout its process and compose its fnal media product. Since its materiality is also manifested at the virtual level, the semiotic mode may be indicated not only by possible structural similarities and diagrammatic forms of representation evidence, but mainly by verbal negotiation in assigning meaning along the architectural process as a whole. By delving into the presemiotic modes, I realized the indissoluble three-dimensional space and time in relation to the building, and the role of affordance and ergonomy within aesthetic matters of the studied edifce. After all, since ancient times, architecture has relied on the human body as both metaphor and unit of measurement (Spurr 2012: 36). Yet the interaction of the user with architectural environments through the senses, including proprioception, makes the notion of embodiment indispensable to set a backbone for the architectural traits most easily transferred to literature. At the same time, the points of intersection of the different connotations of the notion of perspective in different disciplines—architecture, art history, and literature—must also be taken into account.

The understanding of how subject and object, in opposition to Cartesian dualism, comprise two facets of a single unit is fundamental to further investigations of architecture as a process. The concept of embodiment proposed by Maurice Merleau-Ponty (1945) sustains not only the inextricability of body and mind, but also the awareness of the perception and representation of the world (Auslander 2008: 136–139). Embodiment is used as a basis for performance studies when dealing with stage experience by the actors and its perception by the spectators. As previously mentioned, architects Émile Aillaud and Steen Rasmussen established, in two distinct historical moments, the analogy of the building as stage. Chiel Kattenbelt (2006), in a similar line of reasoning, uses the term "corporeality" to designate the effect of the actor's body onstage as in the materiality of that body in the theatrical space. This idea recalls the proposed discussion of how the body of a dancer may act as both technical medium and media product, as when Elleström quotes the famous question posed by Yeats—"How can we know the dancer from the dance?"—in the introductory chapter of this volume (2020: 36). Nonetheless, in the case of architecture, the focus should not be on distinguishing the architect from the edifce, but on determining how anyone can actually experience kinesthetically the three-dimensionality of its fnal media product. That is to say, even if this process initially relies on concepts and transfer of mental cognitive import between client, architect, and other professionals, in order for any edifce to be considered architecture, as opposed to a piece of sculpture, it must eventually be experienced by the bodies of fnal users.

In an attempt to advance relevant aesthetic and ethical discourses within architectural theory, Åsa Dahlin (2002) highlights that it is not enough to (re)cognize architecture's material aspects. For her, reason and emotion, much like mind and body, should not be treated as oppositions, but as parts of the same system. It is therefore necessary to consider knowledge generated by humans' physical as well as mental consciousness. Sophia Psarra (2009), in turn, is concerned with the way theoretical focus on the visual perception of form has shifted to the embodied experimentation of space (13), which guided her methods for investigating the interfaces between conceptual and perceptual facets of an architectural project to be based on the notion of embodied experience. In sum, embodiment—as in proprioception, be it from a spatiotemporal and sensorial understanding, or from a philosophical standpoint—is relevant when showing the effect of the user's body within an architectural environment.

While Rasmussen states that "it is not enough to see architecture; you must experience it [...] you must dwell in the rooms, feel how they close about you, observe how you are naturally led from one to the other" (Rasmussen 1964: 33), Arnheim calls attention to the fact that optical images are projected in two dimensions by the human eye, so the brain does fully grasp a three-dimensional object just from one stand point (Arnheim 1977: 110). To provide a more concrete example, the way in which "we may change our position deliberately to gain a more comprehensive view" (110) can be illustrated by the renowned passage from Marcel Proust's *In Search of Lost Time* in which the narrator observes "three clock towers that change their relative position" (118–119). According to Arnheim, in this scene, Proust anticipates the cinematic technique of traveling, which is related to periegesis, an ancient rhetorical tool employed in the description of places, landscapes, and buildings (see Glaser in press). Accordingly, Rasmussen suggests that users' perception of different standpoints is enabled by walking around and within an architectural site. So even if perspective deforms the object or tricks the eye, the human mind still "organizes, completes and synthesizes the structure" such that the three-dimensional object "will be seen spontaneously as a whole" (Arnheim 1977: 110). The acknowledgment of the timeless existence of an edifce in space in the visitor's architectural experience, along with the peculiarities of the human eye in relation to the mind, heightened my awareness of the relevance of the dialogue between the notions of embodiment and perspective. As previewed, I now present how the different facets of the notion of perspective affect architecture's medial traits to be presented by literature.

In architecture studies, even if it is part of a wider discussion of representation at a more technical level,12 Alberto Pérez-Gómez and Louise Pelletier have claimed that their "ultimate aim is to probe the possibilities of building architecture as a poetic translation, not a prosaic transcription, of its representation" (Pérez-Gómez 1997: 8). The authors have historicized methods of architectural representation since the Renaissance, when the introduction of systematized descriptive geometry signifcantly shifted the terms between the architectural process and its fnal product (Pérez-Gómez and Pelletier 1992: 22–37). Another major shift in the role of perspective within the architectural process was promoted by the consciousness of the observer as the organizer of a composition (Pérez-Gómez 1997: 19). In the same vein, Karsten Harries calls attention to the way architecture "relies on imitation" (1997: 112) as do other art forms. These theorists' discussion of the evolution of forms of representation has contributed to my own understanding of the difference between architectural technical representation by means of perspective, as well as the notion of mimetic representation prevalent in traditional ekphrastic studies.

The origin of the term perspective in art history and the history of science derives from either Arabic geometry (i.e. math) or western perception (i.e. aesthetic theory) (Belting 2011: 1–2). Math plays an important part in painting and architecture, which is not necessarily true for literary studies. Despite being "unoccupiable" (10), the vanishing point of classic perspective in painting allows spectators to objectify themselves externally from a given image, which, in turn, delivers a deictic proclamation from a particular location, in a specifc moment in time. The dynamic caused by perspective in paintings allows viewers to be positioned in places in which they have never physically been; in other words, presence and absence are intertwined in an indissoluble relationship (10). This relationship established from a vanishing point corroborates my investigation of ekphrasis inspired by architecture.

In the realm of the visual arts, the notion of perspective has changed in the wake of the (still controversial) work of Edwin Panofsky, *Perspective as Symbolic Form* (1997 [1937]). James Elkins explains how, differently from our current comprehension of the notion of perspective, Renaissance artists and writers conceived of "unchangeable" concepts, such as the idea of pictures being "in perspective". According to him, the greatest shift in the understanding of perspective was departing from its understanding "as a mute method, a practical subset of geometry" and adhering to "a powerful concept for ordering our perception and accounting for our subjectivity" (Elkins 1994: xi). Thus, when it comes to this favoring of its metaphorical understanding and forgetting it as a practice, there are inevitable consequences to the way we perceive it in painting, philosophy, literary studies (xi), and architecture. That is, for him, "perspective directs our eyes and orders our thoughts […] it seems to control not only what I see, but how I see and how I describe what I see" (Elkins 1994: 212). The diffculties of thinking through and around a certain viewpoint arise because, at least for the visual arts, perspective is mostly self-confned. For Elkins, "the modern concept of perspective is schizophrenic, [in a way], because it has two incompatible aspects". The frst is the "formal, rigorously defned branch of [math]: the perspective that we trace from Brunelleschi's experiments through the latest software", or today's understanding of the term perspective inside architectural studies. The second aspect "means a great deal, from subjectivity to eternity, and it is to be found virtually everywhere, from philosophy texts to political speeches". Elkins calls this second aspect, not derived from an equation or a graph, the metaphorical perspective. He explains that this "is 'our' perspective, the one we think about, and the one that describes how we view the world and constitute ourselves as viewing subjects" (6). Consequently, this claim for an exclusive individual point of view, as proposed by visual art theorists, is likely to engage with the notion of focalization as articulated by narratology studies.

The narratologist Gérard Genette (1972) acknowledged the terms "perspective" and "point of view" as synonyms and coined the term "focalization" as a replacement for both. Grounded on the premises suggested by Genette, Tamar Yacobi (2002) points out that perspective has revolutionized both the theory and the practice of European painting since the Renaissance, activating lines of communication that bring artist and spectator closer together. Likewise, both Liliane Louvel (2010) and Stephanie Glaser (2014) consider the narrator's focalization in ekphrastic passages dealing with architecture. According to Louvel, by offering both aesthetic and artistic context, the visual information brings additional clarity to the literary text. She states that, differently from painting, the presence of architecture in literature deals with the passage from architecture's three-dimensional space to the two dimensions of the literary object. She illustrates this argument with the opening of *Wuthering Heights*, in which a *trompe-l'oeil* effect reveals a labyrinthine space concealing the plot's secrets (Louvel 2010: 118–120). Accordingly, the house's façade, as focalized by Lockwood's character, is treated by Louvel as a historized page of its residents' lives. Glaser in turn bases her proposal on the notion of focalization suggested by Genette and revisited by Mieke Bal.13 For Glaser, the ekphrasis of an architectural site may deepen the narrative or become attached to it on a thematic level (Glaser 2014: 13–15). For me, this process of verbally presenting architecture is likely to engage the narrator's gaze in a way that implies not only the observer's physical position in relation to the architectural space—the literary *persona*'s focalization—but also their eye movement.

Aiming at incorporating focalization and other types of perspective present in spatial descriptions from nineteenth- and twentieth-century novels (*nouveau roman* included), the narratologist Monika Fludernik (2014) developed a typology based on a model for the triple distinction of perspective, used by Holly Taylor and Barbara Tversky in their studies of cognitive linguistics (1996). According to these authors, when describing a space or a place, a speaker can guide the addressee on a mental tour from three different perspectives: (a) the survey, (b) the gaze, and (c) the route. The survey perspective is neutral, like an aerial view. The use of deictic terms is avoided in this type of description, which favors the universal lexicon of orientation (North, South, East, West). This, according to Taylor and Tversky, is most common in literary descriptions, tourism guides, and textbooks. From the narrator's perspective, or from a route perspective, the observer is inserted into the narrated space. In type (b), the description, the observer stands on a fxed location and makes use primarily of stative verbs. Finally, in type (c), by favoring action verbs, the observer traverses the environment (Taylor and Tversky 1996: 371–376). With the goal of broadening this model within the context of narratology, Fludernik combines types (b) and (c) into a single category and creates subdivisions in order to demonstrate that spatial descriptions are not necessarily confned to the extradiegetic level (Fludernik 2014: 472–474). In sum, the focalization of the (ekphrastic) passages will fll functional gaps within the plot of a literary text, and the many facets of perspective will be revealed through literary descriptions of spaces, places, and buildings—that is, of architectural ekphrasis.

Although focalization and perspectivism do not necessarily coincide, both bring about the relevance of the observer's consciousness and embodiment as the organizer of verbal compositions. I therefore propose to add the notion of embodiment and to stretch the limits of perspective and focalization as girders in determining exactly which architectural modes may be successfully revealed by works of fction.

### 3.4 The Deck, or Architecture's Medial Traits

This realization that architecture's modalities and its design process may be conveyed by words led me to the understanding of the actual modes that may be feasibly transmediated to literature. The parameters were set by combining Elleström's proposed model for studying the transfer of media characteristics between dissimilar media to the notions of embodiment and perspective, namely (a) the appeal to the senses, likely to be revealed through the user's bodily interaction with the building; (b) the employed lexis, based on the materiality revealed at the physical level, through materials such as concrete and glass, and the virtual level, through what lies in both clients' and architects' minds; (c) language functions; (d) citations of architects' and edifces' names and titles; (e) meaningful repetitions; (f) possible structural similarities and the importance of diagrammatic forms of representation; (g) verbal negotiation in assigning meaning over the course of the architectural process; (h) the indissoluble spacetime relation in man-made environments, marked by deictic indicators; (i) the three-dimensionality of space, as proven by the ratios between height, length, and depth; (j) the role of affordance and ergonomy in relation to aesthetics; (k) the dynamics of perspective, focalization of the excerpt in question, and the literary persona's position in relation to the vanishing point, which will guide the receiver's eyes; and, fnally, (l) the intercrossing of all the aforementioned evidence with historical, cultural, and social contextualization of the architectural process as well as the positioning of client, architect, and fnal users. These parameters provided the deck for my proposal of an interpretative model for architectural ekphrasis that encompasses four horizontal and non-hierarchical typologies: technical, symbolic, contemplative, and performative.14

The inherent potential to store information throughout an architectural process, from its inception to the edifce *per se*, guarantees its status as a medium. Developed in a dynamic communicative process, each stage of its design is fulflled through specifc qualifed submedia (sketches, plans, blueprints, mockups, CG renderings, and 3D models), all of which require prior technical knowledge. In this case, both semiotic and material modalities play equally important roles within the transfer of medial traits. Conversely, when the beholder's cultural background knowledge is at stake, the semiotic modality prevails. I understand these two types of procedures as technical and symbolic, respectively.

There are cases in literature wherein characters contemplate architectural environments, buildings, or even urban landscapes from a specifc point of view, without the focalizing agent physically interacting with the focalized architectural site. In such cases, the transmediation of architecture may borrow medial traits from other media, such as painting, photography, or sculpture. Here the notion of perspective plays a highly relevant role, since it is the focalizer's relation to the architectural site that will determine its contemplative nature. Architecture is treated as a twodimensional medial confguration in cases where the focalizing agents behave as if they were in an art gallery. Meanwhile, when an architectural site is admired in its three-dimensional totality—as suggested by Aillaud, Rasmussen, and Kattenbelt—the focalizer behaves as if in a theatrical play and the edifce is likewise treated as a stage. Since embodiment plays a crucial role in the understanding of architecture as both process and media product, one may identify passages in which the observer, be it the narrator or a character, takes full advantage of the architectural medialities presented thus far. In such cases, all modalities, along with embodiment, have an impact on the characters' performative actions while touring an architectural site—going up and down, in, out, and around it. The reader ceases to be a mere spectator and becomes involved in a virtual performance. In sum, since only a limited number of architecture's medial traits can be satisfactorily transmediated by literature, the model here proposed by Elleström was crucial not only to delimit its modality modes, but mainly to bridge architecture to literature.

#### Notes


According to her, the type zero must be combined with the external, since it speaks not of the observer, but of the observed object.

14. This model is more thoroughly presented in my doctoral dissertation (Vieira 2016). Its core has been compiled in Vieira (in press).

#### References


Elkins, James. 1994. *The Poetics of Perspective*. Ithaca: Cornell University Press.

Elleström, Lars, ed. 2010. *Media Borders, Multimodality and Intermediality*. Basingstoke: Palgrave Macmillan.

———. 2014. *Media Transformation: The Transfer of Media Characteristics Among Media*. Basingstoke: Palgrave Macmillan.

———. 2018a. A Medium-centered Model of Communication. *Semiotica* 224: 269–293.

———. 2018b. Identifying, Construing, and Bridging Over Media Borders. *Scripta Uniandrade* 16 (3): 15–30.

———. 2020. The Modalities of Media II: An Expanded Model for Understanding Intermedial Relations. In *Beyond Media Borders: Intermedial Relations among Multimodal Media, Volume 1*, ed. Lars Elleström, 3–91. Basingstoke: Palgrave Macmillan.

Fludernik, Monika. 2014. Description and Perspective: The Representation of Interiors. *Style* 48 (4): 461–478.


———. 2011. The Autopoiesis of Architecture. Interview by Loreto Flores. *Revista de Arquitectura* 23: 1.

Spurr, David. 2012. *Architecture and Modern Literature*. Ann Harbor: University of Michigan Press.

Taylor, Holly A., and Barbara Tversky. 1996. Perspective in Spatial Descriptions. *Journal of Memory and Language* 35: 371–391.

Vieira, Miriam de Paiva. 2016. *Dimensões da écfrase: A presença da pintura e da arquitetura em romances de artista*. Belo Horizonte: Universidade Federal de Minas Gerais.

———. 2020. Architectural ekphraseis: Unveiling a Brazilian Wall-less House in Contemporary Fiction. In *Transmediations: Media Across Media Borders*, ed. Lars Elleström and Niklas Salmose, 118–135. New York: Routledge.


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/ by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## Media Representation and Transmediation: Indexicality in Journalism Comics and Biography Comics

*Ana Cláudia Munari Domingos and José Arlei Rodrigues Cardoso*

#### **Contents**


#### 4.1 Introduction

The understanding that all media are multimodal and intermedial in some sense and share basic characteristics among them is now fully accepted by scholars at a time when studies focus on understanding the interaction between them. Comics are one of these originally intermedial arts, since

A. C. M. Domingos (\*) • J. A. R. Cardoso

University of Santa Cruz, Santa Cruz, Rio Grande do Sul, Brazil

<sup>©</sup> The Author(s) 2021 79

L. Elleström (ed.), *Beyond Media Borders, Volume 2*, https://doi.org/10.1007/978-3-030-49683-8\_4

meaning is constructed through the combination of several media, verbal and nonverbal. This is done through different qualifed media and their modalities, such as the graphic image.1 One of the characteristics of the perception of comics as qualifed media2 occurs quite simultaneously with their mediation. The combination of verbal and nonverbal languages found in comics, associated with the structure of the frames, is recognized even before the reading takes place (by word or image in this sense). This is how we associate the sequential combination of verbal language and graphic image immediately as comics, although comics media products may differ greatly in their qualities and functions. The same cannot be said about verbal literary narratives, for example: in this case, we cannot defne merely at a glance whether we are looking at a novel, a tale, or youth literature. With ballet, a quick look at the ballerina's tutu is enough for us to recognize the genre. However, as soon as the dancer begins the movements on the stage, this dancer can be performing a role in a play or even street dance.

In this chapter, our interest is not to analyze this media combination in comics *per se*, but rather, to look into the strategies used by sequential art that prompt the term "journalism comics", that is, how its modalities and modes differ from "fction comics" in order to refer to the extradiegetic world. When analyzing the difference that seems most basic to us in comparing these two sequential art forms—the literary and the journalistic the distinction between "fction" and "history" immediately emerges. Another issue that arises from this comparison is the distinctions and similarities between biography comics and journalism comics. Although biography can be understood as a journalistic genre, the distinctions between these two at times can highlight the notion that it is part of the journalistic strategies. In this sense, the analysis of media products proposed here focuses on the strategies of indexicality established by journalism comics, which puts the semiotic modality of comics in perspective.

Our purpose is neither to analyze the *conditio sine qua non* of comics, nor to add complexity to basic concepts that are still the object of reflection and controversy (although they have already been settled in different areas), but to analyze the ways in which journalism comics and biography comics create indexicality through intermedial relations. We have selected some basic terminology to aid in the understanding of the ideas to which the concepts refer. Some additional clarifications are explained in footnotes. Lars Elleström's concepts of analysis will be discussed in section 11.3.

#### 4.2 A Short History of Comics as a Qualified Media Type

The tactic of narrating facts by means of successive drawings, which requires the sequential reading of images, is quite ancient in human history. To illustrate, one can refer to cave paintings in prehistory, ancient Egyptian hieroglyphics, mosaics, frescoes, and tapestries from the Middle Ages. As a specifc example, one is reminded that in 1064, the *Bayeux Tapestry* told the Norman conquest of England in 58 scenes embroidered on 70 meters of fabric. Also, the *Bible Pauperum*, directed to illiterate Christians, preached religious faith from a series of woodcuts depicting scenes from the Old and New Testaments, focusing mainly on the dramatization of the life of Christ. And the *Harlot's Progress* series of paintings, produced by William Hogarth in 1730, consisted of six sequential paintings chronologically telling the life story of a prostitute in London. These classic examples of sequential narratives can be considered the precursors of comic books, which would have been recognized early in the print works of writer and illustrator Rodolphe Töpffer. Considered by many to be the great pioneer of comics—with the creation of *M. Vieux Bois* in 1827—Töpffer framed his work in the form of a caricatured novel, seeking to distance himself from the role of book illustrator. Töpffer's pioneer work was one of the frst integrations between literature and graphic imagery to go beyond the merely illustrative function that nonverbal images had in their interaction with the word in literary works.

This hybrid nature of comics, derived from media combinations of verbal and nonverbal languages, leads to an increase in the complexity of their conceptual defnition. Eisner emphasizes that the art of comics is an act of aesthetic perception and intellectual effort, since the reader needs to exercise verbal and nonverbal visual interpretative skills, considering that the rules of painting (perspective, symmetry, brush stroke) and literature (grammar, plot, syntax), for example, mutually overlap (Eisner 1995a: 8). The author believes that both functions are irrevocably intertwined, since "sequential art is the act of weaving a fabric" (1995a: 122). Although McCloud defnes comics as "juxtaposed pictorial and other images in deliberate sequence, intended to convey information and/or produce an aesthetic response in the viewer" (1994: 9), it is clear that the idea of sequential art is the only defnition we will need to understand comics. Still, for many authors, such as Chinen (2011), the mere fact that the defnition of sequential art is also applied to cinema is enough for it to be refuted in comics. When we connect the previous statement to Elleström's ideas (2020), we come to the idea that the spatiotemporal modality is narrative-sequential, though there are differences among the other modalities that make these media different.

This conceptual vagueness is justified; capable of assimilating and transforming the languages of various media, such as cinema, literature, theater, and painting, comics also adapt easily to new technologies, and regularly adjust their method of production and meaning to available creative platforms. This is the case of their specifc relationship with journalism, since it was on the pages of newspapers that comics established themselves as a media product, before fnding their narrative space and consolidating themselves as recognized qualifed media.

As far back as the mid-nineteenth century, newspapers were known for their pages with informative and opinionated material in the form of large blocks of text. Even before this period, in 1796, with the creation of the lithographic printing system, readers could access the news combined with visual elements. Created by Johann Alois Senefelder, lithography simplifed paper printing through a planographic process and made it easier to add drawings, thus changing the way newspapers were read. This "visual addition", seen as an informative complement, enabled the emergence of periodicals, especially in the post-revolutionary France of 1830, which had their greatest attraction in caricature. According to Bruchard, in her preface to Daumier's book (Bruchard 1995: 4):

Caricature was part of a movement in which a whole generation of artists, conveyed by an increasingly agile press, were able to establish a way of observing, describing and judging what was trendy in the modern sense of the term, with the scope, impact and media developments that it maintains until today.3

The idea of caricature was based on the publication of prints containing political satire and social criticism, mainly targeting local rulers and politicians and their relations with the community. Honoré Daumier was one of the biggest names in caricaturism, devoting himself to political caricature and the defense of liberal ideals (Bruchard 1995: 4). His drawing "Gargantua", which criticized King Louis-Philippe, "made him spend six months in detention, frst in the Sainte-Pélagie prison, then at Dr. Pinel's nursing home"4 (Bruchard 1995: 4). The author Philipon published a sequence of drawings in the 1834 newspaper *Le Charivari* that also criticized the king in an iconic way: the king's face slowly turns into a pear—the French word *poire* meaning pear, but also meaning foolish. Fined for the publication, Philipon ironically repaid his debt by selling the same print separately (Bruchard 1995: 9).

As for the lack of resources for the visual recording of facts, since photography was still incipient,5 illustrations became an indispensable narrative and informative element of the press. This is why the idea of illustrated journalism became popular in the most diverse segments: local news, big parties, social events, and mainly war coverage started to be narrated with the aid of illustration, though not necessarily sequential. In the USA, the press began to cover actions such as the American Civil War in 1861 with illustrations that were independent from opinions regarding the journalistic event—they merely portrayed different moments of action, such as open battle scenes and the personifcation of heroes. In Europe, so-called special correspondents gained recognition, now covering illustrated wars and world conficts. In this context, great names in illustrated journalism emerged, such as William Simpson (*Illustrated London News*), Sydney Prior Hall (*The Graphic*), and Melton Prior (*Illustrated London News*). In Brazil, the journalist Angelo Agostini was already developing modern concepts of the factual, documentary, and investigative idea of journalistic reporting with works such as "Scenas da Escravidão" (Slavery Scenes), published in *Revista Illustrada* in 1886. His historical comic books critically portrayed the brutality of slavery and denounced, in a crude, documentary form, the reality of punishment, torture, and murder committed against black slaves on farms, shortly before the signing of Lei Áurea, the Golden Law, in 1888. However, in the late nineteenth century, the improvement of photographic fxation, reproduction, and printing techniques was essential for photography to gain the pages of newspapers as an illustrative and informative feature of reports. The use of photography then provokes "a crisis of readaptation in the universe of representational art, 'deprived' of realism by another realism" (Sousa 2000: 24).

For photojournalism, conquering movement proved to be of vital importance, since it enabled one to 'freeze' action, impress it in an almost realtime image, capture the unforeseen, gain control of the snapshot and, with it, hint at the idea of truth: that which was thus captured would be true; the image would not lie (note, however, that although the snapshot allows for more 'sincere' and spontaneous photographic representations, the photographs are nonetheless representations). (Sousa 2000: 29–30)

At this point, photography began to occupy a privileged spot in journalism due to its indexical and iconic capacity to represent reality. This implied that the photographic representation would now be seen as an imaginary documentation of the world through a "realism" that painting and drawing could not portray (Sousa 2000: 33). Furthermore, "it also benefted from the notions of 'proof', 'testimony' and 'truth', which at the time were deeply associated to photography, deemed as a 'mirror of the real'" (Sousa 2000: 33). Curiously, during a photographic exhibition in Paris in 1855, photographer Franz Hamfstangel exhibited, for the frst time, retouched proofs of negatives, raising questions about the very essence of journalistic photography. According to Sousa, "if Hamfstangel invented the retouching of the negative, it also opened the door to the manipulation of the photographic image by trickery" (2000: 32).

With photography occupying the space of the illustration of reality in journalism at the time, the emerging media of comics adjusted to newspapers as a product of entertainment, based on fctional narratives. More specifcally, comics sprang from a journalistic rivalry6 between William Randolph Hearst, owner of *The New York Journal*, and Joseph Pulitzer, owner of *The New York World*, who were fercely vying for the market's growing advertising dollars. Considered by most critics to be the frst comic book, Richard Felton Outcault's *Hogan's Alley* series, released in 1896 in *The New York World*, portrayed the daily lives of various exotic and caricatured characters living in a New York alley. Among them there was the boy Mickey Dugan, who became known as Yellow Kid7 because he wore a yellow nightshirt on which criticism was written. The series established the new cultural and commercial advertising production of comics.

Comics then began to dispute available spaces in the colorful Sunday newspaper supplements, which were mostly a hit among children. In 1907, author Bud Fisher proposed the creation of a "strip-shaped" comic with a generally horizontal fxed format with few frames, facilitating both simultaneous production and distribution to various newspapers. With daily delivery and format limitation, the comic strips began to focus on the theme of humor, portraying comical situations that resembled jokes with generally unpredictable outcomes and simpler themes. This funny style of comics was so strong and infuential that the expression comics is used to this day to specify any kind of comics.8 Parallel to comic strips as entertainment, all kinds of newspaper cartoons which assimilated the ideas of the old caricatures consolidated their role as political and social criticism, becoming tools of opinionated journalism and contestation.

With the emergence of a consumer comic book market, the most successful newspaper comic strips began to be compiled and printed in book form, then called comic books, the beginning of the form we know today. In 1938, with the publication of Superman stories, the golden age of superhero comics was born, ignited by a society crushed by the horrors of World War II and eager for examples of heroes. As Eco (1972: 107) states, quoting Superman's example, "the hero equipped with powers superior to those of the common man has been a constant in people's imagination from Hercules to Siegfried, from Roland to Pantagruel all the way to Peter Pan".

In the 1950s, superhero comics, which were once examples of the values of unity, freedom, and overcoming diffculties, were challenged by rigid and distorted moral concepts at the time. It was in this context that the ideas of German psychiatrist Frederic Wertham, author of the book *Seduction of the Innocents*, were published and gained the spotlight in the USA in 1954, warning of a possible danger coming from comic books, and accusing them of stimulating juvenile delinquency. The destructive power of Wertham's book was immense and ultimately infuenced an aggressive comic book censorship campaign, which eventually became the target of congressional investigation. The result was the creation of a comic self-censorship code—the Comics Code Authority (CCA)—which regulated the industry and wiped out numerous outstanding magazine titles that did not ft the moral standards of the time.

It was in this scenario that underground-styled comics stood out, a counterculture movement aiming at criticizing the values of the time. The main themes of the movement were sex, drugs, rebellion, and the Vietnam War. The underground comics intended to establish an anarchic and critical style, resulting in the development of misleading and irreverent characters and the creation of authorial and challenging stories based on the lifestyle of their authors.

In general, the printed newspaper never proved to be a receptive space for comics that featured longer and authoritative narratives: the preference was usually for the objectivity of the event in news production, which pointed to a journalistic need for informative material with daily or weekly update. Even when there was the possibility of developing longer and more investigative stories, comics were not a production priority due to the time required for production, delays, and the costs involved.

It was with the rise of underground-style comics reviews, usually published in magazines and books, that nonfction author comic books were consolidated, pointing toward the appropriation of the real inherent in history, journalism, and biography. Turning to biographical and autobiographical stories, comics began to concern themselves with portraying daily life drawn from the experiences of the author or the observed character, in the form of personal stories, family stories, or common dramas often intertwined with great facts of history. In this context, involvement with the characters' psychological, sentimental, and motivational issues emerges—illustrated by memory fashbacks—whose subjective nature reveals interpretations and opinions made in a very personal and particular way. The presence of this aspect in comics media products can be analyzed in different ways.

In comics, the very idea of the biographical narrative establishing itself as an element of social and cultural representation—while also acting as historical document and rescuing the past and contextual analysis of an era—eventually strains the limits of journalistic narrative. Since comics use iconic representations of reality, they seek to legitimize situations and contexts through indexical representations, such as graphic images of documents or factual situations, highlighting important and recognizable details to create models of reality representation.

This is how biographical comics often use journalistic resources to document this reality, trying to ensure the credibility of the reader, who needs to make a plausible connection between verbal and nonverbal elements and accept them as valid representations of a factual narrative. Thus, many comic book authors use representations of photographic images, excerpts from newspapers and offcial documents, as well as recurring methods of researching journalistic information, such as interviewing and investigating facts from different sources. This is the case of Will Eisner, Art Spiegelman, and Joe Sacco, authors who subtly or directly incorporated these features into their narrative styles and consolidated a path that would later be known as comic journalism.

Will Eisner was one of the precursors in the creation of biographical comics, portraying his life in New York City, especially in the autobiographical works *The Dreamer* (1986)—which presents a faithful portrayal of the universe of comics creation during the 1930s—and *To the Heart of the Storm* (2000b)—which portrays his life trajectory, especially his upbringing in a poor US environment in the early twentieth century. In this work, Eisner uses a continuous graphic feature to show characters reading the newspapers of the time, whose headlines project the reader in time and space, chronologically showing historical facts pertinent to the plot.

The urban theme of his biographical stories is also present in comics such as *A Life Force* (2001)*, Dropsie Avenue* (1995b), and *New York City Life in the Big City* (2006). In these works, Eisner connects the reality of his characters, who intersect in various stories, using real places and facts that appeared on the pages of the city's newspapers. For example, in *Invisible People*, according to Kitchen (2009), Eisner was inspired by Carolyn Lamboly's true story, found in a *New York Times* report about her suicide in 1990 and the motivations for her desperate gesture. The book *A Contract with God* (Eisner 2000a), which popularized the term "graphic novel"9 and was one of the stories responsible for the recognition and consolidation of comics life narratives, based the creation of its controversial characters on real experiences, according to the author:

In this book, I have attempted to create a narrative that deals with intimate themes. In the four stories, housed in a tenement, I undertook to draw on memory culled from my own experiences and that of my contemporaries. I have tried to tell how it was in a comer of America that is still to be revisited. The people and events in these narratives, while compounded from recall, arc things which I would have you accept as real. Obviously in the creation, names and faces were rearranged. (Eisner 2000a: 8)

In *Maus*: *The Story of a Survivor* (2003), author Art Spiegelman fuses the boundaries between autobiography, biography, and journalism by interviewing his own father, in an attempt to reveal his family's martyrdom during World War II. In reliving the horrors of the Nazi holocaust, Spiegelman eventually reconstructs fragmented identities of Jewish people destroyed by Nazi ideals, picturing each nationality in the form of an animal. Considered one of the most important comic books ever, *Maus* became a reference by winning the esteemed Pulitzer Prize in 1992.

However, it was journalist and comic artist Joe Sacco who really combined comic book narrative and features of the story as an investigative narrator, fguratively represented by the reporter's graphic image. Portraying himself as a reporter in *Palestine* (2003), Sacco enters the Israeli-Palestinian combat zone to hear the voices of war victims, usually conditioned to statistics in history books and the world press itself. With a closer look at contextual information, the author shows a subjective journalistic report, which provides a "face" to the invisible Palestinian characters,10 until then doomed to the forced silencing of identity, always associated collectively with terrorism.

Sacco's work, which includes many graphic novels about wars around the world, is based on his journalistic investigations, withdrawing from interviews and sources of information, in which the author portrays himself and sees his function as a journalistic narrator. By contextualizing facts and journalistic and historical data, Sacco assumes a documentary style without giving up his point of view. Subjectively, the author details his actions and his creative processes: narrating his efforts as a journalist, visualizing his diffculties in obtaining testimonials, and telling his own story within a confict that involves different and contradictory positions.

#### 4.3 Media Representation and Transmediation in Journalism Comics and Biography Comics

Some boundaries regarding the concepts of media and intermediality are important for this article, although it is quite specifc in its objective: to analyze the semiotic modality of journalistic and biographical comics as a way to understand the strategies of building iconic and indexical representations through graphic images. Before moving on to the analysis, we will investigate some of these boundaries. What we understand as media and media modalities are clearly circumscribed in Lars Elleström's "The modalities of media II" in this volume (2020). However, it is essential to elucidate how our analysis appropriates these concepts.

First of all, based on the understanding that sequential art is a combination of several media—verbal and nonverbal—this chapter takes "journalism comics" and "biography comics" as qualifed media types in the terms defned by Elleström; he states that "[q]ualifed media types are simply categories of media products grounded not only on basic media modality modes but further qualifed" (2020: 55) Although the distinction between basic media types and qualifed media types is not absolute since, as Elleström points out, they are dependent on historical and cultural criteria as well as communicative purposes, it is not diffcult to understand this differentiation related to the media products analyzed here. However, two points need to be outlined.

The frst concerns the fact that when we compare "journalism comics" and "biography comics" with "fction comics", similarities and differences stand out. The similarities arise from the comparison between the material, sensorial, and spatiotemporal modalities of these media types: they are all solid, visual, two-dimensional media products produced from static images—and they all belong to the "comic book" medium type. Moreover, they are all narrative, which implies a certain sequential structure that is fundamental for comics, differentiating them from, for example, illustrated books. Apart from dissimilarities in themes and style, their differences lie in their different modes in the semiotic modality. Journalism comics and biography comics both differ from fction comics in terms of the semiotic modality; whereas fction comics generally emphasize the symbolic and iconic mode, journalism and biography comics largely reinforce the iconic and the indexical modes. In accordance with Elleström (2020), it must be emphasized that these are categories of analysis; when described from their different semiotic modes, media products show themselves as different types of qualifed media, although their signifcance depends on the reader and even on the contract established with the text.

The second point that cannot be disregarded concerns precisely that contract. When published, the categorization intended by the editors and authors for their works—designated by the qualitative "journalistic", "biographical", or "fictional"—evokes a certain representation on the part of the readers. In journalism and biography comics, the classification in the catalog sheet causes the mediation of objects—people, things, events, and media—by both verbal and nonverbal media, to build indexical representations of the objects of reality. This means that, although semiotic modes are not absolutely distinguishable—the symbolic, the iconic, and the indexical work in collaboration—one of the semiotic functions is highlighted by the intention of the text, whether journalistic or biographical (or fictional). This is an example of what Elleström, in "The modalities of media II" (2020), points out as the communicative function of semiotic modes: although the semiotic modes are based on presemiotic modes, it is only in semiosis—representation of meaning—that meaning is constructed.

Thus, our analyses in this article focus on the semiotic modes of media products, whose symbolic and iconic representations of graphic images *simulate* the indexicality of journalism and biography. Hence, we offer a semiotic interpretation of media products such as journalism comics and biography comics. Nevertheless, one pertinent detail places this analysis on the axis of intermediality studies: the fact that, to tell a journalistic story, comics appropriate journalistic strategies. In other words, they represent and transmediate the qualities of the media type journalism to simulate its characteristic indexicality. This strategy is also typical of biography comics, but in a different way. Biography comics are more related to historical narratives, rendering the representation of documents and photographs concerning people and their actions within a recognizably historical context the most important aspect. In this sense, the representation of media in biography comics acquires another meaning.

The concept of representation here gains a second connotation: specifcally, the representations in which the object represented is another media type or another media product—an intermedial phenomenon. Elleström calls this "media representation", which is one of the two forms of media transformation, or intermedial transmediality (the other one is transmediation):

My frst step toward the goal of forming a conceptual framework that facilitates a detailed analysis of transfers of media characteristics among media is to establish a distinction between *transmediatian* (repeated representation of media traits, such as a children's book that tells the same story as a computer game) and *media representation* (representation of another medium, such as a review that describes the performance of a piece of music). (Elleström 2014: 12)

As we intend to show in this analysis, media representation is a fundamental trait of journalism comics, since it is used to refer or allude to journalistic media through the presence of newspapers, television, computers, and cell phones being handled by characters (simple media representations) and to create more elaborate representations of specifc media products or qualifed media types (complex media representations). Based on Charles Sanders Peirce, Elleström distinguishes three types of representation: deiction (iconic), deiction (indexical), and description (symbolic) (2020: 51). However, it is quite diffcult to categorize a complex media representation in these terms, especially when one medium imitates another.

In complex media representations, the content of the *quoted* and sometimes imitated medium is crucial to the representation of that medium. The strategy of representing and imitating other media is generally intended to point to the extradiegetic factual reality—producing indexicality. In the case of simple media representations, there is a certain invocation of the very type of qualifed medium—journalism. As for biography comics, the indexicality is shown more by the indexical representation of people and events, signs that often appear "inside" other media, as we will demonstrate below.

One of the characteristics of media representation is that the "frame" of the represented medium can be perceived; that is, we can view the represented medium as a medium inside another, in some of its modalities. The graphic image of a printed newspaper on the table is easily recognized as the "printed newspaper" media type (which is not the case in transmediation, when the frames become indistinct or disappear). A cell phone in the hand of a character in a movie can mean simply that he is a person living a common routine these days; however, it can also mean that he is sending someone a message. That message, if perceived, like any other media product, may be more or less relevant for understanding the story of the flm.

In complex media representations, however, represented media characteristics dissolve in the representing media product and thus insert details that are generally more important for meaning representation. The detailed description of a particular section of a newspaper, for example *The Times*, is not just meant to represent *any* newspaper as a qualifed media type, but a specifc media product, invoking idiosyncratic characteristics typical of that product, such as the ideology of its editorial line. Such a representation is probably simultaneously iconic, indexical, and symbolic. Another example: in Emmerich's 2011 movie *Anonymous*, there are several simple representations of audiovisual media types. Theater, for example, is iconically represented by images of theater buildings and stages. Some representations are more indexical, as when the Globe Theatre is represented through what the reader perceives as real connections, and symbolic, as when there is a character in costume, ready to act, or when a character says "theatre". There are also simple media representations in references to Shakespeare's play titles. Furthermore, there are complex representations of Shakespeare plays in flm, setting in motion the iconic, indexical, and symbolic modes simultaneously—in words, in costumes, in dialogues, in the interpretation of the characters, and in theater settings. In this way, the flm appropriates features of Shakespeare's plays in various ways.

The boundaries between complex media representation and transmediation are sometimes subtle. A scene from a Shakespeare play being played in a movie also needs to adapt the play's dramatic text (which becomes part of the script) to that scene (the scenic text that begins to integrate the movie). If it is completely "staged in" the movie, on a "flmed stage", we can say that there is an iconic simple representation of both a qualifed media type (Elizabethan theater) and a media product (Shakespeare's play)—as well as a transmediation of Shakespeare's play.

Transmediation, according to Elleström (2013), happens when content is re-mediated. A play can be adapted for the cinema; a dramatic text may be performed on the stage; or it may be read aloud to an audience. In transmediation, it is not the previous media that appear, but their content, which is transmediated—adapted to the new media and their modes of mediation and representation of cognitive import. Given the limitations of comic book affordances, which can only adapt media types that somehow ft their characteristics—solid, visual, two-dimensional, made up of static images—media representations are more common than transmediations in comic books, especially considering the need to show the frame, that is, to make a reference also to the qualifed media type "journalism" to simulate its indexicality. In this sense, it is not enough to represent again the content of a particular television report; it must also be shown that the content was mediated by a television report and can be said to be journalistic content.

Based on our interpretation of Elleström's (2014) concepts of media representation and transmediation, we will now analyze three narratives of journalism comics: *Footnotes in Gaza* (Sacco 2009), *Il mondo di Aisha* (Bertotti 2008), and *Le photographe* (Guibert et al. 2003, 2004, 2006) and three narratives of biography comics: *Dotter of Her Father's Eyes* (Talbot and Talbot 2012), *Persepolis* (Satrapi 2004, 2005), and *To the Heart of the Storm* (Eisner 2000b).

#### *4.3.1 Simple Media Representations*

The comics analyzed range from simple representations of technical media to simple representations of specific media products. Interestingly, in view of this need to construct indexical representations, generally the media products shown are not fictitious but "real", often recognizable by the reader as literary books. *Dotter of Her Father's Eyes* brings several examples of this, presenting different and varied simple media representations already in the paratexts, where there are graphic images of photographs that show some documents that, later in the narrative, will be represented again, this time by illustration. An indication of this is that these photographic images state the initial tone of the qualified media type that is built there—in this case, a comics autobiography—signaling that the characters exist outside the narrative, that is, outside the images, through the copy of the photograph. Photographs are important documents in themselves, representing reality, and, despite the contemporary possibilities of manipulation, they are still the best way to build this contract with the reader. Documents, to the same extent, when presented through photographic images, reinforce this "real existence" of the subjects portrayed—falsifying a document is quite different from falsifying a photograph—and thus the narrative acquires "biographical" quality. In this sense, we can understand that even the illustration, which can be perceived as an iconic representation, becomes an indication—an indexical media representation. Perhaps we may also think that it is an iconic representation—the illustration of the photographic image—of an indexical representation—the photographic image (a graphic copy of the photograph).

Still, in *Dotter of Her Father's Eyes*, technical media such as television and radio, newspapers, magazines, and even comics serve to contextualize the historical moment and also create the atmosphere in which the characters of the story live, people who have an intense relationship with the culture and the arts. However, in addition to representing the media as means and tools of communication, they are also used to indicate real situations involving the factual context of the story. In the illustrated scene of the protagonist Mary's conversation with her mother (Fig. 4.1), for example, we see a television set in the background, showing an image that stigmatized the Vietnam War: Eddie Adams's "The Execution" photograph, winner of the Pulitzer Prize of 1969, which shows a summary execution—a soldier shooting a prisoner's head—in front of an NBC TV camera. The facts—the explosion, the war—are not crucial for the story (the narrated fable), but rather indicate truthfulness. Thus, the media product representation is not very important to the meaning of the work apart from referring to the extradiegetic reality, reinforcing the biographical tone. The same can be said about the books and comics that characters in *Dotter of Her Father's Eyes* have at hand. Most of them are simple representations of media, although they show titles and details of specifc and real media products that aim to indicate the "real" readings of the biographed characters. There is even an addition that, according to the narrator, does not correspond to the truth, since the illustrator added a book

**Fig. 4.1** A television showing an image that stigmatized the Vietnam War (Talbot and Talbot, *Dotter of Her Father's Eyes*, 2012: 56). Copyright Mary M. Talbot and Bryan Talbot. All rights reserved

that is not one of the narrator's favorites, but the illustrator's. Thus, the representations of books cease to be mere illustrations of books in general (technical media) and become indications of works that help to defne the characters as real people.

In *Persepolis*, which is also an autobiography comic book, simple media representations pursue these same goals. When Satrapi's mother, the protagonist-narrator, is photographed by a journalist, there is a painting showing the moment of the click, which is only evident as a photograph when shown "inside" the newspaper (Fig. 4.2). The fact that it is in a newspaper, and then in a magazine, demonstrates the problem of exposing the mother's image in a protest, but the content of the media product matters as much as the representation of the media type newspaper and

**Fig. 4.2** The moment the photograph is taken, shown "inside" the newspaper later (Satrapi, *Persepolis: The Story a Childhood*, 2004: 5). Copyright Marjane Satrapi. All rights reserved

magazine. It is through the publication of this photograph that Satrapi fnds out that she is the great-granddaughter of an ancient Persian king, an important fact for the history of the country. Delivering the reality of the outside world also occurs with representations of television, which often comes up to show specifc TV programs and their content, reference that are also made verbally. The representation of media types as vehicles of information is important to signify the sphere of the Iranian government discourse delivery, when verbal media are required to refer to the content of that discourse. Literature representations are also important in *Persepolis*, as it is through literature that Satrapi tries to understand the world and her role in social and political transformations. There are simple representations of books—on the bookshelf, on the bed, on the nightstand, in Satrapi's hands—whose titles and content are not noticeable, which highlights both the technical book medium and the literature media type. There are also representations of specifc media products, when the titles of the books are shown or when the character mentions them. These representations are often complex, as we will explain in the following section. The presence of pop culture and market consumerism, characteristic of Western culture, is often represented by images of posters and album covers. Marjane Satrapi, the author, is a graphic illustrator and her relationship with drawing and painting is brought about by simple representations of the practice of drawing by the protagonist.

In *To the Heart of the Storm*, as in the two previous autobiographies, the profession of the biographer is important. Although it is an autobiography, the purpose of the representation of the printed media here is similar to that of journalistic comics, since the biographer is the cartoonist Will Eisner who worked for newspapers—which explains the many representations of the printed media type in the story. Being read or handled by the characters, for sale at newsstands, or thrown on the streets, the newspaper is more important as a qualifed media type (as a vehicle of "truthfulness") than as a simple technical medium. However, for the biography, the newspaper also fulflls a chronological and informative function. It associates the protagonist's life to his relationship with illustration, and his family's life to the historical context—especially the economic factors surrounding his father's business (specifcally the crises occurring before and after World War I). The arts of illustration and painting are also part of the narrative, revealing many of their processes and techniques to illustrate this biographical datum. The idea of the representing reality in comics is also treated symbolically by painting: an artist destroys his painting when trying to scare off a fy that had landed on the canvas. However, the fy was just a drawing, intentionally created to engender confusion.

The author's autobiographical refections start with the representation of photography by illustration. This reinforces the above-mentioned idea that biography comics build indexicality from a relationship between the graphic image of photography and real-life subjects and events. The atypical frames of a photo album are used to reinforce the frame of the photograph itself as a media type that represents reality and thus triggers the memories of the character.

The simple representations of technical media have partly different meaning in *Footnotes in Gaza, Il mondo de Aisha,* and *Le photographe*. In these cases, in journalism comics, the iconic depictions of printed newspapers and television sets through illustrations serve as indices of activities to produce history and reinforce the importance of journalistic media types. In other words, while in biography comics the graphic representations of photographs of documents and people are vital for producing indexicality, in journalism comics, this indexicality is given by the representations of the technical media themselves.

In *Le photographe*, a narrative about an expedition by Médecins Sans Frontières (MSF) by photographer Didier Lefèvre, the act of photographing, shown by the photographic equipment, is often highlighted to represent that it is a photographic report, even though it is constructed by illustration. In this case, the indication is given by the parallelism between the two graphic images, the illustration and the photographic image of that moment, or of the same people who were portrayed. Therefore, it is not a transmediation of a photographic image to illustration, but an iconic representation of other possible angles of that situation, as if the photographer took photographs through the illustration. This idea is corroborated by the fact that the photographer is often represented with a pen and a notepad in his hands, suggesting he was drawing the scene. When this is opposed to photographic images that show the same places and people in other perspectives, representation acquires indexical value.

In *Footnotes in Gaza*, the journalist also appears photographing and writing on a notepad, and, in the same way, this reinforces the agency of journalism, inserting indexicality. This agency is also marked by the representation of the media type interview, performed through various strategies with the graphic image. From the beginning of the narrative, the protagonist, journalist Joe Sacco, stresses the importance of fnding the people who have experienced the events he wants to report. Once he fnds these people, he begins to interview them: the illustration shows this meeting, while the protagonist holds a tape recorder, notepad, and pen. Rather than illustrating a journalistic practice, the representation of the media interview is primordial to the narrative construction of the historical memory of the war, as the details and contradictions they bring show the uncertainty of information on either side. The simultaneous graphic image of different opinions (sources) about the same event is an indexical representation of journalism, showing this characteristic of the journalist's search for different versions; thus, the likelihood that journalism does not bring up one single truth. Sacco, the narrator character, is present in each of these interviews in the illustration, usually recording the audio or taking notes in his notepad.

The details of the graphical representation of the interviewees simulate the indexical representation of photography, making each one of them quite idiosyncratic, not to mention human, real, in their subjective characteristics. On the other hand, the graphic representation of the protagonistnarrator, journalist Joe Sacco, is quite iconic, almost caricatured, compared to his photographic representation, present in the digital version of the work and on the internet. This seems to be an attempt to highlight precisely "the journalist" among the other characters, thus distinguishing him visually.

In *Il mondo de Aisha*, photography acquires considerable importance as a legal resource as it receives, in the introductory pages, a subtitle: "A work inspired by Agnes Montanari's travel reports". Agnes Montanari is a photographic reporter who conducted interviews and photographs of Aisha and other women oppressed by Yemeni customs. Comics artist Hugo Bertotti adapts this story to comics11 by illustrating Agnes meeting Aisha during her job of shooting photos and interviewing people. In the comics paratext, there is a text by Agnes Montanari describing her meeting with Aisha and the other interviewees. The comics are divided into several stories and, in each of them, a photo of the character and the name introduce the narrative, showing their relationship with the extradiegetic reality—based on the photographic representation. At the end of *Il mondo de Aisha*, the photos of all interviewees are shown. Both the camera and the act of photographing are very often represented. There are also representations of video footage and interviews for television programs by one of the characters.

#### *4.3.2 Complex Media Representations*

Some media representations become more complex as they *enter* the structure of the represented media product, producing meaning from the qualities of other media. Focusing on the work of the photographic reporter in *Il mondo de Aisha*, for example, through the representation of the journalism agent, as well as in *Footnotes in Gaza* and *Le photographe,* we can see that indexicality is built by presenting the act of capturing reality, "as if" the reader were watching the news report through the "screen" of comics. This "as if", treated by Irina Rajewsky (2005, 2010) as "intermedial reference", can be understood, according to Elleström (2013), as a complex representation of media. Another type of *simulation* can be seen when Agnes Montanari sends Aisha her own photos: this *sending* of photos is symbolized in the verbal media by the word "send", and illustrated in the construction of a photo trajectory that simulates a journey through "internet waves" (Fig. 4.3).12 The photographs, when received, are represented by the photographic images framed on Aisha's computer screen. Another complex representation of media in *Aisha´s world* lies in the interweaving of the characters photographic images with the comics pictures, creating sequences in which both media types complement each other in a combination of media that makes it diffcult to distinguish which represents the other.

Although the literary work is authored by Didier Lefèvre as a whole, *Le photographe* has the artistic participation of Emmanuel Guibert (script and drawings) and Frederic Lemercier (colors and montage/diagramming), which then produce a narrative from different techniques of photographic and cartoonistic composition that comes from different views and ways of expressing the world. Thus, we can imagine that the protagonist-narrator the photographer—brings together a little of each of these *arts,* so much so that when he introduces himself to the reader at the beginning of the story, he does this with his iconic, illustrated image. It is only at the end of the story that his photographic image will appear addressing the reader. Prior to that, the only photographic image in which he appears prominently is in a symbolic representation, where he is photographed in the mirror without showing his face. Sometimes the act of photographing is illustrated to show the angle in which the photograph was taken, from which a photographic image is shown in the sequence (Fig. 4.4). Thus, formally, the whole story is a complex media representation, combining and blurring the limits between the arts. In this and other cases of complex media representation, it is possible to perceive the "work" that gives rise to the "image", allowing a glimpse of metalanguage.

In *Footnotes in Gaza*, Sacco portrays the daily work of journalism in the dangerous informational coverage of the war, illustrating several communication professionals performing a vast array of practices. Conversely, he does not avoid criticizing his profession based on the choices made: journalists seem not to care about people, but only about the sensationalism that the event can provoke, as in the representation of photographers who only focus on people in moments of pain and despair for more impacting images. Sacco's harsh criticism also applies to moments of rest: journalists

**Fig. 4.3** The *sending* of photos is symbolized by the word "send" and illustrated by a photo trajectory (Bertotti, *Il mondo di Aisha*, 2008: 134). Copyright Ugo Bertotti. All rights reserved

**Fig. 4.4** The angle in which the photograph shown later was taken (Guibert et al., *Le photographe. Tome I*, 2003: 47). Copyright Emmanuel Guibert, Didier Lefèvre and Frédéric Lemercier. All rights reserved

talk and exchange ideas about work and the nature of the information. The news seems to bring nothing new, because the war always seems to be the same and no one is impressed any longer. Since the "agenda had died" (Sacco 2009: 124), the "hope" is that a new war will break out to stimulate journalists. Here, the complex representation of journalism media, at times when it represents journalists gathering in different situations, makes sense in its own story and even in its own essence, since it emerges from a footnote, that is, the event referred to in a footnote that did not become news because it was not covered by journalists.

Another complex representation of media in *Footnotes in Gaza* is in the illustration of Moshe Dayan's book *Diary of the Sinai Campaign.* Not only does the cover show it in minute details, but there are depictions of excerpts from the book. For example, there is a page with a map that not only shows a detail of Gaza's historical geographical division, but also reveals the need for geographic knowledge for such a story. Graphic elements widely used in *Footnotes in Gaza*, such as maps and other spatial geographical representations, are important since history is precisely about a war over a territory. The incident reported in *Footnotes in Gaza* is not in current history books and is only recalled by a brief citation in a footnote in a UN report,13 which states that Israeli soldiers had panicked and opened fre against the rushing crowd. However, the report issued by Israel is quite different, according to Sacco, who uses this document as a starting point for his investigation.

The illustrated representation of documents (offcial and unoffcial, according to the narrator), for example, the illustration of a detail in the previously mentioned UN report, also shows the intention of making it seem that the strategies for producing a report are really journalistic. Although we can understand some of the above-mentioned media representations separately as simple, they also represent the production of journalistic reports in complex ways, simulating a sort of "making of a documentary" rather than just narrating the story of the note about Gaza which is, as it were, within this documentary. It is sometimes diffcult to distinguish when the characters are in the present narrative and when they are in the past. Thus, there is a strategy of illustrating them with present and past traces side by side so that the reader can distinguish this. In this sense, there is a frame for the very narrative of that Gaza episode *within* the "documentary" *Footnotes in Gaza*.

In *Footnotes in Gaza*, other simple representations of media can become complex as the reader seeks to deepen the understanding not only of the narrated historical event, but also of the reasons why it has become a footnote. Many television show illustrations, for example, add sense to the story, such as news about the war at the time when the comic story was produced, and an interview with US President George W. Bush (Fig. 4.5). In choosing to illustrate violent scenes that were aired on television, or that specifc interview, there is an intention to produce signifcance related to the meaning of the story, as the narrator warns us: "Palestinian cameras do not hesitate to show the open wounds on the victims' chest" (Sacco

**Fig. 4.5** The television shows an interview with US President George W. Bush (Sacco, *Footnotes in Gaza*, 2009: 136). Copyright Joe Sacco. All rights reserved

2009:126). The reader needs to observe the illustrated scenes—not just the illustration of the television set. There is an iconic image of the zoomin on the President that carries indexicality due to the easy recognition of him, while the television frame disappears, transforming a simple representation into a complex one, and fnally a transmediation of the television show onto the comics page. The names of newspapers and television channels and programs as specifc media products are also important, as they have different editorials and ideologies. Thus, they are no longer just representations of technical media; they make sense in the setting of *Footnotes in Gaza* itself.

Biography comics also feature complex media representations, although they may not always be perceived if not supported by the perceiver's repertoire. *Dotter of Her Father's Eyes* includes several, held by the core intention of the work, which is to weave relationships with James Joyce's style and to make direct references to his work. However, there are many allusions that the perceiver must notice and understand.

The popular Irish folk song "Finnegan's wake", whose narrative inspired Joyce's *Finnegans Wake*, tells the story of a drunkard named Tim Finnegan who falls off a ladder, breaks his head, and is resurrected at his funeral when friends pour whisky over his dead body. Composed in 1850 by an unknown author, the story of the popular song is present in the frst chapter of Joyce's book and also in *Dotter of Her Father's Eyes*. In the photographic montage that ends the comics, the chorus of the song—"Wasn't it the truth, they told ye lots of fun at Finnegan's wake?"14—is subtly placed on the last page of James Joyce's *Finnegans Wake*, which reads "My cold mad feary father" (Joyce 1992 [1939]: 628), the same sentence that triggers the story of *Dotter of Her Father's Eyes* (Talbot and Talbot 2012: 3). The representation of the *Finnegans Wake* media product, therefore, is signifcant not only in the simple direct quotation of the passage but also in the signifcance of a circularity, interweaving in the sequential structure of the comics the stories of Lucia, Joyce's daughter, and Mary, the protagonist of the comics—and also the present and the past, in the connection between the end of Lucia's story and the beginning of Mary's.

In the introductory and fnal pages of *Dotter of Her Father's Eyes*, we fnd photographs of the drafts by Atherton, the protagonist's father, during the creation of his book about Joyce, *Books at the Wake* (1959). The photos show changes in Atherton's book summary and pen-written notes showing the creative process. Apparently, the notes are made on a printed copy of *Finnegans Wake*. Representing this process of creating the book, Mary weaves it into her relationship with her own father, who lives apart from the family, inside his offce, reading and paraphrasing Joyce. In this sense, this complex representation of the media product *The Books at the Wake* does not draw an intermedial reference related to Joyce's work itself, but primarily to the protagonist's own "structure" or "process of creation".

In *To the Heart of the Storm*, the sequential structure of the narrative simulates the fashback of cinema by illustrating the protagonist looking through a train window (Fig. 4.6). As he observes the landscape, memories of the moments lived in those city locations are triggered: the scenes of the past are now "projected" through the graphic images of the illustration. The train trip refers to the spatiotemporal mode of the story, in which the train window is the frame of this story canvas, as if a movie were played before the protagonist's eyes.

In *Persepolis*, the veil worn by women is an important graphic element to show the loss of identity, because it is very diffcult to identify the women when wearing it. To stand out, Satrapi presents herself illustrated in one of the panels; while, in the other, her image is cut out, alluding to a strategy of photography in which a cut can be made at the time of photographic recording or later in the process of development or in digital manipulation (Fig. 4.7). Thus, it seems that the "photographer" left her out of the picture with her friends. Another complex representation of media in *Persepolis* is in the illustration of the comics media product *Dialectical Materialism*, the protagonist's favorite story. In depicting excerpts of Iran's history, Satrapi reinvents it from her interpretation, weaving connections with her religious beliefs and thus creating her own dialectical materialism.

#### *4.3.3 Transmediations*

Transmediations are much less frequent in the comics analyzed. As we have already commented, this might be explained by the very characteristics of this kind of media transformation. First, transmediations depend on the possibilities of adapting the characteristics of one media to another. When the source and target media modality modes are the same, transmediation involves less adaptation, while the opposite may even make transmediation impossible. Secondly, we understand that transmediations of parts of media products can often function as complex representations of media, as the producer seems to be willing to make that "piece" dilute "within" the media product. As for representations of traits—or

**Fig. 4.6** The fashback of cinema by illustrating the protagonist looking through the train window (Eisner, *To the Heart of the Storm*, 2000b: 7). Copyright Will Eisner. All rights reserved

**Fig. 4.7** Satrapi presents herself, and her image is cut in the other illustration (Satrapi, *Persepolis: The Story a Childhood*, 2004: 3). Copyright Marjane Satrapi. All rights reserved

characteristics—of qualifed media types, these are (simple or complex) representations of media, as we have already shown here. As we have also commented, the boundaries between complex media representations and transmediations can be subtle. This is indicated by the fact that, in order to analyze the complex media representations and transmediations, it is necessary to interpret the media product in a more attentive and thorough way than merely perceiving the presence of other media. Triggering intertextual repertoires and knowledge about how media work is also important for understanding complex media representations and transmediations, and for recognizing media types and media products being cited, alluded to, or transformed.

By analyzing the photographic image of Satrapi's mother in *Persepolis* "inside" a newspaper page (Fig. 4.2), for example, we easily identify a representation of newspaper media—as a type of printed media—and also the representation of a product of photographic media "printed" in the newspaper. However, in the previous frame, the same image appears without the frame of the photograph, because the frame is constituted by the typical comics panel. This means that the photograph was frst transmediated into the comics through the illustration, and then transmediated into the newspaper that is represented in the comics. It is easier to understand this process if we accept the fact that this photograph exists outside the media product—in extradiegetic reality—and was actually published in a newspaper.

In *Dotter of Her Father's Eyes*, Mary, the protagonist-narrator, adapts excerpts from the biography *Lucia Joyce*: *To Dance in the Wake* (Shloss 2003) to her story. These adaptations are intramedial when she recounts parts of Lúcia's biography through the verbal, symbolic elements of the comics; and they are intermedial when she recounts the biography through the iconic elements of the comics (cf. the distinction between media translation and media transformation in Elleström 2020: 73–74). The case of photographic images is exemplary, as the illustrations often correspond to photographs inside or outside the book; they may be published on the internet, for instance. The scene illustrations of Lucia's dancing performances are transmediations of these photographs, but the reader must compare them, and scrutinize Lucia's biography and Mary's autobiography, to understand them as such.

For both biographical and journalistic comics, it is vital that the document or photographs represented by the photographic images (better yet, copies of the photographs) are then transmediated to the illustration, making the indexicality surpass the typical iconicity of the illustrations (since the graphic images are iconic signs of objects). This is because biography and journalism comics need to demonstrate that illustrated images are not merely imagined; they can also represent something that really exists. Thus, they show the photograph and, later, the same image illustrated. This is how, in *Dotter of Her Father's Eyes*, a piece of the content of a photographic image—Atherton's identity document—is transmediated for illustration (Fig. 4.8).

Transmediations can also be noticed when the content of a certain book or newspaper is *read*—shown—by one of the characters. The illustration of Mary Talbot reading Carol Schloss's book on the train reinforces the idea of a transmediation of the contents of the biography into the story being told in the comics. When Mary *listens* to the interview with American writer Sylvia Plath, broadcasted on BBC in 1962, the excerpt from the poem that Plath recites live—"Daddy, daddy, you bastard, I'm through" (Talbot and Talbot 2012: 36; Plath 1992: 222)—is transmediated again to written verbal media in comics. The

**Fig. 4.8** Atherton's identity document transmediated by the illustration (Talbot and Talbot, *Dotter of Her Father's Eyes*, 2012: 2–3). Copyright Mary M. Talbot and Bryan Talbot. All rights reserved

transmediation of this exact passage from the poem is important because, written just before Sylvia Plath committed suicide in 1963, it speaks of a girl with Electra complex and her father, who died while she still thought he was God.

In *In the Heart of the Storm*, the covers of some newspapers are quite noticeable in their details, showing—in addition to the names of the newspapers—titles, headlines, and even dates. It was not possible to research whether these covers are real or fctional, but it is not unlikely that they convey elements of newspaper covers that were indeed printed. Transmedia evaluation always requires that we carry out such comparisons between source and target media products.

In *Le photographe*, the proof is used as a narrative resource, in which the frame of the photographs is kept, but transformed in the proper sequence of the comics.15 The selection of the photos, made from the original proof by markings, indicating cuts in the photographs or the rejection of some of them, is maintained (Fig. 4.9). The inclusion of these rejected photographs maintains the dynamics of the proof media type and characterizes their integral transmediation to the comics. One can see that the proof's frame disappears as it turns into comics on some pages that are composed entirely of photographs from this proof. The proof transmediation, which ends the comics, includes pen-written annotations and organization labels,

**Fig. 4.9** The selection of the photos made from the original proof by markings (Guibert et al., *Le photographe. Tome I*, 2003: 62). Copyright Emmanuel Guibert, Didier Lefèvre and Frédéric Lemercier. All rights reserved

simulating the characteristic annotations of photographic evidence present in the proof. When photographs are isolated, taken from these cuts, we can still understand that they are transmediations as they become frames, when the photo frame becomes the frame of the comics.

The same happens in *Il mondo de Aisha*, when photographs of Yemeni women, whose testimonials compose the story, lose their frame as they are transmitted to the comics as photographic images, showing labels with their names on them. In another part of *Il mondo de Aisha*, transmediation happens the other way around: the illustration of a scene in one of the comics is transmediated to a camera screen, becoming digital photography (Fig. 4.10). Of course, one may also understand that by placing the image that "shoots" the illustrated scene "inside" the camera screen, the graphic image *simulates* digital photography.

As previously pointed out, complex representations and transmediations often merge, and when parts of media products are transmediated, these transmediations are often part of media representations. Frequently, rather than transmediating a specifc content of a media product, the

**Fig. 4.10** The illustration of the same scene on a camera screen (Bertotti, *Il mondo di Aisha*, 2008: 58). Copyright Ugo Bertotti. All rights reserved

objective is to represent media types or to show different media, understanding that the media, by themselves, represent the world; they are indices of reality. That is, the different mediations of the document (media representations) are highlighted, instead of its content (transmediations). In *Footnotes in Gaza*, excerpts from the UN document are sometimes transmitted to the illustration of printed pages, sometimes narrated by the journalist. At times, these graphic images of documents have titles in a different bold font that appear to be watermarks of the document but which, when transmitted to the illustration, are highlighted.

#### 4.4 Conclusion

We may have been redundant in some statements throughout this chapter, but case analyses are most effective when, from theoretical propositions, repetitions and frequencies (or infrequencies) can be shown. To end the article, we here summarize what we take to be vital results of our investigations:


#### Notes


#### References


———. 2000a. *A Contract with God and Other Tenement Stories*. New York: DC Comics.


Elleström, Lars. 2013. Adaptation within the Field of Media Transformation. In *Adaptation Studies: New Challenges, New Directions*, ed. Jørgen Bruhn, Anne Gjelsvik, and Eirik Frisvold Hanssen, 113–132. London: Bloomsbury.

———. 2014. *Media Transformation: The Transfer of Media Characteristics Among Media*. Basingstoke: Palgrave Macmillan.

———. 2020. The Modalities of Media II: An Expanded Model for Understanding Intermedial Relations. *In Beyond Media Borders: Intermedial Relations among Multimodal Media, Volume 1*, ed. Lars Elleström, 3–91. Basingstoke: Palgrave Macmillan.

Guibert, Emmanuel, Didier Lefèvre, and Frédéric Lemercier. 2003. *Le photographe. Tome I. I*. Paris: Dupuis.


Rajewsky, Irina O. 2005. Intermediality, Intertextuality, and Remediation: A Literary Perspective on Intermediality. *Intermédialités* 6: 43–64.

———. 2010. Border Talks: The Problematic Status of Media Borders in the Current Debate About Intermediality. In *Media Borders, Multimodality and Intermediality*, ed. Lars Elleström, 51–69. Basingstoke: Palgrave Macmillan.

Sacco, Joe. 2003. *Palestine*. Seattle: Fantagraphics Books.

———. 2009. *Footnotes in Gaza*. New York: Metropolitan Books.


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/ by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## Towards an Intermedial Ecocriticism

### *Jørgen Bruhn*

#### **Contents**


#### 5.1 Introduction

Let me begin with a question. What, if anything, unites the following three, very different, media products: a scientifc article called "Trajectories of the earth system in the Anthropocene" by Will Steffen and his colleagues (Steffen et al. 2018), which discusses aspects of what the authors call the Anthropocene epoch; Chinese director Zhao Liang's semidocumentary flm, *Behemoth* (2016), which depicts the effects of coal and mineral extraction in China; and, fnally, the collection of short stories *I am with the Bears* (Atwood et al. 2011)? Apart from being written,

Linnaeus University, Växjö, Sweden

L. Elleström (ed.), *Beyond Media Borders, Volume 2*, https://doi.org/10.1007/978-3-030-49683-8\_5

J. Bruhn (\*)

e-mail: jorgen.bruhn@lnu.se

<sup>©</sup> The Author(s) 2021 117

produced, and published within a few years of each other, the obvious common denominator is that they all respond to and in different ways aim to represent different aspects of the current planetary ecological crisis as defned by, among others, atmospheric chemists, climate scientists, and geographers. They are part of the broad feld of ecomedia that I will defne later.

To put it very briefy, the natural sciences work by meticulously collecting data, which are often based on measurements taken by sophisticated technical apparatuses, in and outside laboratory facilities. The data are, since the massive breakthrough of immensely strong computing powers, analysed by way of the now ubiquitous computer simulations (Roundtree 2014). This enables scientists to produce descriptions of extremely complex conditions not only as they are at the present moment but also in relation to the past and the future and, in the case of ecologically related threats, the data are used to predict future scenarios. Most of such work and the discussions around it take place in the confned spaces of academic journals and scientifc conferences, and this is how most non-specialists get to know about scientifc results and research—via printed or digital or other media forms, often in relation to fairly 'spectacular' breakthroughs or particularly frightening predictions.

In other words, for the natural scientifc results to make any impact outside the relatively narrow circles of scientifc publications and debate, the results need to be *communicated*, and communication per se involves media products with different possibilities and limits. Newspapers, popular science journals, and documentaries are some of the well-known media forms that communicate information about scientifc research, but poetry, science fction novels, feature flms, or art exhibitions can do this too. That the results of scientifc research need to be communicated to the public is, of course, a known fact in several well-developed disciplines, including risk communication, health communication, science communication, and climate change communication studies.1 These felds are based on a tradition of communication studies which are particularly well suited to analyse the history and impact of journalistic media and mass medial forms, but that have not traditionally been interested in aesthetic communication.

In this chapter I will, like these felds would normally try to, take as the crucial, rather commonsensical, starting point that it is indeed media products—three examples of which I mentioned to begin with—that communicate basic ideas about the ecological condition of the planet to most people. I agree with these felds that it is important to investigate these on a sound theoretical and methodological basis, but, in my readings in the feld, I have not yet come across a satisfying theory or method which describes and analyses (and thus compares) media products from the perspective of the broad spectrum of the communication of science that goes all the way from, for instance, the United Nations' Intergovernmental Panel on Climate Change (IPCC) reports to a science fction feature flm, a newspaper article, or a board game representing the ethical problems that are emerging from a warming world. Or the three examples mentioned in my introduction above.

It is my aim, in this chapter, to suggest a theory and method that are able to do exactly this: I call this position 'intermedial ecocriticism', and the most general idea of intermedial ecocriticism is the conviction that the ecological crisis is not a problem or a condition restricted to investigations in the natural sciences, or that possible solutions to the crisis can be reduced to technological solutions. The humanities need to play a role in the question.

#### 5.2 Sketching the Background

The immediate problem of analysing and comparing the three examples mentioned above is that they clearly fall into three very different categories. In the intermedial research tradition I belong to, the three examples are called "media products", and they quite clearly belong to three different "qualifed media types" (Elleström 2010, 2020). It is the fundamental differences in qualifed media types that make it hazardous to make wellfounded critical comparisons between them, unless the three different media are simply, and quite crudely, considered to be different boxes containing the same content. This is probably why the communication felds mentioned above have mostly refrained from making more substantial comparisons across media borders. Therefore, the question is which options are available if the ideas concerning the ecological crisis, as these are represented in a wide array of media forms, are to be analysed in a fruitful way.

In this chapter, I argue that it is possible and even necessary to combine two academic disciplines, intermedial studies and ecocriticism, in order to face the challenge of analysing, discussing, and comparing texts like the three examples mentioned. Instead of using the communication studies approach, which has mostly been developed on social sciences foundations, I have been looking for suitable theoretical foundations in what is often called the 'environmental humanities' or 'green humanities'.

I work from the hypothesis that, in general, the environmental humanities tradition is haunted by a methodological impasse, in that it often tends to do one of two things. The frst is that in terms of media types, and thus academic disciplines, there is a tendency to 'compartmentalize' ecological questions, as if the ecological crisis respects the borderlines defned by media types or academic departmental traditions. The second is that when cultural theorists with broader interests that span different media types and academic disciplines examine ecological representations, they often do so, I will argue, without a sound methodological and theoretical ground: it takes a well-defned common ground to compare, say, a scientifc article with a science fction flm. This is why, in what follows, I argue that the environmental humanities could be supplemented with an analytical tool that is suffciently broad to analyse several media types and suffciently fne-grained to do this in a detailed way.

I do not wish to give the impression that there are currently no environmental humanities approaches that go in this direction, though. One such approach has been suggested by the cognitive cultural scholar Alexa Weik von Mossner. In an attempt to overcome some of the problems inherent in the feld, she recently defned "environmental narratives" as "any type of narrative in any media that foregrounds ecological issues and human-nature relationships, often but not always with the openly stated intention of bringing about social change" (Weik von Mossner 2017: 3). This very broad defnition (depending on how widely the notion of narrative is defned) may be compared with one of the shortest but, at the same time, most comprehensive defnitions of ecocriticism, which has been provided by the infuential literary ecocritic Gregg Garrard. Ecocriticism, for him, is "the study of the relationship between the human and the nonhuman, throughout human history and entailing critical analysis of the term 'human' itself" (Garrard 2012: 5). Garrard's and von Mossner's defnitions are valuable because they open a possible perspective of a broad environmental investigation *across media*, as opposed to the many productive attempts to analyse environmental questions in one, single media type, not only in literature (the originating and still-dominating discipline in ecocriticism) but also in a number of other aesthetic media.2 However, even if these descriptions raise the hope of a truly cross-disciplinary ecocritical approach that can be adopted across, in principle, all imaginable media types, it has not really been fulflled yet. In von Mossner's study, which is built upon a cognitive narratological foundation, the admirably broad corpus is restricted to fctive and non-fctive literary and cinematic narratives. And even if Garrard's defnition is potentially cross-medial, his own corpus is mostly literary material.

What is needed, I argue, is an approach that can analyse and compare media products across, in principle, all media types and, for this purpose, the term 'ecomedia' has been proposed. Since this suggestion lies close to my own interests, I will discuss this position in some detail. According to Simon Estok, the term includes "any media that deals with environmental issues, implicitly or explicitly". It is a reference to 'Ecomedia studies', defned by Stephen Rust as "a historically situated, ideologically motivated, and ethically informed approach to the intersections, of media, society, and the environment" (quoted in Estok 2017: 17). The notion of ecomedia studies is clearly affliated with my idea of intermedial ecocriticism, but there are major differences. The feld is described in the anthology *Ecomedia* (Rust et al. 2016), which defnes itself as a textbook (it includes working questions and fact boxes). It offers a broad survey of qualifed media types, including photography, cinema, advertising, new media, and gaming, as well as other, mostly mass medial forms, which are discussed under three headings. "Frames" deals with visual media, "Flow" deals with broadcast media, and "Convergence" deals with the mixing and transforming of mostly non-aesthetic 'new' media in contemporary culture.

*Ecomedia* offers an admirable up-to-date discussion of crucial aspects of ecocritical discussions regarding a number of qualifed media types across the globe. However, the book refrains from defning media, mediality, or mediation in any sustained form (it does not do so in the general introduction to the book nor in the three opening chapters that defne Frames, Flow, and Convergence). As a result, the book provides ten well-structured and very informative *medium-specifc* descriptions of specifc media forms' ways of relating to environmental questions through engaged case studies, but does not formulate general notions that enable comparisons and discussions across the borders of different media types. So, what the book admirably gains in fexibility, global breadth, and illuminating examples, it lacks in systematicity and thus comparability. In what follows, ecomedia refers to the entire corpus of representations dealing with ecological issues, but it does not refer to any particular theory or analytical method. Regarding intermedial terms, I use the term 'ecomedia' to signify representations of environmental issues in any specifc media product that in any qualifed media type engage in representing in fction or non-fction (or mixing these) either the reasons behind or the results of the environmental threats; Ecomedia representations include the notion of the effects of what has been called the Anthropocene epoch, which necessarily includes a questioning of the distinctions between human and non-human, and between culture and nature.

In respectful but critical opposition to the Ecomedia position, I will suggest a cross-medial ecocritical approach, where media products from a wide range of media types could not only be analysed but also critically compared in meaningful ways. But what would such a method look like?

My fundamental suggestion seems simple: I wish to investigate the broad corpus of ecomedia by applying intermedial terminology. That is, I aim to show that the rich and deep insights of contemporary ecocritical thinking can be combined productively with the analytical strengths of intermedial studies. As mentioned above, this is what I call intermedial ecocriticism.

#### *5.2.1 Three Basic Intermedial Points—And the Question of Representation*

I will refrain from setting out in any detail Elleström's notion of the four modalities of media or the distinctions between basic media types, qualifed media types, and technical media of display, which are all clearly explained in Elleström (2020). Elsewhere, I have constructed an analytical model based on Elleström's ideas (Bruhn 2016; Bruhn and Gjelsvik 2018). What I do want to stress is that Elleström's modelling of media makes it possible to gain a fundamental understanding of three basic intermedial ideas that I consider to be part of the essential infrastructure of intermedial studies. The frst idea is that all media products are medially and modally mixed, which of course is an idea that other researchers have been well aware of but have often described in more intuitive ways (Mitchell 1994; Bruhn 2010). It is tempting to jump to the conclusion that this would mean that there are no such things as media types (or genres, or other comparable categories), but this is not the case: almost all media products are, according to their defnition, part of a qualifed media type, but these types are not only defned by their constellation of media and modalities; they are also contextually and operationally defned. The second idea, therefore, involves talking about a contextualized medium specifcity, meaning that despite the mixedness of media, all media products, according to their defnition, can be characterized with a weak, so to speak, or temporary medium specifcity: any qualifed media type offers temporary "affordances", that is, limits and possibilities (Kress 2010) that enable some things to be expressed and hinder other things being expressed. The third, general intermedial idea is that, apart from having a descriptive analysis and contextualized defnition of the basic media aspects, it is also useful to work with a key analytical distinction between media integration and media transformation.3

This distinction is a result of the a prioric idea of intermedial studies, namely that all media are medially and modally mixed, which naturally leads to the question of how the a prioric mixedness of all media texts may be analytically approached. It is not particularly interesting to simply demonstrate the mixedness of media products; instead, the mixedness needs to be described and analysed. From an analytical, pragmatic point of view, the most important dimensions are media integration and media transformation—the transformation dimension is particularly important in intermedial ecocriticism.

Broadly but briefy speaking, the *media integration perspective* applies to phenomena where two or more medial forms or modes coexist in the same media product at the same time. Examples could be if a painting by Cézanne is represented in a flm and jazz is playing in the background or when images and written words coexist as parts of a scientifc article. On the other hand, the *media transformation perspective* investigates how medial content or form is transformed from one medium to another in a temporal process. For instance, when a scientifc idea (like oxygen) is drawn as connected atoms on a blackboard in a classroom, when a novel by Jane Austen is turned into a flm, or when the idea of climate change, as expressed in scientifc articles, is represented as the main theme in a contemporary environmental documentary or a feature flm.

Media transformation, according to its defnition, contains a temporal perspective. First there is a novel, then it is turned into a flm; frst there is flm, then it is turned into an amusement park; frst there is a painting, then there is a poem representing this painting, and so forth. In this large corpus of cultural objects and processes, introduced and discussed in Linda Hutcheon's *A Theory of Adaptation* (Hutcheon 2006; see also Bruhn et al. 2013), the medial mix lies, so to speak, in the process: certain aspects of a novel (typically, themes, parts of the plot, certain characters, setting, etc.) are transported into a flm, but certain aspects of the adapted work are necessarily left out or changed beyond recognition. The media transformation process is *transferring* certain aspects while also *transforming* everything into a new media product (and a different technical and qualifed medium). Analysing media transformation basically means understanding the interplay between medium specifcity (what characterizes specifc media) and transmedial aspects (that which crosses media borders).

From an analytical point of view, it is helpful to divide all the inherent mixedness of all media products into either 'temporal transformation' or 'synchronous integration', but this distinction is pragmatic rather than essential. Given the idea that all medialities are mixed, it follows, on the one hand, that all media products are in fact an integration of mediality or modality aspects. Given the fundamental idea of intertextuality, meaning that all texts are versions of earlier texts (Allen 2011), on the other hand, we may conclude that all medialities are, basically, the result of a transformation.

Consequently, when performing an intermedial analysis on a specifc media product or on the relations between media products (or when thinking about the general aspects of a media type), one might investigate either mixtures (integration) or traces (transformation). Thus, from a medial perspective, any media product, for instance, a literary text, is comparable to the famous duck–rabbit illusions: depending on your analytical interest and investment, you can choose to perceive a media product either as a combination *or* as a process of transformation; both dimensions are inherent aspects of the specifc media product. To get the fullest possible description and interpretation, one might combine the two approaches, but many specifc analyses will typically focus on one of the two dimensions.

At this point in my demonstration of the general analytical notions, I need to touch upon the question of representation, more particularly the representation of environmental issues, which has been a central concern in contemporary environmental humanities. Rob Nixon is an important commentator on this question, and he stresses the "formidable representational obstacles" of describing global warming (Nixon 2011: 2). He also underlines that the effects of the ecological crisis, which he usefully and famously terms "slow violence", is "a violence that is neither spectacular nor instantaneous, but rather incremental and accretive, its calamitous repercussions playing out across a range of temporal scales" (Nixon 2011: 2). This means, basically, that the dramatic effect of the ecological crisis in general, and global warming specifcally, is a phenomenon that is imbued with a set of complicated representational issues. In recent ecocriticism concerning the representations of global warming, Timothy Morton's idea of the 'hyperobject' (Morton 2013) has been perhaps too infuential in convincing critics of the near-impossible task of understanding and therefore representing climate change as well as other aspects of the ecological crisis.4 I intend to follow Nixon in acknowledging that the environmental crisis presents huge problems of both grasping and representing the crisis, but I do not fnd it useful to consider, for instance, global warming to be an unrepresentable hyperobject.

I prefer a more pragmatic understanding of the abilities of representation. Representation, as I see it, is a fundamental part of human communication: it is a process that uses media products to stand in for all sorts of phenomena of material or mental character, including so-called fctive and non-fctive phenomena. According to their defnition, representations cannot be completely identical to what they represent, and representation is not devoid of performative aspects, so any representations necessarily function on a differentiating scale from higher to lower precision and effect, depending not only on the media products but also on the context in which the media products are produced and perceived. Language, visual communication, and any other semiotic forms of representation are often very effcient but not impeccable tools of communication. In principle, I do not wish to exaggerate the fundamental differences in how easy, or diffcult, it is to represent the coffee cup on the table beside me as I write these lines as compared to the effects of climate change or the victims of the so-called sixth mass extinction (for a broad discussion of the question of representation, see Hall et al. 2013).

Let me try to sum up the frst, general part of my exposition. Intermedial ecocriticism is intended to mark the outline of a broad-spectrum research goal. The most general background of intermedial ecocriticism is the conviction that the ecological crisis is not a topic restricted to investigations in the natural sciences, or that solutions to the crisis can be reduced to any quick tech fx. Instead, as environmental humanities stresses time and again, the ecological crisis relates to key questions that the humanities ask and, consequently, ecological issues are "being reimagined as an ethical, societal, and cultural problem" (Yusoff and Gabrys 2011: 517). Or, to use a slightly longer quote from Ursula Heise's 2017 introduction to a large handbook on environmental humanities:

The environmental humanities, by contrast [to the technico-scientifc approach], envision ecological crises fundamentally as questions of socioeconomic inequality, cultural difference, and divergent histories, values, and ethical frameworks. Scientifc understanding and technological problem-solving, essential though they are, themselves are shaped by such frameworks and stand to gain by situating themselves in this historical and sociocultural landscape. (Heise et al. 2017: 3)

Intermedial ecocriticism, more specifcally, contributes to the explorations in ecocriticism and environmental humanities by stressing that ecological questions, according to their defnition, relate to questions of communication and thus representation, and by stressing in particular that environmental humanities and ecocriticism need a cross-disciplinary and cross-media analytic approach that matches the necessarily broad nature of the environmental crisis.

By using the notion of intermedial ecocriticism, I aim to develop a method to analyse and interpret specifc media products, or, on welldefned methodological grounds, to compare more than one media product across media borders.

In other words, this is the broad outline of some of the outcomes that an intermedial approach to ecocritical questions may achieve, but it is important to stress that combining the feld of intermedial studies with ecocriticism (or other felds inside environmental humanities) may lead to several rather divergent approaches. How ecocriticism and intermedial studies are combined depends, for instance, on the specifc scientifc or even ideological interests of the researcher: it depends on which audiences are meant to receive the results and whether the research idea is meant to be used in didactic contexts. In what follows, I sketch out one among many possible operational approaches of intermedial ecocriticism.

#### *5.2.2 Intermedial Ecocriticism: A Methodological Suggestion*

The obvious initial move, before any analysis is even begun, is to choose one or more media products with subject matters that relate to ecological issues. In short, the media products should be part of the corpus of ecomedia. This must be followed by choosing a specifc question or problem to investigate in the chosen media product(s). Topics to investigate can be anything from the representation of the ecological 'crisis' to narrative aspects of the ecomedia product, to the ways in which the media product facilitates human agency in the Anthropocene, to temporal patterns in Anthropocene representations. Or, and here I am referring to existing ecocritical work, one can discuss whether and how the existential and psychological notion of 'pretrauma' exists in cinema and literature (following Anne E. Kaplan 2015), or different affective patterns in narrative as well as non-narrative media products (following Weik von Mossner 2017). It will come as no surprise that the initial choice of topic has important consequences, because it will direct the analysis of the media product (and if one works with more than one media product, the chosen topic will be the one that is being compared). To explain the approach, in what follows, I will use an imagined ecomedia product: a short, printed newspaper article which describes the melting glaciers in the Arctic, and the topic I want to explore will be examined by asking this question: how, in intermedial terms, are the melting glaciers represented in this article?

After the initial choice of media product(s), as well as one or more topics to be analysed and perhaps compared, three consecutive steps follow: in the frst, descriptive step, the main aim is to defne and analytically describe the fundamental features of the media type in general and the media product in particular. In the second step, after having established the basic features, the aim is to specify, by way of an analysis and interpretation, the exact ways in which the affordances of the media product enable or disable specifc aspects that are to be represented. In the third and fnal step, the results of the analysis can be compared to the analysis of one or more media products.

In the analysis, I apply the descriptive media tools suggested by Elleström (2020), but let me again emphasize that the general features of the approach I have sketched out can be followed by using other analytical tools that facilitate analysis of texts and comparative investigation. There is a more detailed description below of the three-step working processes.

#### *Step 1*

Step 1 consists of, frst, carrying out a descriptive analysis of the (often obvious) technical media of display that characterizes the chosen media product; here the researcher can choose the media of display through which he or she has received the media product, or the researcher can describe the typical or most widespread display function for the specifc media product. With regard to my example, in the short article on the melting Arctic ice caps, the technical media of display is a paper version of a newspaper (or the internet version of this article, which would make the technical media of display the computer screen or the mobile phone screen). I will refer to the paper version here.

After this, what Elleström defnes as the material, spatiotemporal, sensorial, and semiotic categories of the basic media dimension of the media product are described. The basic media aspects of the newspaper article that I take as an example would be that the material aspect would include paper and ink: the spatiotemporal aspect would include the fact that the conventional way to read the article would be to read the text in the sequence produced by the journalist, even though it is possible, with a written text, to read it in your own preferred way. In sensorial terms, a printed article is predominantly perceived through the eyes, and the fundamental semiotic aspect of the newspaper article would include the fact that the verbal written text is mainly interpreted by way of its symbolic sign qualities; if there are photographs in the article, these would relate more to an iconic sign function (both aspects are interpreted according to Peircean semiotics).

The aim of the frst step is to establish a precise and detailed *description* which will enable the next step's more interpretative actions. The description is meant to result in a clear understanding of the different medial aspects of a text, which therefore makes the following interpretive moves easier—and, if necessary, it makes the comparison between media products possible.

#### *Step 2*

Having established the qualifed media type, the aim of step 2 is to analyse and interpret the media product using intermedial analytical tools. The media integration and/or the media transformation perspective can be applied, as well as other intermedial analytical and descriptive tools. The aim of the analysis is to specify the affordance of the qualifed media type as this plays out in the specifc ecomedia product regarding its ability to represent ecological topics. Generally, the media transformation aspect will loom large in many intermedial ecocritical analyses because all ecomedia products are fundamentally, from an intermedial point of view, media transformations of scientifc material into ecomedia.

So, regarding the short, printed newspaper article mentioned above, from a media transformation angle it is an example of how a scientifc measurement of ice caps, which has frst been published in a natural scientifc article, has been transferred and transformed by a journalist into the format of a news item. The interplay between the photograph and the written text of the article can be analysed from the media integration perspective.

The aim of this analysis is to understand which aspects of, in this case, global warming, as this affects the Arctic ice caps, a short newspaper article generally, and the chosen article specifcally, can represent. To a certain extent, the analysis may confrm rather commonsensical and general conclusions, but the advantage of the analysis is that it can be much more precise in pinpointing the exact affordances of the general media type and the specifc media product. In my more extended examples below, I will demonstrate the usefulness of this step.

#### *Step 3*

Having established the specifc affordances of the given media product, step 3 allows the possibility of comparing it with other media products with a directly comparable subject matter. If this step is taken, then the second, or third, or fourth media product must of course be subjected to the two steps described above before conducting the third, comparative step.

#### 5.3 Comparative Case Study

#### *5.3.1 A CarbonBrief Article*

Having explained the mechanics of my method by way of an imagined example, I now wish to do a comparative case study of two actual existing ecomedia products.

The specifc topics I wish to pursue are as follows:


The frst ecomedia product I wish to analyse was published in August 2019 on the website of CarbonBrief, which is an internet-based, factoriented media outlet situated in London. CarbonBrief publishes articles, comments, and other material related to global warming issues on the internet but not on paper: there is often a focus on aspects related to the UK, but the work has a clear global outlook. Material published on the site is based on new events relating to global warming, either events in the world or scientifc reports or new fndings. The article I will analyse was written by Daisy Dunne and is called "Climate change made Europe's 2019 record heatwave up to '100 times more likely'" and was published on the website on 2 August 2019 (Dunne 2019).

The general aim of this popular scientifc article, which consists of about 1500 words, one photograph, and two modifed maps of Europe, is to represent to its readers the scientifc hypothesis that the global temperature on the planet is rising due to anthropogenic infuence. More specifcally, the article argues that different dramatic results of this new planetary condition can be analysed in a new and more exact way by means of socalled attribution science. Attribution science has advantages compared to more traditional climate science, which has been very cautious regarding linking actual weather phenomena with climate change. The aim of attribution science is to link much more directly the concrete and often dramatic *weather* phenomena with more general developments in the *climate*, typically by saying that a certain dramatic weather phenomenon has been made x times more plausible due to the larger effects of climate change.

The question is, how does the article do this—and can it be meaningfully compared to other media products with approximately the same aim?

#### *Step 1*

Given the fact that CarbonBrief is a web-only publication, any reading of the text will be done via a screen, either on a computer (that is how I read it) or on a mobile phone. In my reading of the text, the *technical medium of display* is therefore a laptop computer screen.

The *material* aspect of the article, as displayed on a computer screen, is the two-dimensional space upon which one can read text and see images by way of the technical apparatus making things visible on a screen, for instance, by scrolling on the page. An internet connection is also necessary to be able to reach the website. Several colour photos and/or colour data visualizations are a natural part of all the articles on the CarbonBrief website, and these also form part of the material aspect. The cost of reproducing images is much lower for online publications than for printed articles, so there tend to be more in the former.

In this case, there is a large colour photograph documenting a man shielding himself with a newspaper during the heatwave in London in July 2019 and two visualizations of historical heat patterns in Europe.

The *spatiotemporal* organization of the media product is a bit complicated. On a superfcial level, the article has many similarities with a printed article. However, there are several highlighted words (twenty in all) in the article, which function almost like footnotes in a printed text, meaning that when the reader follows such a hypertext link, he or she is redirected to another text. However, the function of a hypertext reference, as it is used here, and a footnote is not identical. Whereas the function of a footnote is, conventionally, to offer either a short additional remark and/or a specifc reference, the hypertext link in a digital text instead redirects the reader to another article, which the reader gains access to by clicking. The reader can hit the back button on the browser to return to the "Climate change made Europe's 2019 record heatwave up to '100 times more likely'" article, but he or she may get lost or simply become more interested in the new website, so it is possible that there is no return to the original article. Or, to speak in spatiotemporal terms, the hypertext affords a temporal and spatial sequence which differs from the rather strict and relatively simple spatiotemporal sequence of a printed article. The rather strict sequence of reading the conventional written article is broken up into a more fuid and potentially less restricted (in space and time) form. Compared to the limited length of a written article (ending when the article ends), the web article is potentially non-fnite, at least if one follows the links to new links and more new links. It is worth mentioning that most of the hypertext references are 'in-house', so to speak, to CarbonBrief articles that go into more depth about some of the topics in the article.

The *sensorial* dimension of the digital article on the net does not differ signifcantly from that of a written text: the visual interface is perceived via the eyes. However, when clicking around in the article, bodily movement of the fngers on the keys or the mouse pad or on an external mouse is needed, which initiates a sensorial element of touch, too.

The *semiotic* aspect of the article combines in intricate patterns the three sign relations suggested by Peirce (as do most media products). The written text must be read as symbolic signs designating verbal language, whereas the photograph and the two visualizations must be read as two kinds of mainly, but not solely, iconic representations. The initial photograph iconically signifes extreme heat, represented by a male fgure, whereas the two following visualizations consist of maps of Europe upon which coloured areas symbolically signify (by symbolic colour codes) the heat at a given moment in the summer of 2019 and, in the next map, the historic development of heatwaves from 1950 to 2018 as compared to the summer of 2019. Major aspects of the media product rely on the indexical sign function in the sense that many of the most important facts of the text as well as the visualizations are representations of indexical measurements, that is, natural scientifc investigations with a variety of measuring instruments that are then collected and represented in the text.

The frst, descriptive, part of the analysis can now be summed up. The general media type is the popular scientifc journal article, which, more specifcally, is technically displayed by way of computer screens or other electronic devices. This technical aspect leads directly into the material aspect relating to the very basic possibilities that an electronic text offers (in comparison with a printed text). Important here is the fact that, for economic reasons, a higher number of photographs and other illustrations can be included. Also, the hypertext possibilities, understood as part of the spatiotemporal set-up, are crucial when it comes to referring directly, via links, to other texts. In sensorial terms, human sight dominates, but touch is implied, too.

With this initial description of the media product, I hope to have exemplifed that, even though such a description is rather banal, it has the potential to clarify one thing: that understanding the basic medial set-up of a media product conveys signifcant aspects of it, even before going into the elements of the text that the cultural sciences would normally feel more comfortable thinking about, namely the content and form aspects: what it is about and how is it formed.

#### *Step 2*

Based on the descriptions created during step 1, Step 2 is meant to involve a discussion of the general affordances of the media type, and in particular the specifc media product, to prepare for a specifc discussion of the general ecological question and the specifc topic.

The popular scientifc article, as a media type, is a classic example of science communication, and it partakes in what in science communication studies is understood as the "public understanding of science" (Burns et al. 2003). More specifcally, the article is part of the feld of environmental communication, which has exploded in recent years according to a literature review (Comfort and Park 2018).

The purpose of the article is to represent complex scientifc research about fndings in climate science in general, and about so-called attribution science in particular, to the general public. The article tries to cater for non-specialist readers and add to the public understanding of science, as mentioned above. This can be reformulated in intermedial terms; the writer of the CarbonBrief article, Daisy Dunne, conducts a *media*  *transformation* by moving content or ideas from the medium of scientifc article(s), which is the originating background or source, into the new media type, the popular science article. A basic rule of popular science communication of complex research results is to simplify them to make them understandable. Therefore, there is an immediate element in the media transformation that consists in picking out and 'translating' technical terms, highly formalized terminology, and numbers that are diffcult to grasp into something else.

Simultaneously with simplifying and picking out material, the writer faces a more constructive task, namely to fnd appropriate ways to represent the ideas which have been researched in the natural sciences. In intermedial terms, this means that the media transformation aspect connects to the media integration aspect. The question of representing the facts of the scientifc fndings can be partly solved by choosing and combining different media aspects and modalities.

The most important combination aspect in the article is between—in Peircean terms—the signs that are mainly meant to be interpreted as iconic signs (the photograph and the two coloured maps) and the written text consisting of mainly symbolic signs. The written text dominates in the meaning making of the message (in the sense that the text could more or less stand alone and still be meaningful without the images—the opposite is not the case), but the iconic features are very important. The title of the article, written on top of the photograph, designates the entire article's content: climate change made Europe's 2019 record heatwave up to "100 times more likely". But it is also a complex verbal utterance: taking as its starting point the current heatwave, it connects the dramatic weather event to a larger historical trend and stresses the most important conclusion of the underlying research: that heatwaves are becoming more frequent and warmer as a result of global warming. In addition, the title contains a quote which has no author, although it will become clear where it comes from if one reads the article. The rest of the article substantiates this headline and supports the claim with the aid of three main forms of support: (a) quotes and reported quotes from researchers; (b) the material referred to in the hyperlinks, of which several are current measurements of the temperatures around Europe; (c) visual material. The quotes and the material in the hyperlinks build up an argument using theoretical and historical claims combined with scientifc measurements that together aim at producing the overall message, which has a narrative form: it starts from the individual experience (heatwave in London) and leads to a more general fact (heat all over Europe), which are both part of a more elaborate trend (heat in Europe over time). This is an argument that clearly goes from the specifc to the general.

The iconic material, on the other hand, is effectively organized to support that narrative form. It starts from the individual, human-interest perspective in the frst photograph and leads to the second map of Europe where the individual aspect of the frst image is, so to speak, made a condition for many Europeans at the same time. Finally, with the second, historically oriented map, the current heatwave is seen as corresponding to a much more comprehensive pattern. The pictures tell a rudimentary story that supports, or, rather, doubles, the written text: the pictures 'say' that our individual experience must be taken seriously (photograph of man in heatwave), but that the heatwave was probably more than a mere coincidence—it was a European phenomenon (frst map). Not only that, but the extreme weather can be explained by referring to historical climate models (second map), which give the initial individual experience a totally new meaning.

The use of hyperlinks in the text is another combination aspect that I have already mentioned but which should now be understood in more depth. It should come as no surprise that the technical and material aspect of a text has effects on its meaning production, but in this case in particular the spatiotemporal nature of the digital text has certain effects, which are best understood by comparing it to a conventional printed news article. Generally, and on the positive side, the journalist producing a digitally disseminated text can use coloured illustrations and photographs to accompany the text without incurring large additional costs and without technical problems or limits because of space, as compared to a printed article in a journal. From the point of view of the reader, this use of visual material may make the reading easier and more pleasurable. Apart from the overall graphic layout of the article, made possible by the digital platform, a digital text like this facilitates an easily navigated reference system to other digital texts, including references to other media (the BBC, for instance) as well as meteorological sites and scientifc texts. Also, in order to avoid explaining all the scientifc background facts, the CarbonBrief article includes several 'internal' references, that is, references to earlier articles by CarbonBrief. Finally, the article ends by referring to related articles where the points being made can be deepened or discussed.

Making references to other news outlets or to already published material is not unique to the affordances that the specifc technical media aspect of digital journalism allows: the point is that reaching these references only takes one click.

The fnal feature that distinguishes this article from a printed text is the fact that readers can comment on the text immediately; CarbonBrief employs a feedback system called Disqus, which prevents anonymous comments and only allows so-called comments from registered users. In the case of this particular article, at the time of writing (7 September 2019, a month after the publication of the text), only two comments accompanied the text: both were critical of the content of the article. Again, several of these features exist in a printed article in a newspaper or a journal or magazine. The difference is that the illustrations in the online article are cheaper to reproduce, the references to very comprehensive material outside the text can be much more elaborate and easier to get to, and the comments from readers are directly included alongside the article in its present state instead of being, in a printed newspaper, part of a later edition one or several days later, where the readers' comments stand alone without the originating debated text.

There are also limitations to this media type compared to a text in a printed newspaper. First of all, this article is relatively brief in terms of the number of words it contains, which may be part of the strategy of CarbonBrief to make their climate news both understandable and digestible, and although the length is limited, the breadth of material that the hyperlinks connect to makes the article almost infnite in size. Second, a downside to using hyperlinks is that the writer risks 'losing' the reader as he or she is making their way through the article if the reader follows one of the links and does not return to the originating article.

After this long descriptive analysis of the intermedial aspects of the article, it is now possible to offer a more interpretative answer to the questions I posed initially:


In response to (a), when it comes to the representation of scientifc research, I have already pointed to the obvious fact that the CarbonBrief article simplifes the research to a level at which it can be understood by non-specialists. This means that complicated terminologies, comprehensive methodological discussions, and large datasets (or thorough descriptions of these) are not included in the article. The brevity of the article (as compared to most standard scientifc articles) is a sign of this. The use of hypertext links retains the brevity of the article while offering more indepth material to readers who are interested, for personal or professional reasons, in having both the ideas and the references as further information. The article manages, in the short space it occupies, to refer to the fundamental method of 'attribution science' as compared to climate science's normal approach, and this gives the reader a feeling that the sciences are not static but are evolving, indeed progressing. The graphic style of the article (which is a typical intermedial phenomenon in itself, since it considers the visual set-up of the page, the layout, etc.) gives the article an appealing look that makes the reader feel invited to take part in the content.

A large part of the appeal of the article has to do with the use of images: the frst photograph makes the article directly link to related experiences that many Europeans (and others) have had both in 2019 and in 2018. And, as mentioned already, the three image sections of the article not only illustrate the article but also 'tell' a little story themselves while, at the same time, they give the article 'space' and 'air', which is part of its inviting style.

In response to (b), the images are part of the article's attempt to not only represent a 'readable' version of complicated scientifc research but also to make the research relevant for non-specialists. Here, the initial photograph, in particular, is a hook; for many readers, experiences of extreme heat are familiar, and instead of focusing on the anecdotal aspects or the personal feelings of isolation that the heat causes (which is the angle of many mainstream reports on heatwaves), this article uses a personal experience frst, but then makes it a much more widespread feeling, frst on a European level (i.e., in space) and later on the level of a long historical period (i.e., in time).

#### *5.3.2 Charlotte Weitze's Novel* **Den afskyelige** *(***The Abominable***)*

Because of spatial restraints, I will offer a shorter version of the descriptive analysis of the second media product than of the frst one, but this will still make a comparison possible.

First, I need to present the media product. Charlotte Weitze's 320 page novel *Den afskyelige* (*The Abominable*) (Weitze 2019) is narrated by Heidi, a young nurse living in northern Scandinavia, probably Norway, in a relatively near future dominated by global warming. She tells the dramatic story of how she falls in love with a man, Kenneth, who is apparently suffering from hyperhidrosis, which is abnormal sweat production, and she describes in detail how the couple struggles, with mixed results, to live a CO2-neutral life in a globally warmed and warming world. Later in the novel, the reader understands, and so does Heidi, that her lover Kenneth has no conventional diagnosis: instead his many symptoms stem from the fact that he is actually a descendant of a yeti. She leaves her lover, despite being pregnant, but later loses the foetus. After a dam collapses towards the end of the book, she hides on a mountain while trying to save her best friend, Mette, and her baby. However, Mette drowns in the rising waters and the book ends with Heidi taking care of the little baby, not knowing whether the mountain top on which they live is the only place on earth where it is still possible to survive.

It is clearly a novel that deals with global warming, but in an entirely different register from that of the article previously discussed. It will hopefully be possible to defne the differences more precisely by way of an intermedial analysis.

#### *Step 1*

Step 1 of this shortened analysis will just briefy establish the basic medial facts about the novel. Regarding the *technical media of display* dimension, I read the work in a hardback printed book version; the book has a dust cover with an illustration that is identical to the one on the hard cover (designed by Alette Bertelsen), which combines verbal language with the dark profle of a looming animal fgure while also alluding to a mountain with a tiny house on top, thus referring to the end of the novel.

The basic *material* dimension of the novel is printed letters on paper pages, and the *spatiotemporal* set-up is that of most of narrative written literature (see Bruhn (2016): a forward sequence is suggested even though one can indeed read a novel from the end and towards the beginning or read it at random places. Besides turning the pages and holding the book, which requires the sense of touch, the most important *sensorial* aspect when reading a novel is of course using our eyesight to read. Apart from the dust cover, the dominating *semiotic* nature of the novel is the symbolic, linguistic sign function.

There are, to use Elleström's terminology (2020), qualifying and contextual dimensions which place this book solidly in the tradition of written, narrative literature: the generic form of the novel (*roman* in Danish) is stated on the cover and on the title page. However, this is very unusual for a novel, at the end of the book, the reader fnds a page of acknowledgements, thanking funding agencies and colleagues. In addition, Weitze extends her thanks to a number of experts in many different felds including disaster research, psychiatry, and studies of fanaticism and religious beliefs. These experts contributed to her 'research for an open microphone' for a radio show on Danish broadcast Radio 24syv.

To sum up the frst step: it is not diffcult or surprising to categorize *Den afskyelige* as a written narrative novel; the question is how to analyse it from an intermedial perspective.

#### *Step 2*

How then, in the second step, can the text be analysed from an intermedial point of view, including the fundamental axes of media transformation and media integration? Let me discuss the media transformation frst.

Because it is an ecomedia product, the entire novel can be said to be a media transformation of already existing ideas concerning the ecological crisis and global warming, as this has been put forward in a huge network of scientifc, political, and popular scientifc material.

I mentioned that the novel has an (for literary fction) unusual list of acknowledgements that includes not only friends, colleagues, and funding agencies but also a number of experts in several felds. Weitze thus explicitly acknowledges that several aspects of the novel must be seen as media transformations from the felds of climate science, engineering (concerning dams), and fanatical (religious) psychological states, among others. From this point of view, it would be possible to analyse a number of specifc scenes in the book, as well as the overall theme of the book as the intermedial interplay between medium specifcity and transmedial subjects (which was a way of defning the study of media transformation, as mentioned above). In other words, it would be possible to identify which aspects of the forms and contents of the originating texts that may be transformed (the transmedial aspects) and which aspects that are only transported with greater diffculty (because of medium-specifc constraints). To exemplify briefy: the detailed research results of a scientifc article cannot be referred to at length in the novel, but the two protagonists can discuss why it is that a vegetarian diet is much more climate-neutral than consuming meat (Weitze 2019: 162–163). Therefore, almost everything in this novel, from the overall thematic content to many detailed discussions and particular scenes of the novel, might be analysed as instances of media transformations from a number of scientifc disciplines into the supposedly fctive set-up of the novel.

Regarding the media integration analysis, there are different ways to pursue this. For instance, it would be interesting to analyse the ways in which the novel represents media products from the mass media, how mass media products stand up against other medial representations, and how this particular relation infuences the protagonists. One could also investigate the way in which old written notes and photographs from Heidi's past are interpreted as signs of a changing climate and the death of Heidi's sister in an accident when Heidi was a child. These notes and photographs stand in stark contrast to the media products that Heidi and Kenneth produce together: they create pornographic flms in order to make money for their CO2-neutral life. This peculiar media practice may be discussed while also participating in what Stacy Alaimo has discussed as independent, alternative and partly joyful activist outlets' (pornproducers!) way of engaging with the environmental debate (Alaimo 2016).

After this brief intermedial analysis of the novel, it is now possible to discuss the novel in relation to the two questions:


In response to (a), when it comes to representing scientifc research in *Den afskyelige*, the aim of the novel is to represent and simplify the research at the same time to a level at which it can be understood by non-specialists.

The main strategy of incorporating or representing scientifc material in a work of fction, a novel, is to embody it into the standard aspects of the narrative fction. In this case, this takes place in at least two very typical ways. First, the entire storyworld of the imagined future world is informed by contemporary scientifc theories and simulations concerning what the world will look like in a future affected by global warming. Second, many of the discussions of the protagonists relatively directly translate scientifc material into actions, refections, and direct conversations or discussions. In this way, one of the typical medial aspects of scientifc discourse—the attempt to produce a non-subjective discursive style that tries to express the non-subjective character of the scientifc research—is directly opposed in this novel. Or, to formulate this differently, *everything* in a novel must be seen or described from the perspective of human beings, in this case Heidi, who is the narrator of the novel, and the characters of the fction that she narrates. If the ideal of natural science is to produce knowledge that can be repeated by other researchers, who are usually anonymous, in controlled experiments or measurements, the golden rule of fction is quite the opposite: to always embody everything that happens through the minds and bodies of narrators and/or characters; in fction there is no attempt at objectivity in the scientifc sense of the word.

From this comes an answer to (b), my second research question: how do the media products employ strategies that make hard science directly related to the everyday experience of non-experts in climate science? What fction does in its partly imagined, partly scientifcally grounded storyworlds is to produce human fgures who experience on their bodies and in their minds the more concrete but imagined conditions that are argued for by natural scientifc articles. If the aim of natural scientifc media products is to predict *possible* future worlds, in this case including the effects of global warming, the job of fctional storyworlds is to concretize and embody scientifc research into very recognizable specifc situations. In this way, the results produced in natural science becomes directly recognizable and understandable for readers who either do not wish to or are not able to read scientifc articles. This means that the reader of *Den afskyelige* is helped to experience, via the fctional storyworlds, what life might feel like in the future; Charlotte Weitze offers an interpretation of what she believes, according to research, to be future scenarios for a life in the Anthropocene.

#### *5.3.3 Comparison of Article and Novel*

#### *Research Question 1 (Representation of Science)*

The comparison between the two texts follows the two research questions. The frst research question has to do with the ways in which the two media products represent the scientifc research results that in both cases constitute an important source behind the texts.

Here, the most important difference is probably that the CarbonBrief text directly paraphrases, refers to, and quotes science, whereas the novel has a much more indirect way of referring to the scientifc research. In Weitze's novel, science is constantly being put in the mouths of the protagonists, in direct conversations and discussions and in more general references to 'science says' or 'new research demonstrates'. To explain it in a different way, there is a basic narratological distinction between the texts: the CarbonBrief author functions as an explicit narrator who states the scientifc facts, also by referring directly to quotes from scientists, but Weitze's novel is organized in such a way that the scientifc results and theories per defnition are put in the mouths of persons inside the storyworld.

Another clear difference is that the CarbonBrief article, starting with the discussion about actual weather phenomena, is particularly interested in making predictions about the future. Weitze's novel, on the other hand, is set in such a future, but a fctive future; however, the people inside the storyworld do not, of course, experience the world as a future, but as their present living conditions and surroundings.

This means, and this more or less sums up the points about direct/ indirect and future/present, that the popular science article, like science, aims at producing *general notions* about the ecological crisis and global warming. The novel, on the other hand, *individualizes* the scientifc, *generalized* notions of the future risks inherent in the Anthropocene condition.

I have chosen two media products in two qualifed media types that are both successful in many ways in what they are doing, at least if the criteria are their abilities to represent scientifc results and theories in their respective media forms. However, when comparing the communicative effciency (regarding the article) or the aesthetic form and content (regarding the novel) criteria, it is clear that these could be more critically evaluated regarding their representation of science. The article could, for instance, be criticized for taking an unproblematized Eurocentric view on the Anthropocene future; this, presumably, has to do with the fact that Britishbased CarbonBrief, while having a good European and global outlook, nevertheless functions inside a distinct UK context, which explains the choice of frst a photo from London and then the two European maps (as opposed to more global possibilities). Regarding a critical approach to the novel, it would be absurd to criticize the novel for only representing a limited, Eurocentric aspect of the global effects of the Anthropocene condition.

From a medium-specifc perspective, a novel tends to be anchored in time and space almost by the logics of medium specifcity: generally, the raison d'être of fction is to not speak in generalities but, on the contrary, to demonstrate how general social or scientifc rules work on a human scale, that is, in the concrete lives of individuals or individuals as part of social groups. Therefore, if one has to criticize the novel, it should be on aesthetic grounds. A well-known aesthetic ideal is that when science (or philosophy or other external forms) 'enters' art it should do so in ways that mean that the science is introduced in the artistic universe in necessary or elegant or inconspicuous ways (for an example of this position as well as an historical discussion of the idea, see Gadamer 2013 [1960]). This is not always the case in *Den afskyelige*, where some conversations (or individual utterances in dialogues) sometimes read as almost direct quotes from newspapers or scientifc journals without being properly incorporated in the novelistic form as natural, spoken language uttered in a conversation between two human beings. In these instances, the science is shoehorned into the fction, so to speak: the author's comprehensive research enters the novel as research and not as aesthetically motivated form and content.

#### *Research Question 2 (Scientifc Results and Everyday Experience)*

It is of course impossible to completely keep apart the frst research question (representation of science) from the second question, which has to do with how the two media products employ strategies that make science directly related to the everyday experience of non-experts in climate science.

Communication, according to how it is normally defned, relates to the receivers of the communicative activity. Therefore, my admittedly slightly crude starting point was that one of the major targets, for both popular science communication and a mainstream novel focused on climate change, is transferring aspects of climate science results from hardcore science environments into new areas. Doing this also has to do with relating the scientifc ideas to everyday experiences to a certain extent, and both the article and the novel clearly attempt to do this.

Global warming, and most of the other effects of the critical ecological situation, are, as implied by their names, global in scope, and this is undoubtedly one of the reasons why relating these issues to specifc, personal experience is not always an easy task. Nevertheless, I have chosen these two media products as examples of successful attempts to do so, and that is why my research does not ask *if* but *how* they do it.

The article's approach to this, as I have emphasized, is to base the global condition on a local phenomenon: the specifc heatwaves described in the text and represented by photography are used as signs of a global warming tendency. So, the heatwaves, which many people in Europe have relatively fresh memories of, were not only problematic for one person (the person in the photo) but also for many others who can easily identify with him, and supposedly for the entire European population who are, as shown via the maps of Europe, experiencing this warming. The man in the photograph and the maps are characterized by anonymizing—the idea is that anybody will be able to identify with the man and the nations, and the goal of the article is to make the general point, most of all on a cognitive level; for all we know, the man might be an actor and the maps are extremely general.

Again, the novel's strategy is distinctly different: it offers ways of very directly understanding what life might be like in a warmer world, not in general terms, but by way of identifying fully with specifc persons with names, a fctional background, bodies, families, and jobs. Presumably, the intention of the author is that such a concretized, specifc description is a powerful way to make readers understand how it feels (in both the psychological and the sensorial sense of the word) to live in such an Anthropocene future. Needless to say, the pattern from the frst research question returns here. It is a question of the generality of science versus the specifcity of arts and literature.

#### 5.4 Concluding Remarks

My impetus for writing this article is far from original: it stems from the well-known observation that the natural sciences produce knowledge and data which are absolutely crucial for the world to know about and act upon—but that the transfer of scientifc knowledge to the general public is neither unproblematic nor simple. From this basic fact followed an identifcation of a gap in the broad feld of environmental humanities concerning this question, and also in the social scientifc felds investigating, for instance, risk, health, and science communication. As far as I have been able to establish, these felds have not developed the abstract theoretical ideas or the specifc methods to deal with the plethora of different specifc media products or the more generic qualifed media types that deal with these questions. I have not come across a scientifc approach which is able to effectively handle the problems involved in describing, analysing, interpreting, and fnally comparing radically different media products from completely different media backgrounds, even if they are representing clearly comparable scientifc content.

Having frst identifed this problem, I then tried to suggest a theory and method called intermedial ecocriticism which, as the name suggests, combines two different research felds and traditions, namely intermedial studies and ecocriticism. Because it is part of environmental humanities, one of the main strengths of ecocriticsm is its deep understanding of the historical aspects and the critical level of the ecological crisis. As a research feld, intermedial studies, on the other hand, has a thorough understanding of the problems and complexity of representation in general, and, in particular, intermedial studies has developed tools to analyse comparable content and forms in different media.

I tried to sketch out briefy some of the most important aspects of the two scientifc felds before I then ventured to propose an analytical method that borrows, so to speak, the focus on ecological issues from ecocriticism and the focus on representational issues across media from intermedial studies. I suggested analysing and comparing ecomedia products in two consecutive analytical steps, which led to a comparison of, in this case, an online popular scientifc article and a novel. I tried to conduct the comparison by establishing two specifc questions, one regarding how the media products represent scientifc research regarding climate change, and the other regarding how the ecomedia products relate the scientifc results to the everyday non-specialist experience.

When doing comparative analysis across media, it is tempting to end up on an evaluative or even competitive note: which media product is 'best' at doing x or y? Given the fact that what I refer to as 'weak' or 'temporary' medium specifcity does exist, such questions are not uninteresting, and it is easy to see that some media are, in general terms, more suited to represent certain things than other media. It would, for instance, be rather diffcult (but not totally impossible) for the otherwise highly developed and utterly sophisticated medium of symphonic classical orchestral music to express, clearly and unambiguously, the three major changes made in the state budget of the Swedish state from 2018 to 2019—whereas that would be relatively easy to do in a short, written journal article in a daily newspaper.

In the case study examples that I had selected for my investigation, I was not really able to identify such obvious differences in the 'success' of representing the two issues I had focused on. That might be because I had chosen two media products that were in fact relatively similar (they are both examples of narratively ordered material in which the predominant basic modality is written language), but that was not really what I had expected to accomplish with my analytical method. Instead, what the method was able to detect were some relative but not very dramatic advantages accomplished by, respectively, the science journalist Daisy Dunne and the fction writer Charlotte Weitze; these advantages were to a large degree dictated by the 'weak' medium-specifc affordances of the qualifed media they worked in.

It became clear that both Dunne and Weitze had obviously worked intensively to incorporate natural scientifc research into their texts. The main difference was the way in which the article tries to refer directly and unambiguously to ongoing scientifc research and sources, whereas the novel needs to transfer the scientifc research from direct references into either (a) a general construction of a future, fctional world dominated by global warming or (b) speech in the mouths of the protagonists when they discuss how to live in such a world. From a superfcial point of view, the popular scientifc article perhaps could be thought to be much better suited to doing this, but I have tried to demonstrate that Weitze has managed to include quite a lot of scientifc material in her novel (as was clear from her mentioning numerous scientifc expert sources at the end, too).

When it comes to relating the science to everyday experience (my second question), the novel, following conventional wisdom, probably 'ought' to do best at this. By working with fctional persons in recognizable but fctional surroundings, it should be easy for a novelist to make science matter to non-scientists. Weitze did not fail at this task, but even the journalist Dunne shrewdly used the medium-specifc available means of the online popular science article to construct a small narrative consisting of a man in the not-forgotten local heatwave of 2018 in London that circled outwards to a European climate condition that was easy for many readers to identify with. Banalizing my subject a little bit, I conclude that it was a draw between Weitze's somewhat surreal cli-f novel and Dunne's narratively ordered popular science article.

However, even if this competition ends in a draw, I do hope that perhaps the idea of an intermedial ecocriticism might be a winning concept. I maintain that we need tools to analyse and discuss the scientifc research regarding our epoch's most alarming topics, and even if we in the humanities cannot conduct the natural scientifc research ourselves, we can certainly contribute to the thinking about the ecological crisis in crucial ways—in particular by investigating all the different media products, in all the different media types, that do the diffcult work of transporting scientifc material across the media borders to where most people learn what the sciences are doing.

#### Notes


#### References


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/ by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## Metalepsis in Different Media

#### *Liviu Lutas*

#### **Contents**


#### 6.1 Introduction

In this chapter, I attempt to test the applicability of Lars Elleström's model of intermedial relations (2020) to analyses of the device of narrative metalepsis in a number of different media products stemming from two different media types: literature and flm. At the same time, the chapter has the more ambitious aim to illustrate certain aspects of the model itself using the concept of metalepsis. The overarching aim is to highlight the transmediality of the device of metalepsis, that is, its potential to "be employed in various media" (Rajewsky 2002: 13). As such, the chapter is

Linnaeus University, Växjö, Sweden

L. Elleström (ed.), *Beyond Media Borders, Volume 2*, https://doi.org/10.1007/978-3-030-49683-8\_6

L. Lutas (\*)

e-mail: liviu.lutas@lnu.se

<sup>©</sup> The Author(s) 2021 149

an attempt to break with what Lars Elleström sees as one problematic tendency in today's intermedial studies, that is, that "*[m]edia in general are studied through concepts developed for language analysis*" (Elleström 2020: 6). Metalepsis is a good example in such an endeavour, since its defnition was coined within the frame of a linguistic medium, namely literature. This is, I argue, no reason to persist in studying metalepsis from a language-based perspective, which is the tendency even in the numerous cases of approaching metalepsis in other media types than linguistic ones. It is, I argue, not suffcient to acknowledge the transmedial potential of metalepsis. The device should also be seen completely beyond its initial connection to the language-based context, which is best done by using Elleström's model of intermedial relations.

The initial defnition of metalepsis is, as mentioned, clearly linked to language, especially written language, in the form of the qualifed medium of literature. Indeed, French narratologist Gérard Genette gave the frst defnition of metalepsis as "any intrusion by the extradiegetic narrator or narratee into the diegetic universe (or by the diegetic characters into a metadiegetic universe, etc.), or the inverse" (Genette 1980 [1972]: 234–235). The all-importance of the narrator is something that all subsequent studies have had a hard time avoiding, even in the case of media types where the existence of a narrator is not necessary, or even impossible. Indeed, theorists explain metalepses in narratorless media, such as painting or music by analogies to literature, fnding that certain devices are ways to imitate a narrative voice where such a thing cannot exist. Even if this is true in many individual cases, it is my contention that it should not be transformed into a rule. Indeed, narrative levels can be both constructed and transgressed without the use of a narrative voice, as I will try to show in the chapter.

Interestingly enough, the transmediality of the concept of metalepsis was alluded to already in Genette's frst theorizations in 1972. First, he explicitly illustrated metalepsis with examples from other media types than literature—such as theatre, paintings (Genette 1980 [1972]: 235) and cinema (1980 [1972]: 237). Then, implicitly, when he used the term 'world' in his other defnition of the concept as a transgression of "the sacred frontier between two worlds, the world in which one tells, the world of which one tells" (1980 [1972]: 236), he suggested the extension of the concept of narration beyond its linguistic frame and its inclusion into the broader spectrum of representation. A decade after the frst defnition, Genette emphasized the importance of representation for metalepsis even more strongly, avoiding the reference to a narrator, and defning it as the "deliberate transgression of the threshold of embedding" (Genette 1988 [1983]: 88). As Jean-Marie Schaeffer pointed out, this redefnition permitted the transfer of the concept to the feld of representation in general (2005: 325, 327).

This expansion of metalepsis beyond only language-based media to the feld of representation in general should have led to a liberation of the concept of its linguistic straitjacket. Unfortunately, this has not really happened. As in Genette's early defnition, where the act of telling is still central even when a certain move towards representation is made, the linguistic basis persisted both in defnitions and in analyses. This is, for example, very clearly the case for Sabine Lang's detailed typology of the so-called paradoxical narrative devices, which includes metalepsis as the most conspicuous case, where the main criterion is the difference between story and discourse (*discurso* and *historia* in the Spanish original, Lang 2006: 32). This is, to an even higher degree, the case for Sonja Klimek's approach, when she claims that narrative levels only concern "written and not visual fction" (Klimek and Kukkonen 2011: 25). This is also the case, admittedly to a lesser degree, even in a number of studies, from Brian McHale's *Postmodernist Fiction* (1987) to more recent ones such as Werner Wolf's "Metalepsis as a transgeneric and transmedial phenomenon" (2005), the anthology *Metalepsis in Popular Culture* edited by Sonja Klimek and Karin Kukkonen (2011), Jan Alber's and Alice Bell's "Ontological metalepsis and unnatural narratology" (2012) or Jeff Thoss's *When Storyworlds Collide: Metalepsis in Popular Fiction, Film and Comics* (2015). These studies all have in common the aim to expand the application feld for metalepsis to representational media in general. However, despite the opening to other media, the studies are still dependent on the concept's initial connection to language, and have the tendency, criticized by Elleström, to compare only two media types, out of which literature is always the basis (2020: 5–6).

It might seem that even the current study does the same thing. However, even if only two media types are studied, the point is to consider the media beyond their essence, and concentrate on their modalities. This is, I argue, a more productive way of approaching transmediality than studying many different media, because no matter how many media one might study, there would always remain cases that could contradict the results. Looking at a medium's basic ingredients, which is the aim of Elleström's model of intermedial relations, is a way out from this dilemma. It also helps in expanding metalepsis beyond language to representation in general, since representation is the core of what Elleström calls the semiotic modality.

According to Elleström's model, each medium consists of four elementary modalities: the material, the sensorial, the spatiotemporal and the semiotic. These modalities form a transmedial basis on which more complex features can be constructed. These features, that Elleström frst called "compound media characteristics" (2014) and now changed to "represented media characteristics" or simply "media characteristics" (2020: 80), cannot be transmedial in the same way as the elementary modalities, since indeed they are more specifc, and also more or less closely linked to a specifc medium. This can be explained in a simple way by the device of the subjective camera, which is a media characteristic linked to a medium where a camera is necessary. Still, at a level below, the level of elementary modality, a subjective camera has to do with the sensorial modality, more exactly with the visual mode, which belongs to the transmedial basis. It is exactly by acknowledging such properties of lower complexity that one can discover certain similarities between devices from different media which seem completely incompatible at a frst look.

Paradoxical narrative devices such as metalepsis are examples of represented or compound media characteristics. As such, they could hardly be considered as fully transmedial. Not even narration in general can be considered to be a trait that can be found in all media, despite the so called narrative turn, according to which everything is narrative (see, for instance, Richardson 2000 and Hyvärinen 2006). But this is not a good reason to reduce the applicability of the paradoxical narrative devices to only one qualifed medium: literature. The main reason is the fact that narration can hardly be seen as only a literary trait. The other reason is, as I already mentioned, that metalepsis is linked to representation in general rather than to narration through language only.

Still, as already mentioned, there is a tendency even today to concentrate on two media-specifc aspects of metalepsis, which both have reductive effects. First of all, there is the language-based aspect, especially highlighted in the insistence on the need of a narratorial voice for the metalepsis to occur. Secondly, there is the fctional aspect, since several theorists fnd that metalepsis can only occur in fction. Some even fnd metalepsis to be "a popular candidate for a fction specifc category" (Bareis 2008: 164).

I argue that both these aspects can be approached more productively by using Elleström's model of intermedial relations (2020). Indeed, both narratorial voice and fctionality are media characteristics that can be studied in relation to the basic modalities of the media types in which metalepsis occurs, rather than being considered as necessary ingredients for the concept of metalepsis. I will approach both these aspects below, on the basis of a number of examples from two media types.

#### 6.2 Metalepsis in Literature

I start with an example of metalepsis in the qualifed medium where its existence is undisputed: literature. The example is, moreover, one of the best known cases of metalepsis, and has been often analysed as such by theorists as Dorrit Cohn (2005: 123), Sabine Schlickers (2005: 161), Sabine Lang (2006) and Bénédicte Vauthier (1999). It comes from Spanish author Miguel de Unamuno's novel *Mist* (*Niebla* in original), written in 1907 but published only in 1914 (2013 [1914]).

At frst view, the novel seems to be realistic, even though there is a certain play with the narrative instance at the beginning. Indeed, the frst pages are narrated by a certain Victor Goti, in the frst person. However, as it appears after these pages, this Victor Goti is not the main narrator. Goti only supposedly writes a prologue to the novel we are reading, and contradicts some of the facts therein, pretending to know the characters better than Unamuno. Unamuno himself responds to this prologue, in a post-prologue, defending himself against Victor Goti's accusations, before letting a heterodiegetic narrator (who will later prove to be himself) take over the narration. This narrator tells the story of a certain Augusto Pérez, a rich young man without a clear goal in life. After his mother's death, Augusto falls in love, or rather thinks that he falls in love, with Eugenia, a young pianist of lower social status. Eugenia, however, does not love Augusto, but runs away from him with a young man of even lower social status. Having even lost this last goal in life, Augusto decides to commit suicide, but not before going to Salamanca to discuss the matter with a certain Miguel de Unamuno, who had written an essay on the question of suicide from a philosophical point of view.

It is this dialogue with Unamuno that constitutes the famous metalepsis in this novel. Indeed, during their talk, which takes place in chapter 31 out of 33, Augusto fnds out that he is only a character of fction created by his interlocutor. Unamuno reveals to Augusto that he is destined to die, not to commit suicide, as he was planning. Augusto is shocked at frst, but then starts to contradict Unamuno, questioning whether he too exists in the real world. Something that we, as readers, could ponder upon too, since the Unamuno of the story is hardly the same Unamuno as the one in the real world. Augusto leaves Salamanca after the dialogue in a very hasty manner, arrives at home and discusses his fctional status with his servants, who do not understand the matter. Eventually, Augusto dies, but it is not stated clearly if it is through suicide by eating compulsively or if it is by Unamuno's narratorial decision.

In chapter 31, the narrator abandons his heterodiegetic status and becomes homodiegetic, meaning that he narrates at the frst person, as a character in the story. This is how this transition is done, at the beginning of the chapter: "The storm in Augusto's soul ended in a terrible calm. He had resolved to kill himself, to put an end to that self which had been the cause of all his misery. But before carrying out his plan it occurred to him, like a drowning sailor who grasps at a near plank, to come and talk it over with *me, the author of this whole story*" (Unamuno 2013 [1914]: 147, my emphasis). Or rather, I would argue, the narrator of the story, since the character of Unamuno is fctional too, as I already suggested.

Admittedly, the metalepsis here respects Genette's defnition, since the frontier between the world in which one tells and the world of which one tells is clearly broken. However, it is interesting to notice that it is not until this chapter that the narrator appears as a character in the storyworld. Indeed, not only does he use the frst person when speaking, but he is represented as a character, who physically meets Augusto. It is what theoreticians, such as Marie-Laure Ryan (2006: 207) and Monika Fludernik (2005: 79), classify as an ontological metalepsis, the most conspicuous and shocking example of the different classes of metalepsis. What I consider noteworthy, however, is that it seems that even in the case of such a typically literary device, there is a need for a kind of representation that goes beyond the strictly literary context. What I mean is that the narrative levels are less conspicuously transgressed through the literary device of the voice of the narrator. They are more clearly transgressed by a represented physical meeting between two characters that are not supposed to exist in the same world. It is true that the meeting becomes all the more shocking thanks to the narrator's using the frst person, and referring to the story as his own creation, but such a meeting could be described without the use of a frst person narrator, just by letting the different worlds collide. The only requirement is that the worlds should be at different, incompatible narrative levels.

It is probably the requirement of 'narrative levels' that links metalepsis so closely to language. But, as I already mentioned, narration can be realized through other means than just language. And indeed, Genette's defnition of the two worlds on the basis of 'telling' seems to be an unfortunate slip of the tongue. What should be at stake are the levels of representation, and metalepsis should be the violation of the frontier between different levels of representation. Neither the defnition based on the notion of a narrator (I recall: "intrusion by the extradiegetic narrator or narratee into the diegetic universe") is particularly relevant, since the violation can be achieved by other means than the narrator. Just to mention one example, an object that moves from one narrative level to another narrative level would be an undoubted case of metalepsis too.

What is particularly interesting is that the language-based aspect of metalepsis seems to be hard to get rid of even in other media. One example is the cinematic adaptation of Unamuno's novel, made by Spanish director Fernando Méndez-Leite in 1975. The meeting between Unamuno and Augusto is represented in the flm too, but Unamuno is heard as a narrator in voice-over, addressing the viewers and explaining that the meeting is between him, as author, and a character created by him. The words are very close to the text of the novel. However, two additional devices typical of the cinematographic medium are used. The frst one is the editing technique, used here in such a way as to give the viewer the impression that Augusto appears and then disappears from Unamuno's room out of nowhere, almost in a supernatural way. It is as if Unamuno created him right there and then in front of us, the viewers. This is a very good example of how other means than strictly linguistic, in this case visual, can be used in order to create a metaleptical effect. The question is, however, whether the effect would have been metaleptical if the viewer had not been aware of the different narrative levels established previously, in a language-based way, by the voice-over. I argue that even if it is not done in the flm, such an establishment could have been done by other means, and choosing to repeat the narrator's words from the novel is probably the director's choice to show his debt to the original media product, by incorporating the part that can be incorporated in the new medium. Admittedly, to use Elleström's words, the text has been transmediated (2014), from written to oral, but it is still a reference to the original work. The second cinematographic device is the use of sound effects. There is a certain change in the quality of the sound directly before Augusto's appearance and until his disappearance, clearly perceptible for the viewer, who gets the impression that the narrative voice changes its status from being intradiegetic to being extradiegetic—that is, addressing the viewer directly, as a voice-over. It is still the same voice, but it is as if it came from another place and sounding as if reading out loud from a novel. Here, I argue, the change is solely done by devices belonging to the mode of sound. A narrator's voice is imitated in its aural characteristic, not claimed by words. It is something that cannot be done in a novel, where the sound as mode is simply absent.

This whole discussion around the ways the metaleptical encounter between the narrator and the characters created by him is represented can seem rather academic. As a matter of fact, even if we can conclude that there is only one narrative level in the scene, since the narrator is actually just a represented character, this does not mean that there is not a metalepsis there. Actually, the metalepsis had already occurred before the encounter, when the author fctionalized himself, moving from the extradiegetic level to the intradiegetic level. This is shown more clearly by the cinematographic media product, where it is much more important to expose visually the moment of this diegetic change so that everything becomes more comprehensible.

Beyond this rather academic analysis, the question is why metalepsis is used. The question is relevant at a general level, since metalepsis is a device which points to the artifciality of a media product, so it should rather be avoided in flms aiming at realism. In the specifc case of *Niebla*, one answer is suggested in the novel itself: it could be that becoming a character in a literary work is a way to immortalize oneself. And indeed, it is a strategy that seems to work. A little more than a century after its publication, Unamuno is, in a certain way, paradoxically present in the midst of his readers. But is that a result of his transformation into a character in his own book? Is it not rather the result of the success of the books? Beyond these questions, there is something particularly interesting in the way in which reality and fction meet in such a paradoxical way. What happens is that a confusion is created at different levels. For instance, is it the author who moves towards the inferior narrative level—*metalepsis*—or is it the character, Augusto Pérez, who moves towards the superior level—*antimetalepsis* according to Genette's defnition? Indeed, Augusto is the one who decides to visit Unamuno. It is true that this decision is ultimately orchestrated by Unamuno, as it is underlined by his lack of surprise and enigmatic smile when Augusto appears in his offce (2013 [1914]: 147). However, the development of the situation eventually surprises Unamuno, frst when Augusto tells him that "you are nothing more than a pretext for bringing my history into the world" (2013 [1914]: 149), then when he acknowledges his reaction: "These sallies of Augusto were beginning to make me uneasy" (2013 [1914]: 151). The gradual loss of authority over his own creation complicates things, putting into question whether the details are planned by the author or not.

Before this dialogue, the narrator's authority had not been put into question at all, with the exception of the already mentioned prologue. Even then, Victor's words had not really questioned Unamuno's authority in general; they only questioned his credibility concerning the version of the facts. On the contrary, the prologue establishes the extradiegetic status of the narrator, Unamuno, which is reinforced in chapter 31, when the narrator refers to himself as "me, the author of this whole story" (Unamuno 2013 [1914]: 147). Thanks to this clear distinction between the different narrative levels, the metalepsis is very effective in *Niebla*. The fusion of the extradiegetic and the intradiegetic levels in chapter 31 has indeed a strong effect on a reader who until then was certain of being able to keep the levels apart. This gives also the opportunity to double entendres in the novel, such as the one when Augusto tells his servant: "I come from another world, Liduvina, and I am going to another world" (2013 [1914]: 155). This other world could, indeed, be the world of the dead, but also the world of the author, the extradiegetic, real world. This double entendre, besides being a game, can be a way to underline the diffculty to distinguish between fction and life, between dream and reality.

In conclusion, one of the main elements of *Niebla* is the fusion between different narrative levels, especially between the extradiegetic level of the author and the diegetic level of the characters. Moreover, *Niebla* anticipates what Genette would defne as one of the possible effects of metalepsis: the existential doubt. Indeed, Augusto comments upon the meeting with his creator by making an analogy between the act of writing and God's act of creation. The result could be that the real Unamuno could in his turn be a fctional creature, the product of God's dreams. This is actually suggested by Augusto: "Very well, then, my lord creator Don Miguel", he says, using an ambiguous "lord creator" that could refer to God, "you too are to die, you too! And you will return to that nothing of which you came! God will stop dreaming you!" (2013 [1914]: 154). Genette's words about the existential doubt that can be created by metalepsis are close to Augusto's: "The most troubling thing about metalepsis indeed lies in this unacceptable and insistent hypothesis that the extradiegetic is perhaps always diegetic and that the narrator and his narratees—you and I—perhaps belong to some narrative" (Genette 1980 [1972]: 236). In the following parts, I will investigate whether metalepsis can be achieved in similar ways in a media type where language is less central, that is, flm, to see whether the same effects can be created using other media characteristics.

#### 6.3 Metalepsis in Film

One of the reasons why I choose to analyse metalepsis in flm is that flm consists of complex modes which can be combined in creative ways. The material modality of flm is, for instance, fexible enough, as we shall see, to incorporate other media types, or to give the impression of incorporating them. For instance, since the basic medium of flm consists of moving images, it can easily incorporate still images. And since flm has a sound track, it can easily incorporate any media type that is based on sound, such as music, radio, and so forth. Thus, I can analyse the effects that other modes, which are less important in literature, can have on metalepsis.

Another important reason why flm is a suitable study object in this context is that it has a very special relationship to literature. This is not only because flm, as a qualifed medium, has a strong tradition of adapting literary works (which is a form of transmediation, to use Elleström's words [2014]), but also because flm, as I just mentioned, can incorporate written text. For instance, since the visual mode is of great importance in flm, a written text can be shown on the screen, (which is a form of media representation, according to Elleström's model [2014]). Moreover, flm can imitate more complex literary media characteristics in very interesting ways. The use of a narrator in voice-over, as we saw in the adaptation of Unamuno's novel *Niebla*, is just one such example, even if not the most interesting one, since it is quite obvious. In intermedial theory, this is what Werner Wolf would call "intermedial imitation" and Irina Rajewsky "intermedial reference". It is, as Wolf puts it, when "the signifers of the work and/or its structure are affected by the non-dominant medium, since they appear to imitate its quality or structure" (2002: 25), or, as Rajewsky puts it, when "the given media-product thematizes, evokes, or imitates elements or structures of another, conventionally distinct medium" (2005: 53).

The adaptation of *Niebla* is illustrative as an example of the diffculty of completely breaking with the literary model, in the case of metalepsis. As I already mentioned, I argue that a narrator's voice-over was not actually needed in order to prepare the feld for the metalepsis to occur and to be understood as such by the public. Meanwhile, it is understandable that the director imitates certain details from a specifc source media product (the specifc novel by Miguel de Unamuno) when adapting it to another media type. It is also understandable that metalepsis, being a very complex media characteristic, cannot be imitated without imitating its smaller units, such as the narrator's voice. What is less understandable meanwhile is that many flms imitate a narrator's voice as a general feature (that is not from a specifc novel) in order to achieve metalepsis. I will present here two such cases, *Stranger than Fiction* (2006) and *Le tableau* (*The Painting*) (2011), but I will also go beyond the narrator's voice and analyse whether other flmic media characteristics are used when staging the metalepses. By doing so, I aim at showing that metalepsis is not necessarily dependent of a narrator's voice. The third case is an example of a flm, *Woman at War* (2018), where the narrator's voice is not used at all in the creation of metalepsis.

#### *6.3.1 Metalepsis Through Narrative Voice in Film*

The frst flm I choose to analyse is Marc Forster's *Stranger than Fiction* (2006). As the title suggested, the play with the relationship between fction and reality is the main theme of this flm, and is achieved through a metalepsis which actually reminds of the one in *Niebla*. Indeed, the main character, Harold Crick, played by Will Ferrell, hears the voice of a narrator, played by Emma Thompson, who seems to create the events he is experiencing, and that the viewers are viewing on the screen. The viewers hear the narrator's voice-over as well, and have no reason to consider this strange at all. Indeed, this is a case of an intermedial imitation of a narrator's voice, common for many flms. The frst time the viewer reacts is when the character himself reacts, very early in the flm, when starting to hear the voice, which describes him and his ongoing actions.

Interestingly enough, what the metalepsis actually does is to draw the attention to the artifciality of the device of a narratorial voice-over. What in literature is unavoidable, that is, the use of a narrator narrating the events, is unnatural in flm, and has become an accepted device only because of its frequent use. The existence of an extradiegetic voice at the same level as the storyworld is actually a logical impossibility, since the two worlds should not be able to coexist. The ability to keep the worlds apart, even if they are represented as coexisting, is only achieved through training, and a viewer who experiences such a case for the frst time would probably react with the same surprise as does Harold.

The similarities with *Niebla* do not fnish at this point. Indeed, not only does the character have a physical contact with his creator, but he also hears that the creator plans his death, as Unamuno did for Augusto. Harold reacts at frst as does Augusto, trying to change the course of the events. For instance, he spends all his time at home trying to control his destiny by doing nothing and thus not taking any risks, but even so, he is almost killed by a wrecking crew which demolishes his apartment, because of having mistaken the building for an abandoned one. Having thus realized that he cannot control the plot, Harold accepts his fate and tries to enjoy the time he has left. One day, by coincidence, Harold hears the voice he had in his head when watching a television interview with author Karen Eiffel. Indeed, the voice is the same as the narrator's, but as a character. When confronted by Harold, who did his best to fnd her, Karen Eiffel is not aware of what she had caused. She is just a character in the story, but in a strange loop, it is the story she writes. Therefore, she is shocked too.

It is certainly not by chance that a British actor has been chosen for the role of Karen Eiffel. First of all, the British accent of the narrator's voice contrasts in an obvious way with the American accents of the other characters, who are supposed to live in Chicago. This contrast contributes to creating the impression that the voice is heard at another narrative level, the extradiegetic one, that is, the level that should not be heard by the characters from the diegesis. The fact that the voice had been disembodied until the author frst appeared on the TV set also contributed to this impression, as did the sound effects. Besides, also by effect of contrast, British accent can give an elevated impression, which can be considered as a reference to literature. All these devices are flmic media characteristics, relying on the mode of sound, and could thus not be used in written literature, where the mode of sound is absent. It is therefore an interesting example of how intermedial imitation can be used in innovative ways, thus casting new light on concepts that previously were considered to belong to literature.

When Karen Eiffel, the author, appears as a represented character in the flm, her accent is still British, and her voice is the same, but it loses its distinctiveness. This is, I argue, the result of the viewers being used to the flmic conventions. A voice belonging to a character who is visually represented is less distinctive than a disembodied voice-over. Even if the voice-over is known by the viewers to belong to one character, not showing the character and using certain sound effects can easily give the impression that the utterance comes from another level at the moment it is made. That is why the frst sign of metalepsis, when Harold actually hears this strange voice, is so shocking for both Harold and the viewers.

The second example of metalepsis in flm that I choose to analyse is Jean-François Laguionie's *Le tableau* (*The Painting*) from 2011. It is an animated flm which can be considered as a metalepsis from the start until the end, since its frst scene is a metaleptic entrance into the world of flm, where the viewer stays until the last scene. Besides, the flm is a very good example of different types of metalepses, according to classifcations established by theoreticians in the feld, showing how flm as basic medium can contribute to a better understanding of these classes and of metalepsis in general.

The most important distinction the flm addresses is the one between vertical metalepsis and horizontal metalepsis. This is a distinction frst made by Klaus Meyer-Minnemann and Sabine Schlickers (2005) and by Sabine Lang (2006). Their defnition of vertical metalepsis as a transgression of the border between worlds situated at different narrative levels (the world of the narrator and the world of the narrated events) is close to Genette's initial defnition, that is, "any intrusion by the extradiegetic narrator or narratee into the diegetic universe" (Genette 1988 [1983]: 234–235). Horizontal metalepsis, defned as the transgression of the borders between worlds situated at the same narrative level (Meyer-Minnemann and Schlickers 2010: 140), is an expansion, rather than a contradiction, of Genette's defnition. Still, the concept of the horizontal metalepsis has raised certain controversy among theoreticians. Sonja Klimek considers, for instance, that we should "respect Genette's initial defnition" and restrict the use of metalepsis only to cases where there is a vertical infringement, that is, between "the world of the creator" and "the world that is created within the artefact" (Klimek 2011: 26). My analysis of the metalepses in *Le tableau* will try to show how cinematic medium can help to reconcile these views.

When I claimed above that the whole flm is a metalepsis, it was because it is framed by two vertical metalepses, which form the entrance into and the exit from the world of the flm. The whole story thus becomes the result of the initial metalepsis, and its fctional status is regularly reminded by metaleptical comments or by metaleptical scenes. The frst scene is not only worth studying as an interesting case of a vertical metalepsis, but also because it occurs both visually and verbally. Indeed, while the camera is zooming in, giving the viewer the impression that the frame is left behind and that the limit established by the canvas is transgressed, a character in the painting, a girl called Lola, comes to life and addresses the viewers with the words: "Voilà ! Vous venez de pénétrer dans le tableau" ("Here you are! You just entered the painting"). Such a comment could be interpreted as a metalepsis of the discursive kind according to Dorrit Cohn ("métalepse discursive" in the French original), who found it more "harmless" ("inoffensive" in the French original) than the "diegetic metalepsis" ("métalepse de l'histoire" in the French original) (Cohn 2005: 122). John Pier would consider this example a "minimal metalepsis" ("métalepse minimale" in the French original) (Pier 2005: 249), considering that the transgression of the frontiers is only suggested in such a case, not real. Monika Fludernik would classify it as "rhetorical metalepsis" (2005: 79–81) as opposed to the "ontological metalepsis", according to a classifcation made previously by Marie-Laure Ryan. According to Ryan, ontological metalepsis "opens a passage between levels that result in their interpenetration, or mutual contamination," while rhetorical metalepsis only "opens a small window that allows a quick glance across levels, but the window closes after a few sentences, and the operation ends up reasserting the existence of the boundaries" (Ryan 2006: 207). Indeed, the comment is quite close to the example these theorists use to illustrate these kinds of harmless metalepses, namely an extract from Honoré de Balzac's *Les illusions perdues* where the narrator addresses the narratee with the words: "While the venerable churchman climbs the ramps of the Angouleme, it is not useless to explain the network of interests into which he was going to set foot" (quoted, for instance, by Fludernik [2005]). In the French original: "Pendant que le vénérable ecclésiastique monte les rampes d'Angoulême, il n'est pas inutile d'expliquer le lacis d'intérêts dans lequel il allait mettre le pied."). An even better example, also mentioned by theorists, comes from another well-known French nineteenth-century novel, Alexandre Dumas's *The Count of Monte Cristo*:

We will leave Villefort on the road to Paris, travelling—thanks to trebled fees—with all speed, and passing through two or three apartments, enter at the Tuileries the little room with the arched window, so well known as having been the favorite closet of Napoleon and Louis XVIII., and now of Louis Philippe. (Dumas 2019 [1844]: chapter 10)

What complicates things in the flm, however, is the visually represented 'penetration' of the painting. It could be argued that this is only a metaphor of the act of immersion into the painting, since we as readers or viewers penetrate in some way the world represented by the media product. But this interpretation is a part of the semiotic modality. At a lower level, the spatiotemporal modality, the fact remains that the zooming in of the camera gives the viewer the impression of entering a space, which is only a virtual space, represented on the screen. Indeed, neither the painting nor the space represented by the painting exists in reality. Still, by using visual devices offered by the cinematic medium, the viewer gets the impression of transgressing the two borders: frst the border between reality and the world of Lola as narrator, and then the border between that world and the world of the painting. This second border is even more clearly marked by the frame of the painting.

My point is that Lola's verbal comment is not really needed to give the viewer the impression of the metaleptical transgression. What the comment does is that it reminds the language-based original defnition of metalepsis as the transgression of the frontier between "the world in which one tells, the world of which one tells" (Genette 1980 [1972]: 236, see even above). The fact that it is an intrusion into an ontologically different world appears more clearly in a movie, thanks to the visual dimension, but this does not mean that there are no narrative levels involved in this transgression. The world in which one tells corresponds to the world of the camera before it penetrates through the frame into the world of the painting, after which the character of Lola can enter in direct dialogue with the viewer. To be more precise, this example corresponds to Genette's frst defnition of metalepsis, "any intrusion by the extradiegetic narrator or narratee into the diegetic universe" (Genette 1980 [1972]: 234–235), since it is the narratee, meaning the recipient of the narration, who is entering the diegetic universe of the painting. Following John Pier's dichotomy, this would be a descendant metalepsis, of the kind that Pier exemplifes with an extract from George Eliot's *Adam Bede* (1859), when the narratee and the narrator together transgress the demarcation line between the narrative levels (Pier 2005: 250).

However, there are even more interesting examples of metalepses in this movie, especially of the kind that could be analysed as horizontal (that is, I recall, metalepses which occur between worlds at the same narrative level). Indeed, after the metaleptical start, the flm represents the events in the world of the painting in a traditional way, without several levels or worlds involved, making the viewer forget the initial metaleptical intrusion. All of a sudden, Lola, who has abandoned her ability to communicate with an extradiegetic narratee and has become a character inside the world of the painting, arrives at the frontier of her world, the canvas of the painting, and jumps out of it, followed immediately afterwards by two other characters. They enter an extradiegetic world, an ontologically different world, the world of the creator, the painter, a world which could be the same as the initial world of the narratee, the viewers' world. This is of course a hierarchically different world, so the transgression is undoubtedly vertical. According to John Pier's typology, this would be the case of an ascending metalepsis, which he exemplifes with an extract from Mark Twain's *The Adventures of Huckleberry Finn* (1883), more precisely when Huck goes up one level and enters Mark Twain's world (Pier 2005: 252). This transgression is so important in the movie *Le tableau* that exactly the moment of the jump is used on the DVD cover.

What happens next in the flm, though, is very interesting from a theoretical point of view. When Lola jumps into her creator's world, she lands on the frame of another painting. Shortly after that, the second painting falls down from the wall because of Lola's weight, and Lola falls through the canvas into the world of this second painting, representing a war. She becomes a character inside that world, able to interact with the soldiers there.

This example of metalepsis could be analysed as an ontological metalepsis, as opposed to the more 'rhetorical' or 'discursive' cases studied above. Indeed, Lola literally travels between different worlds in a way that could be analysed through the lenses of the possible world theories. According to Alber's and Bell's attempt to apply these theories to metalepses (Alber and Bell 2012), such a travel between different ontological worlds would correspond, at least partly, to Daniel Lewis's concretist philosophical position. According to the Concretists, as opposed to Abstractionists, such as Nicholas Rescher, Saul Kripke, Jaakko Hintikka or Alvin Plantinga, an individual cannot exist simultaneously in the same world. Thus, when Lola penetrates the world of the painter, she disappears from the world of the painting.

What is more important is that this is a horizontal metalepsis, since Lola actually crosses the frontier between two worlds situated at the same level: the worlds of two paintings hanging on the same wall. It is this kind of metalepsis that Sonja Klimek fnds incompatible with Genette's original defnition, since the transgression is between worlds, not between narrative levels. But what this movie example shows is that in order to move from one painting to the other, Lola must necessarily pass through her creator's world, the world 'in which one tells'.

One could argue that this example cannot be applied to literature, where the movement between two parallel worlds could occur without a necessary transition through a higher narrative level. But does the fact that such a transition is not represented imply that it does not occur? Is it not rather that the transition is implicit, and does not have to be spelled out explicitly? My answer would be that this depends. In some cases, such as when a character from a fctional work appears in a later fctional work, it is probably better to talk about what Marie-Laure Ryan calls "transfctionality", meaning "the migration of fctional entities across different texts" (Ryan 2013). But in the cases when such a migration is actually represented, when a character is represented when he or she literally leaves the storyworld, the paradoxical dimension is more obvious. However, even if I cannot provide a defnitive answer as to the nature of horizontal metalepsis in the context of this article, it has appeared as obvious that a transmedial perspective is necessary in order to better understand this narrative device.

Before going over to the following flm, I want to mention succinctly the metalepsis at the end of the flm. What happens there is that Lola fnally meets her creator, a painter, and talks to him, the way Augusto does with Unamuno. Her comment to him is, "But who painted you?" A question which is relevant even from a theoretical point of view since, indeed, the painter from the flm is nothing else than represented, even though he is not an animated, but a live action character. The question also underlines that metalepsis has the same effect in flm as in literature, as we saw in *Niebla*: making one doubt about his or her own existence.

#### *6.3.2 Metalepsis Without Narrative Voice in Film*

The third flm to be analysed is also the one in which the metalepses are created without any implication of a narrator. Thus, the language-based features are overshadowed by other features, both visual and aural ones. What is particularly interesting is that the aural features are not related to language, but to music, and the so-called world in which one tells has no relevance, since there is no narrator speaking. Therefore, the flm is an illustrative argument against the language-based defnitions of metalepsis, and a good example of the transmediality of the concept.

The flm in question is an Icelandic-Ukrainian coproduction from 2018 entitled *Woman at War* and directed by Benedikt Erlingsson. The plot is completely centred on the main character, Halla, a choir conductor and eco-activist, played by Halldóra Geirharðsdóttir, and on her attempts to sabotage an aluminum plant in the Icelandic highlands. From this point of view, the flm would ft Jørgen Bruhn's defnition of ecomedia, that is, "representations of environmental issues in any specifc media product" (Bruhn 2020: 121). However, at the same time, as Halla sabotages this plant in ingenious but dangerous ways, an application she made years earlier to adopt an orphan child from Ukraine is approved. The flm follows the two different lines of plot in parallel, thus showing that illegal activism can very well be compatible with a humanistic act.

Even if the plot is thrilling, the merits of the flm lie rather in its cinematic language, as critics have noticed. Just to mention one reaction, Peter Bradshaw, from *The Guardian*, fnds the flm "confdently and rather stylishly made" (Bradshaw 2018). However, Bradshaw fnds this stylishness a bit exaggerated and unmotivated, especially when it comes to the use of music:

There is also a lot of Icelandic folk music from a tuba, accordion and drums trio, and a rather beautiful singing group, whose performances turn out to be diegetic. That is, the camera pans around to show the musicians themselves, standing weirdly, incongruously in the background of the shot, variously trilling or parping away. It is a comic effect that is a bit distracting, subject to diminishing returns, and which ironises and undermines the action and obstructs your natural tendency to invest emotionally in Halla's dilemma. (Bradshaw 2018)

It is not my aim to contradict Bradshaw's opinion, but as a specifc feature, the use of music is one of the most interesting details of *Woman at War*. Besides, it is through music that metalepsis is constructed in this flm, and that is done in innovative ways. This is a good example of what Werner Wolf in an article calls 'musical metalepsis', which he contrasts with verbal or pictorial metalepses, that is, metalepses constructed through verbal or pictorial means (Wolf 2019). As it appears clearly in Wolf's article, instrumental music alone, as a media product, cannot achieve metalepses. It does so only when it is incorporated in flm: "what in 'absolute' instrumental music would be impossible but which the plurimedial combination of music, narrative and the moving image in the sound flm can produce, namely musical metalepsis" (Wolf 2019: 29).

I follow Wolf's defnition of the concept of musical metalepsis, and analyse how music can be used in order to create metalepsis in other media where it is used, since *Woman at War* is particularly interesting from this point of view. Indeed, Bradshaw is right to point out that what is important in the use of music in this flm is the performance of the musicians. Admittedly, the music itself is important in the flm, from different points of views. Already the fact that Halla is a choir-leader, with several scenes presenting her while leading the choir, is a sign that music should be given particular attention. The score itself, specially composed for the flm by Davíð Þór Jónsson, is original in its minimalism. Generally, it seems to be used in conventional ways, meaning the ways it has been established in classical Hollywood flm. Here is a short list of the possible functions of music in such flms:

It can establish setting, […] it can fashion a mood and create atmosphere; it can call attention to elements onscreen or offscreen, thus clarifying matters of plot and narrative progression; it can reinforce or foreshadow narrative developments and contribute to the way we respond to them; it can elucidate characters' motivations and help us to know what they are thinking; it can contribute to the creation of emotions […] [it] can unify a series of images […] [and] encourages our absorption into the flm by distracting us from its technological basis […]. (Kalinak 2010: 1)

All these functions apply to the use of music in *Woman at War*. However, there is much more to it. To start with, still in the traditional way, music highlights the double action of the flm. The Icelandic music trio, who play the drums, the piano and the accordion, generally play music that reinforces the atmosphere of danger that has to do with Halla's sabotaging of the aluminium plant. The other group, a Ukrainian female choir singing folk songs, is used for the other plot line, that is, the adoption of the Ukrainian orphan. The fact that the two groups only play together at the end, when Halla meets the girl she adopts, could be seen as a metaphorical use of music, to "clarify" the plot, as Kathryn Kalinak mentions in the quotation above.

As Bradshaw notices, music is also used in unexpected ways in *Woman at War*. One of these ways enters indeed in complete contradiction with what Kalinak listed as a function of flm music, namely that it is "distracting us from its technological basis". It is exactly when music is used metaleptically that this happens. Indeed, already in the frst scene, when the credits still run, the music suddenly seems to lose its extradiegetic status, since the Icelandic trio appears in the frame, when the camera pans the beautiful Icelandic landscape. They stay there, "weirdly, incongruously" in the feld, as Bradshaw rightfully notices. What Bradshaw misinterprets, however, is that music thus turns out to be diegetic. Indeed, just because the musicians are visually represented does not mean that they become characters in the diegesis. It is a misconception, only caused by the conventions from traditional flms, that everything that is seen on the screen exists at the same level. It is enough to reconsider this by applying the mode of hearing, which should apply in the same way: everything that is heard in the flm exists at the same level. Which obviously is not true, since extradiegetic music, or extradiegetic narrators in form of voice-over, are used frequently. So why could not extradiegetic elements be visually represented on the screen without "distracting" the viewers, as Bradshaw claims?

There are, indeed, clear arguments against considering the music in the flm as diegetic. Actually, the music is not heard by the characters. Only two characters hear the music the same way as Harold Crick hears the narrator's voice-over: Halla (a couple of times) and a Spanish backpacker who happens to be close by when Halla is hunted in the highlands (once). When these two characters show that they hear the extradiegetic music, it is when metalepsis actually occurs. Because, I argue, until the two different levels, the extradiegetic and the diegetic, interact, there is no real transgression. Or rather, I should say, the transgression is covered the same way as the use of extradiegetic music is covered by its frequent use. Because, as I already mentioned above in the analysis of *Stanger than Fiction*, all coexistence of diegetic and extradiegetic elements is transgressive, and thus metaleptical. So, what these examples show is that metalepsis actually loses its raison d'être when it is used in conventional ways.

The extradiegetic and the diegetic collide in another way too in *Woman at War*. This is also through music, or rather through the visual representation of the performance of the musicians. Indeed, the Icelandic trio actually interferes with the events in the diegesis at least three times. Two of these instants involve the use of music. Indeed, by changing the rhythm, the musicians warn the two characters that are able to hear them (Halla and the Spanish backpacker) that something dangerous is going to happen. In both cases, the characters react, and the course of action is changed: rather slightly in the case of the Spanish backpacker, who turns his head and discovers Halla running in the felds, and more importantly in the case of Halla, who realizes that the police are stopping people at the airport, and decides not to take that fight. The third case is even more clearly metaleptical, since the musicians physically interfere with the events in the plot. It is when, while playing the extradiegetic music as usual in Halla's apartment, they decide to put on the TV set so that she can fnd out that the police have started the hunt. The TV set thus functions metaleptically as well, since it is at the same time both a diegetic screen and a metadiegetic screen, according to the distinction made by Andrea Virginás in this volume (2020).

In conclusion, the musical metalepses in *Woman at War* are particularly interesting to study in a transmedial perspective, and by concentrating on the modalities of the media product, according to Elleström's model of intermedial relations (2020). Indeed, what appears clearly is that when the mode of seeing is applied to extradiegetic music, that is, when the musicians are visually represented, what was hitherto a conventional use of extradiegetic music can be experienced as metaleptical. More importantly, when the effects that the extradiegetic world has on the events in the diegesis are visually shown, the metalepsis becomes clearer for the viewer.

#### 6.4 Conclusion

This chapter did not aim at changing the defnition of metalepsis. Its frst defnition, formulated by Genette in 1972, is still valid as a general starting point: "any intrusion by the extradiegetic narrator or narratee into the diegetic universe (or by the diegetic characters into a metadiegetic universe, etc.), or the inverse" (Genette 1980 [1972]: 234–235). I tried to show that metalepsis is a transmedial device, which can appear even in media products that are not language-based. What seems to be mandatory is the capacity to represent. And I thereby do not mean to represent in general, because all media represent in some ways, but to represent in some detail such features that are necessary for perceiving, for instance, ontological clashes and narrative levels.

What I particularly wanted to show is that the concept of narrator can obstruct the expansion of the applicability of metalepsis to other media. An intrusion, or rather a transgression, is needed, between two worlds, but these worlds do not necessarily have to be created by a narrator's voice. Consequently, the initial defnition can be improved, but it already possesses a fexibility, since it mentions "diegetic universes". Admittedly, it lacks even a specifc reference to horizontal transgressions, that is, between parallel universes, but I argue that that kind of transgression can be studied within the same framework, since the parallel universes are necessarily created from a superior level.

In order to prove my point, I started with the media type in which metalepsis was frst defned, that is, literature, and studied one of the best known examples of metalepsis: the meeting between Unamuno, the author and the narrator of the novel *Niebla*, and a character created by him, Augusto Pérez. A closer look showed that even if the narrator is important in that scene, the most shocking is the representation of the whole meeting. Indeed, it is frst when Unamuno, the narrator, becomes a character who physically meets Augusto that the metalepsis becomes really obvious and astonishing, both for Augusto (and eventually even for Unamuno) and for us, readers. What is interesting is that the shocking effect can be expanded to existence in general, since ultimately, even in reality, human beings can have the impression of being the creation of a God. This effect is obviously possible even in the case of other media types, as is explicitly mentioned in the flm *Le tableau*.

After the literary example, I analysed metalepses in three flms, chosen not only because they actually contain metalepses, but also because they create it in different ways. The frst example, *Stranger than Fiction*, is also the one which is the closest to metalepsis in literature, since the metalepses are occurring between a narrator, whose voice imitates a narrative voice in literature, and a character created by this narrator. However, the ways in which the metalepses are achieved rely especially on the mode of sound, which is absent in literature. So, even if what we have in this flm is a socalled intermedial imitation of a narrator, the ways in which that is done highlights the transmedial potential of metalepsis. A potential that is even clearer in the other two flms that I analyse, since they take gradually greater distance from the use of a narrator.

The second flm that I analysed, *Le tableau*, starts by representing a narrator who performs a metaleptical transgression, but this initial imitation of a literary narrator is not repeated, and the metalepses later on are constructed with the help of flmic media characteristics. One of these metalepses, constructed by applying to the visual mode, can be used as an argument for the possibility of horizontal metalepses.

The last flm, *Woman at War*, is the one in which no reference is made to literature when it comes to its metalepses. Here too, it is through the mode of sound that the metalepses are principally created, but this time by using the qualifed medium of music. What the analysis shows is that these 'musical metalepses' are all the more effective when the visual mode is used too, showing music as a performance act.

All in all, the analyses have shown that a transmedial perspective on metalepsis is benefcial, even in the case of literature. This is in line with Elleström's view on media not as essential and hermetically closed entities, but as open and fexible concepts (2014, 2020). It has also proved productive to concentrate on the modalities of media, and to analyse how different modes can affect metalepses. What the analyses have shown too is that the complex device of metalepsis can be used to illustrate aspects of Elleström's model of intermedial relations, especially those related to modalities (2020).

#### References


———. 2020. The Modalities of Media II: An Expanded Model for Understanding Intermedial Relations. In *Beyond Media Borders: Intermedial Relations among Multimodal Media, Volume 1*, ed. Lars Elleström, 3–91. Basingstoke: Palgrave Macmillan.

Fludernik, Monika. 2005. Changement de scène et mode métaleptique. In *Métalepses. Entorses au pacte de la représentation*, ed. John Pier and Jean-Marie Schaeffer, 73–94. Paris: EHESS.


McHale, Brian. 1987. *Postmodernist Fiction*. London: Routledge.


Rajewsky, Irina O. 2002. *Intermedialität*. Tübingen: Francke.

———. 2005. Intermediality, Intertextuality, and Remediation: A Literary Perspective on Intermediality. *Intermédialités* 6: 43–64.


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/ by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## Seeing the Landscape Through Textual and Graphical Media Products

*Øyvind Eide and Zoe Schubert*

#### **Contents**


Ø. Eide (\*) • Z. Schubert

University of Cologne, Cologne, Germany e-mail: oeide@uni-koeln.de

L. Elleström (ed.), *Beyond Media Borders, Volume 2*, https://doi.org/10.1007/978-3-030-49683-8\_7

<sup>©</sup> The Author(s) 2021 175

#### 7.1 Introduction: Being in the World

All humans have experiences from being in an environment. For most of us, this is based on movements within some sort of space,1 and the experiences can include cycling in the woods, walking a city street, or riding through mountains. Exceedingly, over the last two centuries, this movement has been supported by engine driven devices, such as cars, trains, and airplanes; in earlier days, oars, wind, and animal energy were used for transport. The experience of existing in the world involves interaction with a landscape where other beings also dwell and machines are used. An environment where humans and other animals coexist can pose risks, support, and cooperation amongst other types of impact for the individual. It provides, in the sense established by Gibson (1986), affordances. The affordances go in different directions, forming networks of connections between beings based on a rich set of possible forms of communication (Ingold 2000).

One key element of communication is the landscape itself. Many fundamental aspects of communication, both in general and of human languages specifcally, are connected to spatial organisation and movement (Levinson 2003; Tversky 2019). This is the case for all types of media used in human communication. Spatiotemporal modes are active in all media products, however different they may be in different types of expressions. The decoding and understanding of these modes as part of the spatiotemporal modality of a media product is always, in one way or another, connected to the recipient's embodied memories of experiences in an environment.

Such memories can be specifcally linked to a concrete reference. In a novel, where the events take place in Prague, the reference to Náměstí Republiky is recognisable as a concrete place by a reader who knows the place. References can also be type based. The island in which Robinson Crusoe was shipwrecked cannot be experienced in the same way as the square in Prague because it does not exist. However, the landscape described is still recognisable by readers based on general knowledge of landscapes, including beaches and forests. The embodiment is also linked to the situatedness of the reading process, as pointed out in Have and Pedersen (2020: 200).

Space, communication about space, and media, including transformations between them, are all interwoven. In this chapter, we will focus on expressions describing landscapes in different media. In order to better understand how different media are used to communicate about and convey understanding of landscapes, one central aim of this chapter is to include a focus on the transformative processes connected to the expression of spatial understanding. We begin by discussing traditional ways of expressing knowledge about landscapes: written and oral texts and maps, and their combination in geocommunication systems. Furthermore, we will use the four media modalities explained in Elleström (2020) as a system for understanding and comparing these media forms as they are used for spatial expressions. By focusing on transformation processes, we will aim to attain a better understanding of the limitations and possibilities within this approach. Moving beyond graphical two-dimensional expressions and written texts describing landscapes into virtual three-dimensional 'landscapes' will enable us to better comprehend the spatial media characteristics infuencing media products.

The relationship between the spaces in a virtual world and the spaces we know as embodied animals is complex and many-faceted. This establishes virtual reality (hereafter: VR) both as mediated expressions (VR is a type of communication device) and as an experienced space, linked to an equally complex set of connections between different forms of embodiment and different sets of affordances.

These claims will be further explored through discussing landscapes in a number of different senses. We will start off with the real environment we move around in as embodied creatures. We will focus on how this has been mediated through centuries and millennia of human culture. We will then move over to the landscapes we move around in through avatars, that is, VR. That enables us to discuss in a more abstract way connections to modelling and modelling processes in general. This will include studying concrete types of spatial expressions through new media, such as VR applications. These are less affected by convention and have a much shorter tradition of developing their ways of using the available affordances, due to being new media in a historical sense. The technical development also changes the affordances themselves.

#### 7.2 Descriptions of Landscapes

The human mind-body system has the capability of linking media products, expressed in a wide range of different media, to embodied experiences. This is used when creating media products based on pre-knowledge on the side of the creator, from oral, written, and pictorial media and dance to flms, computer games, and VR. We will begin this chapter by focusing on traditional ways of making descriptions: texts, written and oral, and maps.2 Furthermore, with two-dimensional static documents as our starting point, we will use the media modalities as a system for understanding these specifc media forms in a structured way. This will show us that they are similar in both material and sensorial modalities, as they are both fat documents approached mainly through seeing.3 The differences between documents carried by computer screens versus paper and similar technical interfaces will not be discussed at this stage.

One main connector between texts and maps is place names, which are used by both media types to reference something external to the text and map. The links from documents to this externality can have different levels of realism. The reference function of place names is fairly similar in text and maps, but their documentary contexts are fundamentally different. This links to the spatiotemporal as well as the semiotic modalities, as we will see below.

#### *7.2.1 Written Texts*

Textual documents using the writing systems we know from languages such as English are generally based on a basic level of sequentiality, where letter follows letter and word follows word. This is the case even when writing directions vary (for instance, left-right, right-left, top-down) and some writing systems have a less strict sequentiality. The sequentiality is expressed in a spatial form and it is never absolute—there is always a possibility of moving back and forth in texts. Some forms of writing, such as certain poetic forms and footnotes, are designed to break the sequential pattern of reading; however, this becomes an addition to the low-level sequentiality. The order of the letters making up a word and the words making up a sentence are fundamental for the semiotic systems of symbols that languages constitute even if this order can be broken and played with. The sequential conventions are elementary parts of the code of writing systems. Still, these sometimes differ and are not as 'absolute' as the space of a fgure, such as a map.

In this sequential structure, a place name is a word (expressed as a number of characters) which is related to other words around it (Firth 1957: 11). Syntactically, the primary relationships are within the sentence. When the same page of text includes several place names, the relationships between the targets of their denotations are not based on the actual locations on the page, but rather on the meaning which can be found in the text. To competent readers, the locations described are decodable through the relationships expressed within the text, in context of their own pre-knowledge. The space of the landscape referred to by the text is a reconstructed space and the semiotic system used in the text, also to relate to the landscape, mostly relies on symbols. The reconstruction of the referenced space happens when the reader interprets the meaning of these symbols, and understands the links between strings in the text and places in the landscape. This reconstruction is not dependent on whether the landscape is understood as a real place by the reader, but rather on whether it is recognisable or imaginable. This is partly based on the embodied experiences discussed above, partly on impressions from media products, and partly on other inputs. This implies that the distinction between VR and other media is not essential for our ability to reconstruct spaces.

The spatial relationship between places can be established in two different ways. It can be expressed directly by relational language constructions such as 'north of' or 'down the river.'4 In this case, landscape descriptions can be understood even if they are not connected to any landscape preknown to the reader. The language constructions establishing relationships between places are suffcient to get a basic understanding of the landscape described, but they are usually strongly under-specifed (Eide 2015: 120–125).

Spatial relationships can also be assumed based on the knowledge of the specifc landscape being described. In this case, the mere mentioning of two place names will establish a spatial relationship between them if they are both known to the reader, as long as 'known' implies that the locations of both places are known. No other language mechanisms than the place names are necessary in order to express such relationships. In neither case is it necessary that the referred places are real. The relationships between places in J.R.R. Tolkien's Middle Earth are known to many readers, even if there is no physical landscape to reference. However, in the case of fctitious worlds, the two places in the last example must belong to the same 'world'.

In usual reading, these two ways of establishing relationships are utilised together. This includes relational expressions such as 'three miles down the river from Prague' or 'the hills between Prague and Wrocław.' References to places can be expressed in a number of different ways in different languages, and are often mixed with other references, to places as well as to other things. In German news, the capital is often used to refer to a country. 'Moscow claims that …' is a typical reference to the government of Russia, where the capital acts as a reference to the city of Moscow, where the government is located. 'The Municipality of Oslo' can refer to a geographical area, an actor, a community, a group of people, and more. Another variable factor of establishing these relationships is the distinctions between different languages used. Indo-European languages have well-developed preposition systems for expressing spatial relationships, whereas Finno-Ugric languages usually rely on grammatical cases.

#### *7.2.2 Map Documents*

Similarly to the aforementioned reference system for written text, the way we reference places on maps includes the use of place names, which can establish links from the map to a location through pre-knowledge in the mind of the reader.5 However, the only understanding needed in order to read the precise spatial relationship between two places claimed by the map is to grasp the geometrical reference system of the space of the map. Once the scale and direction of the map is understood, each pair of places can be precisely related to each other by measuring the distance and the direction between them on the map. Furthermore, unnamed places are related to all other places in the same way. A black dot on the map can, given a certain map schema, be understood as a boulder, without any name needed in order to express the claim that this is a specifc boulder in the external landscape referred to by the map (MacEachren 2004).

The links are not just denotational, they are also connotational. The black dot on the map also links to the concept of what a boulder is (given it is known to the map reader), in addition to linking to the reader's knowledge of a specifc boulder in a real or fctitious landscape. Place names on maps are not needed in order to refer to particular places, but they help the map reader establish the connections to specifc places. A red dot can be understood as a town, and letters next to the red dot will usually establish the claim that this town has the name expressed by the letters.

Nothing can be put on a map without simultaneously making claims about how the mark on the map relates to other marks on the same map. Relationships can be fuzzy, but they are there; borders of the fuzzy area indicated can be read from the map, establishing a span of the possible direction and distance between two places based on the scale and orientation of the map.6 Even unmarked places on the map (the white areas) are related spatially to other parts of the map, in their expressing 'nothing relevant enough to be put on the map exist in this part of the space referred to by the map.' An interesting difference between maps as they are discussed here and architecture is pointed out by Miriam Vieira: maps describe/model space, whereas the role of architecture is to create space in the sense of altering or organising it (Vieira 2020: 60).

#### *7.2.3 Oral Texts*

We will now widen the scope beyond texts and maps expressed on fat documents, and proceed to discuss spoken words. Up until the spread of recording equipment in the frst half of the twentieth century, spoken words were linked to presence in time—the message could only be heard at the time it was uttered.7 The message was created, transferred, and received within milliseconds. Additional understandings of the message could follow much later, in the same way as when one only understands the full meaning of a warning when what is warned against actually takes place. In other cases, the creation of the message could be a reproduction of an expression that had existed for a long time, as in the re-performance of oral tradition.

While oral communication often included visual presence, this was not necessary in the same way as time based presence: it could be dark, foggy, or there could be physical hindrances between the communication parties (as in the case of catholic confession). Since the arrival of recording equipment, the time presence is no longer obligatory for the oral communication to carry through. We now have the capabilities to communicate not only over long distances (as with TV and radio) but even across time. This secondary orality (Ong 2002) is relevant to communication about space due to the disappearance of the present human body from the communication situation. Until around 1900, only the human body in itself was available as a combined technical medium for and producer of oral text. This meant that gestures and facial expressions were usually available in most communication situations, as was the possibility to comment and ask questions. This is no longer the case, which has led to new forms of oral communication—new oral qualifed media types (cf. Elleström 2020: 54–60). We will now move into a more detailed analysis of the consequences and infuences of this for oral communication about space.

One example of oral communication about space is the explanation of a travel route from one person to another, discussed by Vieira (2020: 71) as a mental tour. This interaction involves a series of place references within a real landscape, in the form of place names either in a strict sense or as other spoken words and/or non-verbal sounds. Oral communication is based on the technical interface of sound, where hearing is the sense of reception. This means that there is time manifested in the material interface, but no space—almost. There are cases where the sounds of the pronunciation of words represent space in an onomatopoeic sense ('the road is straight for a lo-o-o-ong time'). The musical aspects of chanting about places, where, for instance, heavily marked high-pitched tones can be used to represent mountain peaks, can describe space through auditory iconicity (Tirén 1942: 47; cf. Graff 2004). However, in the material we have worked on, these examples are not frequent, leaving symbolic representations establishing virtual spaces as the main way to represent spatiality through the sound of oral communication. Elleström (2020: 78) points out that, in the case of pop music, there is a mix between auditory text and auditory image. This mix is always there in oral texts, but the balance between the aspects varies.

If we consider the situation of face-to-face oral communication, the parties have access to a number of additional affordances. One can ask questions and request clarifcation, but one can also use gestures and other bodily movements to indicate spatial aspects, even in an indexical sense: 'We went to that mountain [gesturing the direction towards a visible mountain peak].' Furthermore, oral spatial communication is a common way of using maps (Hyttfors and Tirén 2011: 5), as the basis for a series of gestures indicating past or future travels. These gestures may also include leaving traces with a pen or a pencil (Wood 1993). In this sense, it can include traces of human bodily acts in the material interface.

Oral maps do exist as a concept, but they are quite insignifcant compared to texts and map documents. They make up neither a technical nor a qualifed medium. The concept is used mostly for descriptive oral texts. As opposed to maps as they were defned above, they do not have space manifested in the technical interface and they are approached mostly through hearing, even if some gesture can be involved when the communication happens in real time between people present at the same place and seeing each other. What is called oral maps represents a subclass of (and is therefore classifed with) oral texts, with an especially strong focus on space.

#### 7.3 Geocommunication

All the media described above can be referred to as 'maps' in ordinary language, and this use of language also stretches far into academic communication in several disciplines (Eide 2021). Elleström (2020: 56) claims that everyday language only covers a few basic media types. This is true. It is also the case that the media types that are covered are not based on clear defnitions of the terms. Furthermore, as pointed out by Simonson (2020), media borders are constantly being pushed and will change over time, due to the active engagement with and across the borders of media, in line with Elleström's description of qualifed media (2020: 54–66).

Additionally, the word 'map' is extended further to include, for instance, mental construction in human minds (cognitive maps).8 While this use of language is common also historically, it is inconvenient in scholarly work. Hence the more limited defnition of 'map' given above. It is nevertheless useful to have a wider concept for the media-based methods humans and other animals use to communicate about landscapes. In this chapter, we will follow Brodersen (2008) in using the word 'geocommunication' for this purpose.9

All the media forms described above fall into the broader concept and can be described as geocommunication. That also includes other forms such as braille text and maps, smell, sign language, and dance. These other forms, common among a number of animal species, will not be discussed any further in this chapter. Vieira (2020) shows how an equally complex space/time situation is found also in architecture, linked to the negotiations between the persons involved in practical work. All situations where the qualifed media listed above are used also have a potential for including other aspects of geocommunication.

#### 7.4 References Between Maps and Texts

When studying different media and media products, it is important to also focus on the links between them. The reference functions of maps and texts describing space do not only link media products to landscapes, real or fctional. Additionally, they link media products, internally and with other media products, through the existence of co-reference (Eide 2009). Co-reference is a basic feature of human communication. In linguistics, it is used in the resolution of anaphors: 'We went to Rome. We had a good time there.' In this example, 'Rome' and 'there' co-refer; they both have the city of Rome as their denotation. Co-reference in text is fundamental in the establishment of spatial meaning, way beyond anaphor resolution. Most texts include a series of references to locations, in which co-reference is used to establish connections between such references. In editions of texts, such as historical source material, it is common to include a gazetteer, a list of place names mentioned in the text with references to where they occur in the text as well as information about their assumed real or fctitious location. The place name occurrences grouped under one headword in a gazetteer are grouping together a set of co-referring place names.10 This can also be done when the denotation is uncertain or not known at all, or when it is assumed to be a non-existing place. The criterion is common denotation, not the nature of what is denoted.

This form for co-reference does not exist in any single map image, as one place in the referred landscape can only be represented as one visual item on the map image. Only in multi-image maps, such as printed maps with a zoom function or with a series of maps depicting the same area, does the understanding of the map include a reliance on co-reference. The co-references link locations on the main map image and the zoomed map image, or in multiple map images denoting different aspects of the same area. Co-reference is also used in rectifcation of digitised maps, where points on the scanned map are linked to points on a digitally georeferenced map. The scanned map is morphed to ft the digitally georeferenced map and is thus fxated to a standard projection. The rationale for connecting two points in rectifcation is that they refer to the same place in the space referred to by the two maps.

Connections between documents in the form of texts and maps are established by a competent reader in a comparable way. When using a map to trace the movements expressed in a travel narrative, places in the text are linked to places on the map through denotational co-reference. Place names in both are important; however, identity in names is neither a necessary nor a suffcient condition for co-reference. Relational expressions, as discussed above, can be used. Furthermore, the names have to be interpreted correctly. Someone reading a travel narrative from Canada may know that there is a much larger 'London' in England, but may still choose London, Ontario, as the place on the map she assumes 'London' in the text co-refers with. On the other hand, reading about 'Königsberg' in a book on the history of mathematics, it is of no avail to look for the identical name on most modern maps; Kaliningrad or Калининград will be the two most likely series of letters to be found referring to the same city as the one the historical name 'Königsberg' refers to.

Automatic mapping of texts, which is more popular than ever in the age of widespread digital maps, is done in a similar way. A computer programme is used to identify all place references, usually focusing on place names. Then these names are run through another programme to georefer them, that is, to fnd geographical coordinates for each of the place names. Various types of cleaning processes may be added in order to fnd the most likely candidate for several different places with the same name. These can include both digital decision-making systems and human intervention. The next step is using the coordinates to position the place names on the map, possibly with links to frequency information, the context around their occurrences in the text, and other relevant information. The quality of such systems varies depending on the nature of the source texts, and for historical sources, it is hard to reach acceptable quality without human intervention.

So where do we fnd co-reference here? The assumption in the creation and use of such tools is that the geocoding can identify a place with coordinates based on occurrences of names of the place, so that the coordinates refer to the same place in an external landscape as the names in the text. Ideally, this process is supported by additional criteria and can identify places across different names, while distinguishing between different places with the same name. Without the co-reference assumption, the process would not make any sense.

Mapping of written texts is a much more common procedure than mapping of oral texts. When working with recorded oral material, such as historical sources, the normal procedure would be to frst transcribe the oral recording into a written text, and then identify and map the place names in the transcript. There are cases, however, where oral storytelling is mapped directly. When other phenomena than the mere existence of a place reference in the oral text is to be mapped, an interactive situation is needed in order to capture the richness of the knowledge possessed by the persons being interviewed. The mapping also includes additional qualitative aspects. In the tradition of mapping indigenous historical use and occupation of land based on oral testimony, gestures and other embodied movements complement oral texts in the telling of historical spatial stories, which are then transferred to a static paper map, a so-called use-andoccupancy-map (Tobias 2009).

These maps document indigenous use of land, based on interviews with witnesses about their memories. The stories told come into being in interaction between the witness and the interviewer. In this interactive performance, time is manifested in the material interface. Parts of the information exchanged in the interaction are transferred to the map where it is recorded as a timeless trace—it is moved from time to space. Gestures make up an important part of the performance, and some of these lead to traces on the map. While being expressed on a 'timeless' document, these traces can include cues which can be used to re-establish some of the time aspects from the interviews. Combined with other pieces of evidence, the document is used to establish a coherent understanding of the often overlapping historical use of the land. This is then used as documentation in future work on land rights issues, often including legal processes.

Creating texts based on maps is not common. Classical ekphrasis, where poetry describes works of plastic art, is not common in texts with maps as sources, even if it happens in architecture (Vieira 2020). Map-based texts, however, are well known from other situations. In fction literature, especially in novels or series of novels where the site of action is a fantasy world, it is common to draw maps of the spaces before the text is written.11 While the text in those cases is not a description of the map, they are still based on the understanding of a fctitious landscape as it is expressed on the map. The map works as a geometric structure for the text—as a fgure used to organise the spatial aspects of the meaning to be expressed in the text.

Textual descriptions of maps are found in scholarly works, for instance on the history of cartography.12 Facsimiles of maps discussed in detail tend to be included. In these cases, the description is a guide for the eyes of the beholder, rather than a text meant to replace the map. Additionally, in such cases, the denotational function of the map rarely plays a signifcant role in the textual description.

#### 7.5 On the Realism of Landscapes

Until now, we have described landscapes referred to from texts or maps as either real or unreal. This is a simplifcation of how texts and maps work. While there are clear rhetorical cases of claims for existing landscapes, such as in historical texts about The Second World War, and claims for nonexisting landscapes, such as Narnia, these are only two specifc ways of referring media products to landscapes. In works of fction, the landscape described in the text is often an adjusted version of the real world. For instance, in Robinson Crusoe, the world described is the one we know, with the exception of the island where Crusoe spent his 21 years after one of the shipwrecks in the book. Map examples include Soviet propaganda maps from the cold war where places were systematically displaced (Monmonier 2018: 125–129).

Any linking of a place in fction to any map is part of an interpretative act, including the assumption that the fctitious text can meaningfully be mapped onto a real world map. But what is a fctitious text and what is not? For some people, texts describing the Apollo 11 expedition are fctitious, and the truth/fction nature of religious texts is permanently contested. While the fact that the links to real world maps is an act of interpretation is important in scholarly projects creating maps of literary works, it is questionable how relevant it is for a general reader. Reading is organised according to the principle of minimal departure (Ryan 1980) in the understanding expressed by Elleström (2018: 27): the extracommunicational domain, which is the prior knowledge of the reader, is key to establish an understanding of any work of art. As pointed out already by Ryan (1980), deviations from prior knowledge are only made when unavoidable.

When an assumed non-fction text is read as a presentation of actual history or geography, the principle of minimal departure is also applied. Deviations from what the reader knows from beforehand will be negotiated. However, whether the results of the negotiations include changes to the world view of the reader in her extra-communicational domain will be infuenced by the reader's assumptions as to whether the text is fctitious or not. If the work is assumed to be fctitious, the reader will create an adjusted world view for that work of fction and others based on it (sequels, adaptations, fan fction, etc.). The assumed world of Robinson Crusoe includes the island where he shipwrecked, but the assumed real world does not, even after reading the book. If the text is assumed to be non-fction, the change to the world view stands a much higher chance to be general to the world view of the reader, such as a new atlas correcting the locations and forms of the misplaced Soviet places from the cold war. This is a prototypical distinction which glosses over many cases not clearly located in either category. Historical fction is often assumed to have a true historical core, so that a reader might adjust her general extra-communicational domain for at least some of the adjustments she makes during reading. As the level of persuasiveness is not directly linked to the validity of the underlying claims, as pointed out by Tseng (2020), the adjustments can only be understood in relationship to the openness for the persuasions from the reader's perspective.

Further complexities are added by the fact that maps, being models, always simplify what is mapped. There are a number of systematic deviances on maps. One example is a traditional paper based 1:50,000 scale map depicting a road and a house. The road is drawn as a black line, the house as a black square, and the real world distance between the middle of the road and the middle of the house is 20 metres. Each map symbol exaggerates the size to avoid them being too small to be identifed. The road might be depicted as 1 mm wide, and the house a rectangle of 1 times 2 mm. Given the scale of the map, 1 mm represents 50 metres. Thus, if the location of the two symbols were precise they would overlap and, being the same colour, the house would normally be read as a small parking lot or some other part of the road. In such cases, not only the size of the symbols but also the locations are adjusted. The house is typically moved at least 1 mm away from the road, possibly more. Based on the fact that this is the smallest distance between two such symbols used on this series of maps, the house will be read by a competent map reader as being next to the road, without any other assumption about distance than it being less than around 100 metres.

All these types of situations, of which many more exist, show that neither maps nor texts are either true or false, and understanding the complexities in how they relate to facts and fction is dependent on the readers competence. These complexities similarly exist in new media, such as computer games and VR. What sets these apart is the way the environment is experienced in new media, which creates different situations, as we will see below. A competent reader will have well-founded assumptions about the realism of texts and maps, which will often work well. However, in some cases, they will be imprecise or even downright wrong, as when a fction book is read as if it were non-fction. Going forward, we will therefore not assume that fctitious and real maps are two different categories in real media use. We will instead reason in terms of degrees of realism in the way a landscape is depicted by maps and texts. Realism must here be taken both as intention by the author or cartographer and as testable empirical facts.

#### 7.6 Maps Are Not Texts, But Neither Are Texts

When reading a written text, listening to an oral text, or reading a map, one interacts with two different types of material interfaces using two different senses. The written text and the map are both fat documents with space manifested in the material interface, whereas the oral text is made up by vibrations in the air with time manifested in the material interface. The latter is often complemented by a human body using arms, fngers, and other body parts, as well as external objects, to indicate directions and make references. These three are also often combined, in either pairs or all three together. This can also be the case with other media types such as VR, as we will see below. Humans possess a wide area of mixed geocommunication methods. In this section, however, we will see each of these analytically isolated as a specifc media type with its specifc modalities.

What our eyes see and what our ears hear lead to meaning. This transfer process has been studied extensively over millennia. In the following description, the basis is Elleström's four media modalities (2020). These modalities do not indicate any essential division (the material interface has no clear border to the spatiotemporal modality); the distinctions are analytical. Furthermore, there is no obligatory development in time from one modality to the other. We do not frst meet a material interface, then use our senses, then go through spatiotemporality so that we, in the end, can apply semiotics. The different parts of the process are all mixed together.

As one may expect for media with a special link to landscapes: the spatiotemporal modality offers a certain amount of complexity when we study maps, written texts, and oral text (Elleström 2018: 19). What is established as senses interacting with physical material (in our cases sight and documents or hearing and sound waves in the air) goes through the spatiotemporal tunnel into a world of meaning. Here the signs seen and heard make sense in interaction with the pre-knowledge of the specifc human mind.

In the sensorial and material modalities, the two types of spatial documents—written texts and maps—are similar to each other, but quite different from oral text, which is one separate track of sounds in the air. In the semiotic modality, on the other hand, the map with its geometrically based spatial reference system is different from oral and written text, where we see a language-based spatial reference system. Written text loses its closeness to maps and gains a new closeness to oral text. This is not to deny virtual time in maps and virtual space in texts. However, these are established at another level, based on the meaning of each expression. The space of the map and the time of the text is a primary level in the semiotic process of making meaning in the context of landscapes and embodied human experience.

This implies that transmediation (Elleström 2020: 81–83) between oral and written texts, on the one hand, and maps, on the other, is slowed down by the differences between how the two media forms refer to external landscapes in their representational systems. Between oral and written texts, so different in their material interface and sense interaction, transmediation can be done with much less effort at the semiotic level. When it comes to integrating different media products in the same material interface, on the other hand, written text and maps integrate so well that it looks almost seamless, when maps illustrate textual documents as well as when texts add information to a perimap. Integrating oral text with either maps or written texts was much harder and drew more attention to the hybridity of the resulting media product, at least until the introduction of computer carried recorded sound.

Transmediation between both is complex and will often be based on going back to the original story. This was clearly expressed for plastic arts and poetry already in 1766, when Lessing clarifed the background for his suggestion that painting should be based on quite different principles from those underlying poetry: painting is about objects in space, and poetry about events in time (Lessing 1893 [1766]). The relationship between map/text studies and interart studies generally and Lessing specifcally is discussed further in Eide (2015), also indicating which of Lessing's observations are still relevant and which are based on dated and now irrelevant aesthetic principles.

#### 7.7 Interpretative Journeys

There is an important additional aspect of spatial reasoning as it is expressed in texts and maps. The spaces we experience in landscapes, and are reminded of in media products, also form basic aspects of human thinking outside the actual navigation of landscapes. Two examples from classical rhetorics are the art of memory and the linearisation problem. We will also look into as if-travel in physics.

The art of memory traces its roots back to Simonides (see Yates 1966: 223) as a memory technique used in the preparation of speeches. Of special interest to us here is that its operational mode consists of linking a landscape to a textual narrative, in this case oral. It is not about remembering every word exactly,13 but rather about remembering the right order of the topics to be covered in the speech. One assumes, for instance, a garden and a walk from place to place. At each place, the speaker mentally stores objects reminding her of the upcoming topic. This has been shown to work as a memory technique across cultures and is still part of commercially available mind training packages.

The starting point for the art of memory technique is an already created speech, and the spatial confguration is used to remember the main points in this speech. Thus, an oral or written text is transferred into a mental space, which is remembered and later used to re-create the original speech. Spatial memory is used as a mental storage technique.

The linearisation problem is also known from classical rhetorics. The problem here is to create an oral or written text in the frst place. How can one establish a linear text from a number of different topics which all need to be included? The solution here is to create a space where the topics are spread out, and then navigate this space like a walk. This walk is assumed to be linear in the same way as a speech (Levelt 1981) and it is important that the route passes through all of the topoi,14 that is, all the topics. In this case, it is not about remembering a text but creating it, out of a mass of relevant topics.

In the development of thinking towards scientifc hypotheses and eventually theories that can be formalised in propositional language, active use of drawings is an important part of the building of mental models (Nersessian 2008). This has been documented as video recordings during social interaction among groups of physicians at different career levels (Ochs et al. 1994). The video footage shows a fgure being drawn on the blackboard by one participant, then others adding their own marks to the drawing. These can either be unclear lines created by moving a fnger over the blackboard drawing, or additional drawings with chalk. Furthermore, in the model space used to discuss particle phenomena, movement is described. Not just particles moving, but particles and other parts of nature are even addressed with words such as 'me' and 'you.' Thus, in the process of making sense of physical reality through the models used to talk about it, metaphorical travels through the model space—and presumably indirectly through micro-space—is done by humans in the room where the discussion takes place. This happens both through oral language including gestures—and with the use of fgures.

All these examples show the fundamental power and potential for meaning-making in thinking about space in a fgurative way. This happens either as a space standing in for another space, however different it may be, or as a space standing in for a more abstract structure. Meaning is then expressed through stories about travelling through that space. This shows how the differences between fgures, such as maps, on the one hand, and texts, oral and spoken, on the other, can be used creatively as a tool in developing, expressing, and negotiating meaning. As Simonson (2020), we see also here how media borders are pushed. Model-based thinking is fundamentally tied to the complexities in the relationship between texts and maps, as it passes the tunnel from senses and physical interfaces through space and time, real and represented, to a semiotics-based establishment of meaning.

#### 7.8 Modelling and Media Transformations

The previous examples showed how visual forms are used in individual and social negotiations of scientifc meaning in modelling processes. Indeed, what has emerged over the last years in digital humanities is a model concept which is in line with the concept of 'media product' in Elleström (2018: 11): "Since being a media product should be understood as a function rather than an essential property, virtually any material existence can be used as one, including not only solid objects but all kinds of physical phenomena that can be perceived by the human senses." The functional aspect of 'model' is made explicit through the focus on the verbalised form 'modelling' rather than on the noun 'model'. A model defnes anything which is used by someone (a person, a group) in modelling. The physical item is important, but cannot determine if the process is a process of modelling, in line with the criteria for something being a technical medium of display (Elleström 2020: 33–40).

A map is a media product insofar it is understood as a human-made expression intended to convey some sort of meaning to other humans.15 This leads to a question archaeologists struggle with, not only for maps but also for other forms of expression. How can one decide if lines on the wall of a cave are parts of a media product or not? The frst attempt to clarify the question would be to identify the species behind the marks. If the marks are made by the claws of an animal they will not be seen as establishing a media product. But even if they are assumed to be human made, they might be the results of sharpening a tool, and not intended to represent rivers and hills. If it is decided that the marks are made by a human and do indeed represent a landscape, then it is seen as a media product, and commonly described as a map. This map is a model of the landscape.

This also applies to scholarly and scientifc modelling—the same scratches could make up a model also in this sense. Imagine, for instance, two scientists in the late nineteenth century trying to understand the hydrological structures in the landscape around. Seeking refuge from the hot sun in the cave, they start discussing different hypotheses and eventually, as they have left their writing utensils at the camp, use the wall and sharp stones to express different models, in line with the example from physics above. Scholarly and scientifc modelling is an activity defned by who takes part in the activity and with which purposes. Thus, a media product is a model insofar as it is used in modelling, and it is a scientifc or scholarly model insofar as it is used in scientifc or scholarly modelling.

The maps therefore represent models when they are used as models of a landscape. This happens in most intended use of maps, but not all. Maps are also decorative objects which can be used for other purposes than modelling. The modelling aspects are based on their representational function.

But what is a map a model of? This question involves some additional complexities. From the outset, the map is a model of a landscape with some degree of realism. Some maps, however, are clearly presented as maps of a fctitious world, and are assumed to be so by most readers. A map of Tolkien's world is assumed to represent the same world as in the books written by Tolkien, and maybe also as in the flms, the fan fction, and other derived works. Such maps will be assumed to be more or less correct. Thus, there exists something that the truthfulness of a map of a fctitious landscape can be tested against. This is a shared conceptualisation between the users of the text, testable against documents describing this world—in the case of Tolkien's cosmos, frst and foremost his own work. A fctitious world can have an assumed internal truth system, following some sort of spatial principles. It does not have to be coherent,16 but it is assumed to be truthful within the context of the fctitious work. The establishment of the intracommunicational domain (Elleström 2018) follows certain rules.

A virtual world, as it is known through VR systems, adds another level to the experience of space. Independently from whether that virtual world is mediating a fctitious space or not, it does not only have to follow some sort of spatial principles, but also the principles of physical sensation made by experiencing the known space. There is a specifc type of embodiment present in this experience. As pointed out above: from the outset, the map is a model of a landscape with some degree of realism. VR might, at frst glance, appear to present a model of space with a higher degree of realism, but what happens if a virtual space is not tested against the real physically experienced world, but rather against another media product? What if the virtual world is established through a media transformation (Elleström 2020: 79–83) from, for instance, a text to a VR system?

Going forward, we will focus on space as it is expressed and experienced in VR. On the one hand, space is crucial or even existential for media products which are experienced as VR. On the other hand, immersion is often seen as an inherent part of VR. Through analysing experimental media transformations, we will show that a transmission of the perception of space—which is as realistic as possible—only creates an illusion of a multi-dimensional space. This experience is therefore more connected to the traditional ways of expressing knowledge about landscapes (as it was described above) than one might expect. The linking between a media product that is conceptually creating space and the connection to be experienced (also through traditionally mediated space as it was outlined above) is widening the discussion in new ways.

#### 7.9 Landscape, Space, Reality, and the Virtual

We will now extend the discussion to three-dimensional modelling as a process of creating virtual worlds and accordingly VR systems. In such systems, spaces are expressed by computer screens, mobile phones, head mounted displays, or other technical media, enabling a virtual world. The way spatiality is expressed in these systems makes the representation of spatial relationships and landscapes, also morphing landscapes, accessible in ways not available through other technical media. Investigating VR as a medium for conveying experiences of landscapes opens up new perspectives on mediated space and time. Furthermore, examining how scholarly and scientifc models are expressed in multi-dimensional space allows analysing the spatial understanding, not only of the creator but also of the user. This has signifcant impact on how models can be used in scholarly and scientifc work.

The purpose of creating a 3D model is dependent on its intended use, but also connected to the medium to be used to visualise the model. Displaying 3D models through VR devices and thus representing an interactive and explorable three-dimensional computer generated environment (a virtual world) is usually associated with expectations deriving from a general conceptual understanding of VR as a specifc type of reality emulation. It is more challenging to establish a conceptual defnition of VR. It must be based on an understanding of the complexity of immersion, in line with Virginás' discussion of the video quality of reproduced recordings in Lynch's *Lost Highway* (2020: 160–161).

This complexity is linked to the special nature of embodiment in the use of VR equipment, and links to performance studies (Vieira 2020: 8, 10). Maybe the experience of travelling can be extended to the context of architecture, beyond the possibility to see and move in non-existing buildings. This begs the question, however, of the vital role of embodiment in architecture and how far this can be re-created in VR. Can, in fact, other media forms experienced in VR, for instance, theatre, be seen as experiences of the represented media forms, given the differences in embodiment (Elleström 2020: 81–83)?

Experiencing a virtual world is conceptually connected to the idea of being immersed within an artifcial environment. It implies the possibility of strongly realistic user interaction with the virtual surroundings simulating an experience, which gives the user an impression of being part of an illusionary perfect other reality. This other reality can mediate, for example, past events and distant places, but also entirely different, unprecedented worlds. However, at this level, it is questionable how new VR is. The idea of realistically modelling an immersive representation is not only expressed in current development and implementations in VR but also expressed much earlier in paintings, optical illusions, and other forms of pre-digital media products.

The dream of VR as it arises from the conceptual model presented above has never been fulflled completely or even satisfactory for most users. There are limitations both technically and in terms of which graphical representations are possible. In order to feign a fully realistic experience of the unrealistic, a VR system would have to trigger all the sense and perception systems of the user. Existing VR implementations and devices nevertheless have a potential for representation which can be unlocked without insisting on a specifc level of performance in the VR system.

Setting aside the conceptual expectations and considering existing implementation of VR within the technical possibilities available today, one can explore in practice the actual possibilities for representation. VR can be used for modelling highly realistic yet completely different realities—completely different in the sense of different to any known place but still realised through representations of known objects and known physical characteristics. One example is SpaceVR, a project that attempts to make the universe explorable through VR. Such attempts apply analogously to many unattainable spaces in the world known to us. Accordingly, the attempt to make known, but physically inaccessible places explorable through VR is a frequently observable phenomenon.17 Photorealism is one important goal in VR, but other ways of representation are also important, including highly abstract forms; there are, for instance, several implementations in VR representing paintings by Hieronymus Bosch.

Vieira (2020) established the role of the architect as a sort of a theatrical producer. Also in the analysis of VR, it is highly useful to include practical experiences from the creation of VR systems. This balances the focus between conceptual ideas and possibilities on the one hand and the actual possibilities of implementation on the other. While it is important to plan VR implementations within the framework of existing technical possibilities, the creation of three-dimensional worlds with the help of computers additionally refects the perception of known reality. As pointed out for other media above, the users will assume similarities with their known environments and only change their expectations when they have to.

For the experience of VR, a virtual environment must be created. This environment consists entirely of three-dimensional objects. None of the visual characteristics from our known reality is self-evident. Everything, including refecting light distinguishing between different materials (for instance, wood or plastic), shadows, and the appearance of highlights in white, is calculated by algorithms and can be infuenced by the creator of the virtual world. Only what the user can see needs to be calculated, and when something can only be seen at a distance, it can appear realistic even if in lower quality. Consequently, there is no need to adapt the representation to the user's world of experience, at least not when considering the possibilities for implementation. As in theatre and flm sets, the world can be set up by invented backdrops. The artistic process of creation, in this case the modelling of virtual surroundings, always has artifcial aspects which open up for new interpretations.

The ability to experience space and to move in the virtual world is usually, but not always, linked to different user interactions. Ultimately, almost every decision in the modelling process can be made by the modeller. Metaphors and experiences from known reality are often used; however, the way we orient ourselves can only partially be transferred. Orientation in an unknown space is always of key importance and will be of immediate concern to a new user. As we saw above, everybody has the experience of being in an environment, even if the experienced environments vary signifcantly. The creation of VR systems is connected to a desire to share experiences, but also to create a world which cannot be experienced in reality based on the experiences the user already has. Such worlds can still be expressed through mediation, and, as pointed out by Simonson (2020), this also happens in other media than VR. The experience of movement in a virtual environment establishes a direct reference to a reality understandable by humans. The ability for abstraction we saw through maps and text has its counterparts which can be analysed in the implementation of a VR.

A commonly used case for VR is games, which are similar to video games known from other technical media. Even VR applications not meant to be games often use interfaces and elements known from video games. In the context of games, users interact with a so-called *cyberspace*, a completely virtual surrounding in which gamifcation aspects are not essential. The experiences of how space is represented in video games are used in a comparable way in VR. Maps are often important elements in this context, being used to discover the virtual world. The abstractions used in these maps are similar to the maps described above. The same language is used. Modelling a virtual world is often combined with designing maps, which is of essential importance to user interaction, communicating the space to the user. Besides this communication, characters can also talk to each other about places that are not covered by maps.

The use of both maps and texts in video games is well established for communicating information about the spatial structure of the virtual world, and for telling stories to the user.18 Maps mostly function as navigation tools: they show where you are and sometimes they also help users navigate, by showing them where they can go. Pathfnding and navigation are often parts of the users own decisions in the game. The map covers all places that are available, implying also that some places are not of interest or not reachable. One can design virtual worlds where the user can get lost. The double nature of maps as tools for overview and control and as tools for navigation (Wood 2010) is often present in the use of game maps.

Places visualised on a map establish the expectation that they will in fact be reachable. The appearance of a map often fts the design of the virtual world it is describing. The places covered by the map are only references and cannot in themselves establish a story around the places being referred to, or cover their meaning. Compared to descriptions of places in texts, the reference will be less likely to create emotions connected to the place (Eide 2015: 152–153). The experience of actually visiting the virtual place can overcome this.

Video games and maps have a quite special relationship, as shown by an endless number of examples from the last decades. Similar to board games, maps can become a central element of the game also through functions that go beyond pure orientation.19 As a result, positions of players on the map are elementary in the sense that each location and spatial relationship on the map is important. At the same time, the respective positions must be visible and perceivable either by an opposing team or by other players. A live mapping of all objects of interest is possible in a virtual environment, where also moving objects such as cars can be shown. The virtual world and its functions are established through maps. The map can function as an interface, independent from the user. Sometimes the map itself is responsible for offering challenges.

Groups of media computer science students at the University of Cologne have examined modelling processes of media transformations, partly through VR implementations and partly through attempts to model maps in VR (Eide et al. 2020). Analysing the results of the students' work allowed us to make some observations of the modalities of VR applications. Given that specifc media products in VR differ from each other, and the rapid technological development, the results are bound to be preliminary.

We identifed three general questions that pinpoint specifc decisions made through the modelling. The questions and the students' answers are shown in Figs. 7.1, 7.2, and 7.3. Our observations for VR as a non-bodily media product can be summarised in the context of media modalities (Elleström 2020). The material modality as a latent corporeal interface of the medium, and how its different modes are expressed, depend partly on which type of VR device is used. The bodily movements of a user can in some cases be fully recognised through motion detection. The movement of the head is essential to control the point of view. The materiality of a controller, gloves, and a VR headset is normally established as demarcated. Less clearly demarcated material manifestations are expressed by artifcial sounds and lights added to the virtual world. The perception of the interface of VR happens through the different senses, as modes in the sensorial

**Fig. 7.1** How do you point your users to 'interesting' aspects of your VR-environment? Do you explicitly point your users to aspects relevant for your narration?

modality. Feeling, tasting, and smelling are usually not included; however, seeing and hearing are attended through the VR headset with earphones.

Space manifested in the material interface can be implemented through a map. Cognitive space is explorable through looking around in the virtual world. Virtual space is present in the virtual world. Time manifested in the material interface can be infuenced by interaction to the extent it is implemented in the system. Perceptual time is established through interaction or animated camera movement. Virtual time is given through animations.

VR uses a complex and many-faceted semiotic language. The basis for the photorealistic expressions one sees is iconic. In the image, however, there is always a set of symbolic elements. These can be graphical as well

**Fig. 7.2** Do the objects in your VR-environment present your users a functional (symbolic) or a realistic view?

as in the form of text, or as texts used as elements of interaction. Some parts of VR are close to map semiotics (MacEachren 2004; Brodersen 2008), whereas other aspects trigger the indexical complexity of digital photography and video (Lister 2007). Being highly dependent on concrete implementations, it is hard at this stage of our research to generalise the semiotics of VR. Further research is needed to establish how different the various technical implementations used for VR are semiotically, and whether it is possible to establish a general core understanding of VR semiotics.

**Fig. 7.3** Would you say your VR-environment is more a game or a narration?

#### 7.10 Being in <sup>a</sup> Virtual World, Experiencing Space Real and Unreal

By putting on a theatre mask, the actress transforms into the role character.20 In other theatre forms, the masks are invisible, but still metaphorically present. In all embodied, actor-based art forms, the actor meets the role character in a shared body, whose trajectory is both that of the role character and of the actor, as discussed also by Simonson (2020). In the 1930s, the mask of a gangster was gradually glued to the face of Humphrey Bogart, opening up for a new type of hero when the actor who had impersonated several criminals became the doubtful but inherently good guy in flm noir of the 1940s.

When putting on a VR mask (a head mounted display), only the eyes are fully covered. This is necessary in order for the targets of light being emitted from two small screens, creating an illusion of a three-dimensional room. The earphones provide some replica of surrounding sound, while the sound of the physical space around the user of the VR system can be heard as distant, sometimes even unreal or dreamlike, dependent on the specifc equipment used. This might be compared to the architecture of the theatre building (Crossley 2020: 96–97). Clarifying the relationship between the consequences of the different types of equipment and settings is beyond the scope of this chapter, but Crossley's discussion of theatre as a hypermedium could be extended by also looking at the theatrical text as basis for a VR application, in addition to VR within a theatrical event.

As Alice experienced behind the looking glass, everything is the same yet everything might be twisted. In the end, the room on the other side turned out to be merely a dream. This is in line with literary functions, such as metalepsis, as pointed out by Lutas (2020), also for its counterpart in flm. The idea of the computer game as life (Gaskill 2009) is a mirage, no more real than the dream of jazz music simplifed to one single tone (Skagen 1993). Yet, it is a powerful argument ad absurdum, pointing towards the science fction nightmare of the virtual that can kill you, in our century unifed through the popularity of the Matrix trilogy. What can kill you is real and not just virtual.

What we see in our VR worlds is both real and virtual. The lights and sounds are real. The 3D and surround effects are virtual. The avatars we meet can represent artifcial intelligences on which our acts have no impact. They can also represent our best friend or partner, in which case mistakes done in the VR system can alter or even destroy lives. Like any other technical medium, from parchment and paper to Instagram and Twitter, it can be used to send messages to the ones who do not care. It can be used for fully fctitious games, where all is virtual and nothing has consequences. It can also be used to communicate with others who rely on our words, images, and acts, in the realness of letters and images establishing or breaking family bonds and starting or ending wars. The novelty of VR is not the fact that it is virtual. The virtual has always had a potential for linking to reality. The novelty of VR is its way to relate senses to a mix of real and fctitious environments. It hides the realness in a secluded space by being in a media artefact, and hides its virtualness though photorealistic immersion.

VR is not harmless. Books are not harmless. Whether we say that books kill, or we say that books are but tools used by humans in order to initiate acts that lead to killings, is a matter of where we stand. Nevertheless, what we say about books we can also say about VR. Behind all the differences between media types—behind their different ways of representing something beyond themselves and behind the differences in time and space and mixes of them manifested in the material interface—they are all tools used to make the unreal real and the real unreal. The various media types are used to communicate across the abyss between human minds. As shown in Elleström (2020), the basis for all the discussions in this volume is communication.

In spite of all the differences presented in some detail in this chapter, all these media forms can represent landscapes and events taking place there. None of them are the landscapes and events they represent. The differences between their own landscape and soundscape are what establish them as different media, qualifed and technical. The links to the landscape come about based on our knowledge of landscapes. The only thing the media can do is to remind us of the landscapes we know, reshuffing elements in them. The rest is up to us.

#### 7.11 Conclusion

This chapter focuses on how one can see the landscape through textual and graphical media products. This is important in the context of the publication *Beyond Media Borders: Intermedial Relations among Multimodal Media* because the landscape itself and spatial thinking is a basis for human thinking and communication (Tversky 2019). The recipient's embodied memories of experiences in an environment (real or virtual) are always connected to the decoding and understanding of spatiotemporal modes which are active in all media products.

We have analysed different media expressions related to landscape, with a focus on spatial representations in different forms and on how transformations can deepen our understanding. We started with written and oral texts and maps, and their combination in geocommunication systems, which are traditional ways of expressing knowledge about landscapes. By studying them as different systems for communicating spatial understanding, we clarifed how each system works. The sequentiality of textual documents is expressed in a spatial form, but is still recognisable based on the code of writing systems, which is different from the other visual media we study.

Interestingly, the spatial relationship between two places is established differently in texts—oral as well as written—compared to map documents. By focusing on place names as reference system, the different forms of spatial communication were highlighted: the geometrical reference system of the space of the map versus the primary syntactic relationships between words within a sentence. Opening up the discussion to spoken words also highlighted the importance of the human body as part of communication. Different media products are moreover concretely linked. The reference functions of maps and texts describing space do not just link media products to landscapes, real or fctional; they also link media products, internally or with other media products, through co-reference.

Mental space, mental models, movement, and memories are basic building blocks in journeys, physical and mediated, with different levels of realism. Scientifc modelling and fction are different, yet connected (Suarez 2009). Thus, we were able to expand our research and discuss in a more abstract way how all this relates to modelling and modelling processes in general. By applying the concept of modelling from digital humanities and showing the connections with Elleström's concept of media product (2020), we could see that both are functions defned by the purposes of those who take part in the modelling and communicative activities. Thus, a media product is a model insofar as it is used in modelling, and it is a scientifc or scholarly model insofar as it is used in scientifc or scholarly modelling.

This analysis was then used to study concrete types of spatial expressions through new media types such as VR applications. Being new media, in a historical sense, means that they are less affected by conventions and have a much shorter tradition of developing their ways of using the available affordances. Indeed, the affordances are still in formation and the technical development changes the affordances themselves. Moving beyond graphical two-dimensional expressions and written texts describing landscapes into virtual three-dimensional 'landscapes' enables us to understand better the spatial media characteristics infuencing media products. An abstract versatility results from the existential basic property of VR spaces as they are used to create other spaces for users being in a virtual world, experiencing space real and unreal. This refects on how space can be mediated and formed through experiences, memories, and illusions. Thus, landscapes in a number of different senses were discussed. We started with the real environment that we move around in as embodies creatures and how it has been mediated over centuries and millennia of human culture, before shifting our attention to the landscapes we move around in through avatars, that is, VR.

Our claim is that the relationship between the spaces in a virtual world and the spaces we know as embodied animals is complex and many-faceted, establishing VR both as mediated expressions (VR is a type of communication device) and as experienced spaces. This is linked to an equally complex set of connections between different forms of embodiment and different sets of affordances. We need more research and more practice in this area to be able to clarify how all these parts interact in order to establish human mediated experiences in communication.

#### Notes


bard in fxating passages, which were then put together in series suitable for the performance (Havelock 1986).


#### References


———. 2016. Sand in the Mapmaking Machinery: The Role of Media Differences. *Livingmaps Review* 1 (1): 1–13.

———. 2021. *Where is the map? In Mapping the Unmappable? Cartographic Explorations with Indigenous Peoples in Africa*, ed. Ute Dieckmann. Bielefeld: transcript Verlag.


———. 2010. *Rethinking the Power of Maps*. New York: Guilford Press.

Yates, Frances A. 1966. *The Art of Memory*. London: Routledge and Kegan Paul.

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/ by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## The Model Applied

## Summary and Elaborations

*Lars Elleström*

#### **Contents**


#### 8.1 Summary

In "The modalities of media II: An expanded model for understanding intermedial relations", I proposed an extended and developed model—or, rather, a conglomerate of models—designed for systematic investigation and analysis of the basic features of all forms of media and their interrelations (Elleström 2020a). Instead of beginning with a limited set of established media types and their traits and interrelations, the model is founded on a delineation of the concept of media product and an explanation of media modalities—types of basic media traits—that are shared by all media

L. Elleström (\*)

Department of Film & Literature, Linnaeus University, Växjö, Sweden e-mail: lars.ellestrom@lnu.se

<sup>©</sup> The Author(s) 2021 213

L. Elleström (ed.), *Beyond Media Borders, Volume 2*, https://doi.org/10.1007/978-3-030-49683-8\_8

products and therefore also media types. This allows the model to account for the crucial fact that media products and media types are both similar and different.

I also emphasised that the point of models is to put aside specifc details to enable a view that is more generally valid. I want to offer a broadly applicable, well-developed and distinct but fexible theoretical framework. Although the model encourages methodical analysis of media and their interrelations, it does not offer a proper methodology. It suggests ways to tackle things but does not imply any fxed methods for investigation. Nevertheless, it is hopefully helpful for developing such methods and methodologies that must ultimately be formed based on one's specifc aims and goals and in the service of various research problems and questions attaching to mediality at large and, more specifcally, media interrelations.

This was demonstrated in how the model was applied in the contributions to the two volumes of *Beyond Media Borders: Intermedial Relations among Multimodal Media*. The chapters, written by scholars from various disciplines, tackled a broad range of basic and more or less established qualifed media types, such as theatre, architecture, flm (with or without embedded sound), live and online performances, audiobooks, written literature, novels, book illustrations, posters, music, opera, comics, graphic novels, journalism, biographies, scientifc articles and maps. The chapters also dealt with various sorts of moving pictures, a variety of radio and television programmes, and different forms of oral and written verbal language.

Furthermore, the model was combined with a variety of complementary theories and methods selected for dealing with the more specifc issues of the chapters. Likewise, the chapter authors used it in the service of a range of different research questions, aims and goals. Mark Crossley (2020) investigated the challenging relationships between theatre and other qualifed media types. Andy Lavender (2020) scrutinised how the work of actors and performers can be comprehended in terms of multimodality. Andrea Virginás (2020) analysed the representation of electronic screens in feature flm, which has recently become common in flm culture. Chiao-I Tseng (2020) looked into the issue of story truthfulness and affective intensity in feature flms in the light of how they use historical archive footage, found footage and various flm techniques. From a context and user perspective, Iben Have and Birgitte Stougaard Pedersen (2020) researched how audiobooks are experienced and what it means to 'read' an audiobook. Focussing on mobile language learning, Heather Lotherington (2020) explored multimodal encoding of contemporary digital mobile communication. Mary Simonson (2020) investigated historical and contemporary media products that communicate less straightforwardly, partly because of their intermedial confgurations. Kate Newell (2020) examined different forms of movement of pictures across media platforms and contexts, and the mechanisms that facilitate such movement. Based on thorough considerations of the modality modes of architecture, Miriam Vieira (2020) analysed the presence of architecture in literature. Ana Munari Domingos and José Rodrigues Cardoso (2020) scrutinised the construction of 'reality clues' in journalism comics and biography comics. The aim of Jørgen Bruhn (2020) was to combine the ideas of ecocriticism with the analytical strength of intermedial concepts. Liviu Lutas (2020) scrutinised the broad transmediality of metalepsis, the paradoxical transgression of boundaries between logically distinct represented worlds. Finally, Øyvind Eide and Zoe Schubert (2020) explained the various mechanisms that connect different media types representing landscapes.

Whereas the authors used different parts and aspects of my model in their chapters, they also employed some central concepts that, in different ways, attach to the model in important ways without being properly developed in "The modalities of media II" (Elleström 2020a). This is simply because doing so would lead too far and disrupt the structure of the already voluminous chapter; there are several other issues that would also deserve more extensive treatment. To slightly compensate for this, and hence further clarify the bonds between my theoretical framework, other broadly used concepts and the many chapters in *Beyond Media Borders*, I will elaborate here on three of the concepts that were neglected in the opening chapter, although in other publications I have integrated them in the framework of my model. The frst concept is adaptation, which several of my colleagues have discussed in their chapters. The second and third concepts are narration and language, which were present in most of the contributions to *Beyond Media Borders* in one way or another. Because of this omnipresence, I will concentrate on those two concepts. What follows here, therefore, is not at all a summary of the contributions to *Beyond Media Borders*; it is rather an enlargement of some of its conceptual nodes.

#### 8.2 Adaptation

In the context of humanities and mediality scholarship, adaptation is generally understood as a phenomenon involving different media types (typically theatre or written literature and flm). Therefore, I have argued that adaptation should be seen as a form of transmediation (Elleström 2013, 2017). I also briefy explored transmediation in "The modalities of media II" (Elleström 2020a: section on "Media transformation"). As stated there and elsewhere, I use the term 'mediate' to describe the process of a technical medium of display that realises presemiotic (potentially meaningful) sensory confgurations. For instance, a piece of paper is able to mediate visual sensory confgurations that are (once perceived and rudimentarily brought into semiosis) taken to be a food recipe, a bar chart, a scientifc article or a musical score. If equivalent sensory confgurations (sensory confgurations that have the capacity to trigger corresponding representations) are mediated for a second (or third or fourth) time and by another type of technical medium of display, they are *transmediated*. In our minds, some of the perceived media characteristics of the target medium are, in important ways, the same as those of the source medium; they form a recognisable virtual sphere. This enables us to think that the musical score that is seen on the paper is later heard when it is transmediated by the sounds of instruments. In other words, the score's vital characteristics are *represented again* by a new type of sensory confguration (not visual but auditory signs) mediated by another type of technical medium (not a piece of paper emitting photons but sound waves generated by musical instruments) (Elleström 2014: 20–27).

Just as there are many different media types and communicative situations, there is a plenitude of transmediation varieties. Although this has the potential to make the study of transmediations an exceedingly broad feld, research has only paid full attention and given names to a few forms of transmediation. In the study of art forms, the general term for transmediation of media products to other media products is 'adaptation'. However, for good reason, researchers such as Kamilla Elliott have argued that, in effect, adaptation studies must consider not only transfer of represented media characteristics between single media products, but also among media types in general (Elliott 2003: 113–132). This is because one cannot normally reduce vital media interrelations to the connection between only one source media product and one target media product.

Nevertheless, one does not even tend to call all types of transmediation of specifc media products 'adaptations'. For instance, one seldom refers to transmediations from written, visual and symbolic (verbal) media products to oral, auditory and symbolic (verbal) media products—that is to say, the reading aloud of texts, or the other way around—as adaptation (however, see Groensteen 1989). The same goes for transmediations from nontemporal to temporal visual and iconic media products—from still to moving images (as discussed by Dalle Vacche 1996)—and for transmediations from written, visual and symbolic (verbal) media products to oral, auditory, iconic and symbolic media products; that is, the 'setting' of text to music (Lin and Chiang 2016). We also generally exclude transmediations of media types such as libretti, scores and scripts from the domain of adaptation.

Instead, the archetypical adaptation is a novel-to-flm transmediation. However, few scholars have delimited the concept of adaptation exclusively for this specifc type of transfer. Adaptation studies today frequently work with not only theatre, literature and flm but also qualifed media types such as computer games, opera, comics and graphic novels. This is the case in, for instance, Linda Hutcheon's *A Theory of Adaptation* (2006). The authors in *Beyond Media Borders* who write about adaptation attach to the general idea of transmediation and focus on adaptation from novel to feature flm (Lutas 2020), from verbal biography to comics (Domingos and Cardoso 2020), from opera to flm (Simonson 2020) and from novel to illustrated and graphic novels, television and radio programmes, ballet, opera, theatre and flm (Newell 2020). However, Hutcheon and the authors in this volume only address media with developed narration, which is a common way of implicitly delimiting the notion of adaptation.

#### 8.3 Narration

As stated (Elleström 2020a: section on "Heteromediality and transmediality"), transmediality means that media products and media types may (or actually do) communicate equivalent things; they can, to some extent, represent the same or similar objects (in Peirce's sense of the notion). This implies that there may be transfers in time among media. Because narration is transmedial in the strong intermedial sense that it transgresses basic as well as qualifed media borders, many narratives can be transmediated: their characteristics may be represented again by different media types and, despite the transfer, be perceived to be virtually the same. For many adaptation scholars, the transmediation of narratives is indeed the central concern. However, one does not have to refer to one's work as adaptation studies in order to examine transmediation of narratives. Transmediation of narratives is exceedingly common, not only in everyday communication but also in more complex and offcial systems of communication such as education, research and legal processes. It also fourishes in religion, art and entertainment.

I have developed my concept of transmedial narration (Elleström 2019) from Marie-Laure Ryan's work on transmedial narratology (Ryan 2005), but framed it within a broader concept of communication and attached it to the model of media modalities. Circumscribing this, one must frst note that narration does not exist by itself; it happens when we communicate with each other. Consequently, narratives and stories are not something that we fnd foating around independently, but something that minds communicate. Therefore, one can and should compare narration with other forms of communication. Narration is a communicative form that is specifc and important enough to deserve special attention. At the same time, it is only a variation of, and in practice sometimes not at all clearly delimited from, producing virtual spheres in general.

Nevertheless, I have proposed to defne a narrative as a virtual sphere, emerging in communication, containing events that are temporally related to each other in a meaningful way. Thus, the core of a narrative is exactly this: represented events that are temporally interrelated in a meaningful way. As this core consists of several elements, one could also describe it as a scaffold. I have also suggested that a whole virtual sphere containing such a core and normally also other media characteristics should be called a narrative and that the scaffolding core should be called a story. Thus, one can simply describe narration as the communication of narratives.

It follows from this that what one perceives to be the same story may be realised in dissimilar settings in different narratives. What one recognises as the same story can be narrated in different ways. For anyone acquainted with areas such as literature and flm narratology, this conclusion does not come as a surprise. However, scholars have debated the nature of the sameness of stories. For transmedial research, the crucial point is that it is possible, common and often useful to perceive that vital core constituents of some narratives—certain events being temporally related in certain ways—are more or less similar to vital core constituents of other narratives, possibly represented by other media types. This is scrutinised particularly by the authors in Volume 2, Part I of *Beyond Media Borders*, "Media Transformation" (Simonson 2020; Newell 2020; Vieira 2020; Domingos and Cardoso 2020; Bruhn 2020; Lutas 2020; Eide and Schubert 2020).

Given these conditions for transmedial narration, one must also emphasise that stories may either be construed for the frst time by the perceivers of media products (because of salient structures emerging as the narratives develop in the mind) or recognised (from earlier encounters with narratives or events in the world). In other words, the story may be based either mainly on intracommunicational objects arising in the virtual sphere, or on extracommunicational objects in the form of already known stories or perceived events. In any case, stories have no autonomous existence despite what one might be led to believe by certain narratological discussions. Stories are always results of some sort of interpretation performed by certain people in particular communicative circumstances—never objective existences, but possibly intersubjectively construed (cf. Thon 2016).

The theoretical distinction between a complete narrative and its scaffolding core story is essential for understanding transmedial narration: stories are embedded in narratives and, to a certain extent, may be realised by dissimilar media. However, the surrounding narratives and the representing media products are often confated in narrative theory and sometimes termed discourse (but not by Seymour Chatman in 1978: 23–24). Yet, there are not only two levels here—called, for instance, story and discourse—but rather three (cf. Genette 1980 [1972]; Bal 2009 who also suggested three-layer distinctions, although quite different from mine). I suggest that (1) a media product with particular basic media traits and other formative qualities provides certain sensory confgurations that are perceived by someone; these sensory confgurations come to represent (2) media characteristics forming a complete narrative with all its many specifc details and features; furthermore, the perceiver comprehends that this narrative surrounds (3) a scaffolding core, the story, consisting of represented events that are temporally interrelated in a meaningful way.

Based on these defnitions and distinction, one can more clearly describe why stories and parts of their surroundings in the whole narrative may often be realised fairly completely by several kinds of media. It is because many media types have the capacity, to some extent, to represent events, temporal relationships, meaningful relationships and an abundance of other media characteristics. The story is normally only one of several transmedial media characteristics in narratives. The complete narrative of a certain media product can include a multitude of different media characteristics that may be more or less transmedial. However, as a rule, a story, consisting of the essential temporal structure of a narrative, is more transmedial than the complete narrative, although probably never wholly transmedial.

Considering that narration involves comprehension of represented events that are temporally interrelated in a meaningful way, it is clearly a cognitive process that depends on collateral experience and, more specifcally, often on cognitive schemata. Sensing interrelations to be meaningful is at least partially a question of being able to relate them to things with which one is already familiar. Narratives are not objectively construed but depend on the mind's inclination to form gestalts.

However, it is ultimately the more inherent factors of media products that trigger the mind-work of communication and, to some extent, determine how and to what degree various media forms may realise narration. It is clear that the same perceiving mind, harbouring a certain set of knowledge, experiences, values, memories and schemata, will interpret different media products in very different ways even if it perceives them in comparable circumstances. This is obviously because the media products are unalike in various ways and because the divergences are highly relevant. In order to understand how dissimilar media types can communicate narratives, one must scrutinise the fundamental similarities and differences among media types and the extent to which these differences in modality modes matter (Elleström 2019: 53–58, 2020a).

Although differences in modality modes are largely responsible for differences in the kind and degree of narration in various media forms, examining them does not offer a convenient shortcut to full understanding. Thinking in terms of media modalities is not a quick fx. The basic presemiotic and semiotic traits are always embedded in complex surroundings, which means that they generally need to be analysed in their interactions with each other and with additional factors. Nevertheless, modelling narration in terms of media modalities facilitates a methodical approach to the issue of transmediality. Having different material, spatiotemporal and sensorial modes implies having partly dissimilar capacities for narration and, similarly, the use of different sign types has consequences for narration.

The material modality is perhaps the least crucial category of media traits for determining narrative capacities. Solid media products such as written verbal texts, as well as non-solid media products such as spoken verbal texts, clearly have very high narrative capacity, as decades of intense research has demonstrated. Furthermore, organic media products such as moving human bodies, as well as inorganic media products such as dolls in motion, may form complex narratives.

The spatiotemporal modality is much more critical for narration because the scaffolding core of narratives consists of represented events that are temporally interrelated. The key issue then becomes the extent to which the representation of a temporal object requires a representamen with certain spatiotemporal qualities. There is not much to indicate that media products should have specifc spatial traits in order to be able to narrate successfully. Moving human bodies and dolls in motion are threedimensional and suitable for narration. Written verbal texts are twodimensional but also have the potential to be superbly narrative media products. Spoken verbal texts emanating from a singular source are spatial only in a limited way, but are still well suited for narration.

However, there are some relevant differences between temporal and static media products. Moving images that are inherently temporal may effortlessly represent sequences of events and form elaborate narratives (cf. the discussions of narration in silent flm, sound flm, opera, theatre and performance in Lavender 2020; Lutas 2020; Simonson 2020; Tseng 2020; Virginás 2020). This is not to say that one must understand the represented events to be interrelated in precise accordance with the temporal unfolding of the media product. In contrast, still images are, by defnition, static and therefore incapable of representing events that one inescapably perceives in a certain temporal order. However, this is not the same as being incapable of representing temporally interrelated events. It only means that the scope of possibly represented events is reduced (assuming that the size of the still image is not huge) and that the perception of possibly interrelated represented events is not strongly directed by the physical interface of the media product (cf. the analysis of narration in architecture, a non-temporal and very much iconic qualifed media type, in Vieira 2020).

Nevertheless, the difference in spatiotemporal modes reduces the narrative potentiality of still images compared to moving images—at least if one considers media products constituted by single still images. However, it is possible to construe media products consisting of a whole set of still images. In itself, this does not enhance the narrative capacity, but it does open the way for the use of a special kind of symbolic element, namely the convention of sequential decoding. Perceivers who have learnt to process parts of certain kinds of static media products in a regulated order may distinguish represented events in temporal sequences that are as stable as those produced by physically temporal media products (as demonstrated in the narratological analyses of illustrations, comics and graphic novels by Domingos and Cardoso 2020 and Newell 2020).

This line of reasoning is also applicable to the difference between spoken verbal texts and written verbal texts: the distinction between temporal and static media products cuts through both images and verbal texts. Spoken verbal texts are temporal because the sensory confgurations of such media products change constantly. Written verbal texts are static because the sensory confgurations of such media products remain the same from one moment to the other (unless, of course, one perceives the text while it is being written or is a part of a temporal, visual media product such as a flm). This means that spoken verbal texts, just like moving images—given that one allows for a certain volume of temporal extension—readily represent sequences of events and may therefore produce intricate narratives (as seen in the analysis of audiobooks and radio speech, respectively, by Have and Pedersen 2020 and Simonson 2020). In contrast, written verbal texts are normally static and if we think of written verbal texts in rough analogy with solitary still images—namely as consisting of single entities such as one letter or one word—written verbal texts are equally handicapped when it comes to representing events that are inevitably perceived in a certain temporal order.

In the case of language, however, the convention of sequential decoding is so strong that one normally understands written verbal texts as consisting of large sets of subordinate symbols that are bound to be decoded in a highly regular manner. As in the case of sequential decoding of still images, this can lead to the discernment of represented events that are temporally interrelated in a manner that is as stable as those formed by physically temporal media products (cf. the discussions of narration in non-temporal, language-based media types such as various forms of written literature [Have and Pedersen 2020; Lutas 2020; Newell 2020; Vieira 2020] and popular scientifc articles [Bruhn 2020]). This is why so many researchers—misleadingly, I would argue—have claimed that written verbal texts are temporal. Such a conception obscures the difference among the physical appearance of representamens (the traits of media products), the process of perceiving the physical appearance of representamens, and the virtual appearance of represented objects (the traits of virtual spheres).

Thus, the fact that one perceives all kinds of media in time has some bearing on their capacity to represent temporally interrelated events: conventionalised orders of decoding may strongly enhance the narrative capacity of static media types. However, this does not erase the substantial differences between inherently temporal and static media.

The sensorial modality also plays a role for the narrative capacity of media products. This is mostly because the senses (understood here as the external senses) are not developed cognitively to the same degree. Sight and hearing are our two most advanced senses, in that they are strongly connected to complex cognitive functions such as knowledge, attention, memory and reasoning. This means that sight and hearing are both well suited for narration. Indeed, all contributions to *Beyond Media Borders* that engage in narration deal almost exclusively with visual and auditory media types.

However, this does not exclude in principle the other senses. One may use the faculty of touch for reading braille, for instance, or sensing the forms of reliefs and three-dimensional fgures forming narratives. It is also fully possible to consider successions of interpersonal touches that form casual, narrative media products. Children playing and adults having sex may well communicate elementary narratives by way of sequences of touches that are performed and located differently.

I presume that it would also be possible, in principle, to construe language systems mediated by taste or smell. In practice, however, they would probably be ineffcient, as a speedy decoding of symbols requires quickly performed sensory discriminations. However, one can use taste and smell to create at least rudimentary narratives. A well-planned meal with several courses served in a certain order may be construed as narrative, to the extent that tastes and taste combinations may be developed, changed and contrasted in such a manner that gives a sense of meaningfully interrelated events. A series of scents may be presented in such a way that represents, say, a journey from the city through the woods and to the sea, including encounters with people and animals with smells that reveal certain activities.

The three main modes of the semiotic modality are iconicity (based on similarity), indexicality (based on contiguity) and symbolicity (based on habits). All of these semiotic modes are immensely important for the realisation of narration. The majority of the more acknowledged basic media types that are commonly reasonably well defned and have accepted names in ordinary language are dominated by iconicity or symbolicity. One can clearly characterise most of the recent examples of potentially narrative media types by a semiotic hallmark. Verbal texts, whether they are visual, auditory, or tactile, rely heavily—although certainly not exclusively—on symbolicity: the conventional meaning of letters, sounds, words and so forth. Moving and still images, whether they are visual, auditory or tactile, are understood to signify primarily through iconicity, based on perceived similarities between representamens and objects. Although series of touches, tastes and scents are hardly acknowledged as media types in common parlance, one could make a case for recognising them as basic media types dominated by indexicality: real connections between the perceived sensory confgurations and what they stand for.

For the sake of clarity, I have tried to isolate the possible contributions of various media modes to narration. By highlighting modal differences, it is possible to discern media traits that contribute to narration existing in different degrees. However, media products are normally more or less multimodal—in very different ways—which makes the above generalisations fuzzier, the differences among media types more subtle and the issue of transmedial narration more multifaceted. The model of media modalities does not offer a lexicon of transmedial narrative capacities as much as a methodical approach to examining narration in a wealth of dissimilar media products and media types. In each specifc media product and media type, the present modes of the modalities add, in profound interaction, to the forming of virtual spheres and possibly narratives. In a certain media product, all of the various presemiotic modes contribute to forming certain sensory confgurations: a cluster of physical representamens that together come to represent—iconically, indexically or symbolically—a certain cluster of objects that possibly forms a narrative.

Therefore, I support Karin Kukkonen's conclusion that "[i]f, with Ryan, we understand narrative as a cognitive construct, different modes in multimodal media work together to provide the reader with clues to fll gaps and formulate hypotheses" (Kukkonen 2011: 40). Importantly, however, I go beyond the rather coarse notion of mode used by Kukkonen and in so-called social semiotics in general: modes understood as text, image, gesture and so forth. For me, multimodality is a more fne-grained concept that can be more precisely circumscribed as four kinds of multimodality: multimateriality, multispatiotemporality, multisensoriality and multisemioticity. As a rule, actual media products and media types have many modes of the same modality. For instance, media products that consist of both organic and non-organic materiality are multimaterial. Media products that are both spatial and temporal are multispatiotemporal. Audiovisual media products are multisensorial. Furthermore, many media are multimodal in several ways simultaneously.

Finally, most media products are multisemiotic to the extent that sign types typically work in collaboration. In an early article advocating the value of applying Peircean semiotics to the study of narratives, Robert Scholes suggested, "we cannot understand verbal narrative unless we are aware of the iconic and indexical dimensions of language" (1981: 205), which is certainly true. Even though symbolic signs are clearly the most salient ones, verbal language does not work solely through symbolicity. In visual language, for instance, elements such as lineation, letter size, letter form and empty spaces may create iconic meaning. In auditory language, iconicity is often produced by certain sound qualities, intonations, rhythms and pauses. By the same token, most media types signify through iconicity, indexicality and symbolicity in combination, although they are typically dominated by certain kinds of sign functions. However, one can fnd instances of communication and narration characterised by such extreme multimodality that virtually all kinds of modality modes, both presemiotic and semiotic, are included.

#### 8.4 Language

Having emphasised the multisemioticity of language and noticed that language may be visual as well as auditory, it is time to get into the details of the concept of language itself (Elleström 2020b, forthcoming). How does it ft into the theoretical framework presented in "The modalities of media II: An expanded model for understanding intermedial relations" (Elleström 2020a)? Should the concept be understood as a media type? Not really. In her contribution to *Beyond Media Borders*, where she investigates a broad range of language forms in the context of language learning, Heather Lotherington accurately notes that "[l]anguage has traditionally been described as a *medium*: of communication, and of learning", although "[l]anguage is, in fact, an abstract until it is materialized: mediated physically, in speech and signed conversations, and technologically, in printed documents, social media sites, roadside signs, movies, games, and suchlike" (Lotherington 2020: 219).

If language is an abstraction, then how should it be circumscribed? In line with a common practice, I suggest that language might be comprehended as *a communicative sign system*. Given that a system is something that is highly organised, a communicative sign system must rely strongly on robust habits or, in other words, conventions. In terms of Charles Sanders Peirce's semiotic vocabulary (Peirce 1932: CP 2.297 [c. 1895]), this reliance suggests that communicative sign systems are *symbolic sign systems*. Therefore, I defne a language as *a system of symbolic signs used for communication*.

Given such a defnition, it is clear that there are several forms of language. Apart from so-called verbal language, humans have developed a plenitude of symbolic sign systems involving a broad variety of signs, such as mathematical and logic symbols, maritime signal fags, declamatory gestures in theatre and even fowers that communicate socially through their kinds, colours and arrangements. Symbolic sign systems may consist of anything from thousands of symbols and very elaborate and detailed conventions to considerably fewer symbols connected more roughly through limited sets of habits. In their contribution to *Beyond Media Borders,* Domingos and Cardoso (2020) thoroughly analyse the combination of verbal and nonverbal languages in comics, involving both highly structured visual verbal language (involving conventional use of letters, punctuation, grammar, etc.) and less formalised nonverbal language (based on the habits of forming and combining visual, static images).

Verbal language, understood as language involving features such as words and grammar, is based on extensive and very systematic conventions. However, verbal language is not really 'one' system, even disregarding the crucial differences between language forms such as Hindi, Spanish and Afrikaans. Some decades ago, Jan Mulder argued in detail for the notion that written and spoken languages constitute separate semiotic systems. Based on the simple but far-reaching observation that what is seen (written words) is different from what is heard (spoken words), Mulder argued that it is "terribly wrong" to see written and spoken languages as "mere variants of the same thing"; they are "entirely different semiotic systems" (Mulder 1994: 43). I think this idea is correct, even though the two different systems of symbols have been developed to allow for advanced mutual transmediation: changing from writing and reading to speaking and listening is often an effcient although defnitely not seamless procedure. Hence, in more general discussions, it is fully comprehensible to treat verbal language simply as a common system of symbols (as Vieira 2020). At other times, however, it is crucial to be more specifc about how communication is realised, which requires specifying whether written language (Bruhn 2020; Lutas 2020; Newell 2020), spoken language (Lavender 2020; Simonson 2020) or both (Eide and Schubert 2020; Have and Pedersen 2020) are at stake.

There are actually at least three different verbal systems of symbols. Sign language also interconnects strongly with written and spoken verbal language, although it is clearly a sign system of its own and not only a variant of the same thing. Each of these three or more systems can be subdivided into even more specifc systems such as written Swedish, spoken English or Chinese sign language.

Although the defnition of language as systems of symbolic signs used for communication captures the core of languages, it is naturally not an exhaustive description of how language can be conceptualised. Whereas written verbal language, my primary example here, is certainly always per defnition based on systems of symbols, written verbal language may also include other sign types. In accordance with Peirce's notion of sign types always being mixed (Peirce 1932: CP 2.302 [c. 1895]), there are probably also always at least some elements of iconicity and indexicality in semiosis dominated by symbolicity. Depending on factors such as history and culture, nonprofessionals and scholars may see these other sign types as more or less inherent or alien facets of what they understand to be normal written verbal language. Even though one can give written verbal language a transhistorical and transcultural core defnition in terms of symbolic sign systems, it is clear that its realisations vary considerably in relation to time and space.

To exemplify this, I will focus here on the presence of iconicity in written verbal language. Iconic meaning-making is not based on habits or conventions, as is the case for symbolic meaning-making. Iconicity is grounded in perceived similarities among sensory perceptions and cognitive structures. In the domain of written verbal language, seen from a Western perspective, this means that symbols such as letters, words and punctuation, as well as spatial arrangements of these, may to some extent produce meaning. This is not only because of learned, habitual connections between the visual impressions and what one takes them to represent, but also because of perceived similarities between what one sees and what the visual impressions are understood to signify. For instance, visual empty spaces between words and sentences suggest semantic spaces or differences. A more blatant example can be provided with a sentence like this: 'I can see two mOOns.'

Most Western scholars have long been strongly inclined to dismiss iconicity as, at best, only a peripheral part of verbal language, oral as well as written. Since the spread of Ferdinand de Saussure's linguistic dogma of the arbitrariness of language in *Course in General Linguistics* a century ago (2011 [1916]), belief in iconicity's existence in or relevance for language has sometimes been considered naïve and has even been subject to scorn and ridicule.

However, this has not been the case for Eastern scholars. According to a recent article by Ersu Ding, Saussure's quick dismissal of iconic signs in language as peripheral and linguistically uninteresting has been met with much greater scepticism in China (Ding 2014: 121). As Ding succinctly put it, "his holistic scheme of sign formation simply could not accommodate anything other than a systemic pairing of the signifer and the signifed as a result of structural differentiation" (Ding 125). In China and other Eastern countries, such as Japan, the basic concept of language, including written verbal language, has thus been more inclusive—even though the core of language as a system of symbolic signs used for communication has not been denied, to the best of my knowledge. Therefore, one could dare to say that the concept of written verbal language in the West and the East has largely been the same, but not really the same. In any case, there is now massive support from empirical research, from both the West and the East, that iconicity, in varying degrees, is a solid component of verbal languages from all over the world.

What, then, is the relation between the two concepts of language and medium? As I defne it, a media product is a material entity—or, more broadly, a physical, intermediate entity—that makes communication among human minds possible (Elleström 2018). Thus, media products are material objects, actions or processes that are indispensable for connecting human beings mentally. Consequently, they must be conceptualised in terms of both physicality and cognitive capacities. In other words, media products have both pre-semiotic traits and semiotic traits, meaning that they manage to trigger meaning-production in various ways based on their material, spatiotemporal and sensorial traits.

Furthermore, nonprofessionals and scholars tend to group media products into categories. People sometimes pay attention mainly to the most basic features of media products and classify them according to their most salient material, spatiotemporal, sensorial and semiotic properties. For instance, we think in terms of still images (most often understood as tangible, fat, static, visual and iconic media products). Still images are an example of what I call a basic medium (a basic type of media product).

However, such a basic classifcation is sometimes not enough to capture more specifc media properties. We then qualify the defnition of the media type that we are after and add criteria that lie beyond the basic material, spatiotemporal, sensorial and semiotic properties; we include all kinds of aspects regarding how the media products are produced, situated, used and evaluated in the world. One may wish to delimit the focus to still images that, for example, are produced with the aid of cameras—photographs. Photographs are an example of what I call a qualifed medium (a qualifed type of media product).

In contrast to media types, which involve pre-semiotic and semiotic traits, I comprehend languages somewhat more narrowly as systems of symbolic signs. From these dissimilar defnitions, it follows that one cannot simply equate languages with media types. Media are material, and, as Lotherington puts it, language is something abstract "until it is materialized: mediated physically" (Lotherington 2020: 219). However, languages and media types are clearly interconnected. One way of putting it is to say that a language can be a vital semiotic part of a media type. For instance, language is an essential part of email and telephone conversations as well as of theatre and stand-up comedy. Put another way, specifying the defnition of a language from an *abstract* symbolic sign system to *the concrete use* of a symbolic sign system brings you closer to understanding the language as a media type. Fully concretising how a symbolic sign system is mediated—that is, how it is realised in specifc material, spatiotemporal and sensorial ways—makes a symbolic sign system equivalent to a media type, as I conceive it.

This equivalence is actually partly achieved in the distinction between at least three different verbal sign systems mentioned earlier: written, spoken and 'signed' verbal language. The notion of written verbal language defnitely includes a specifc sensorial mode, visuality, and often a spatiotemporal mode: the visual signs being realised on a fat surface. Likewise, the notion of spoken verbal language defnitely includes a specifc sensorial mode, audibility, which requires materiality capable of transmitting sound waves and temporal extension. The notion of 'signed' verbal language in effect fully covers the criteria for being understood as a media type: sign language is mediated by human bodies acting in time and three-dimensional space to produce movements that form visual signs; these signs are clearly part of a verbal symbolic sign system involving strong facets of iconicity and indexicality. Considering that sign language is also designed specifcally for face-to-face communication with and among people with hearing impairments, it is a qualifed rather than a basic media type. Other fully defned verbal sign systems include Braille writing, designed for tactile verbal communication with people with visual defciencies and, by the same token, a qualifed rather than a basic media type.

However, one can hardly describe 'normal' written verbal language (based on vision) as a qualifed media type. Instead, I would say that this category includes several closely related basic media types. As mentioned, written verbal language is defnitely visual and normally realised on fat surfaces, but it can also occur on rounded and irregular surfaces, and letters and words may be three-dimensional in themselves. Written verbal language is most conveniently mediated through solid and inorganic materiality, but it is fully possible to write in gas and liquids or with the use of organic material such as living bodies. Thus, written verbal language is quite an open category that includes a variety of basic media types, depending on whether the more exclusive Western or more inclusive Eastern comprehensions are included.

#### References


Ding, Ersu. 2014. Ubiquitous but Arbitrary Iconicity. *Semiotica* 200: 119–135.


———. 2014. *Media Transformation: The Transfer of Media Characteristics among Media*. Basingstoke: Palgrave Macmillan.

———. 2017. Adaptation and Intermediality. In *The Oxford Handbook of Adaptation Studies*, ed. Thomas Leitch, 509–526. New York: Oxford University Press.

———. 2018. Modelling Human Communication: Mediality and Semiotics. In *Meanings & Co.: The Interdisciplinarity of Communication, Semiotics and Multimodality*, ed. Alin Olteanu, Andrew Stables, and Dumitru Bortun, 7–32. ̧ Cham: Springer.

———. 2019. *Transmedial Narration: Narratives and Stories in Different Media*. Basingstoke: Palgrave Macmillan.

———. 2020a. The Modalities of Media II: An Expanded Model for Understanding Intermedial Relations. In *Beyond Media Borders: Intermedial Relations among Multimodal Media*, ed. Lars Elleström, vol. 1, 3–91. Basingstoke: Palgrave Macmillan.

———. 2020b. Language, Cultural Differences, and Media Types. *Comparative Literature and Cross-Cultural Studies* 2020: 2. Forthcoming.


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/ by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## Index1

#### **A**

Actor, 50, 51, 62, 67, 143, 160, 180, 201, 214 *See also* Actress; Performer Actress, 20, 25, 201 *See also* Actor; Performer Adaptation, 16, 33–53, 104, 107, 112, 113n11, 155, 158, 187, 215–218 *See also* Transmediation Affection, 5, 18, 26, 65, 68, 113n10, 127, 129, 139, 158, 171, 177, 204, 214 Aillaud, Émile, 60, 67, 73 Altman, Rick, 27n4 Analogue, 67, 74n8, 150, 157, 196, 222 *See also* Digital Appino, Nikki, 18, 19, 24, 25 Architecture, 59–73, 181, 183, 186, 202, 214, 215, 221 Arnheim, Rudolf, 64, 65, 68

Artifcial intelligence (AI), 202 Atherton, James S., 103, 107, 108 Atwood, Margaret, 34, 37, 39–42, 44, 46–49, 53, 54n5, 54n7, 117 Audiobooks, 214, 222 Audiovisuality, 48, 49, 91, 225 Aural, *see* Hearing, sense of Auslander, Philip, 67

#### **B**

Bal, Mieke, 70, 74n13, 219 Balbusso, Anna, 43–45 Balbusso, Elena, 43–45 Ballet, 48, 49, 51, 80, 217 Bareis, Alexander, 152 Basic media, *see* Media types Bernstein, Arielle, 51 Bertotti, Ugo, 92, 100, 110 Biography, 79–112, 214, 215, 217 Biselli, Mario, 60, 73n1 Boenisch, Peter M., 5, 26, 27, 27n2

1Note: Page numbers followed by 'n' refer to notes.

© The Author(s) 2021 235 L. Elleström (ed.), *Beyond Media Borders, Volume 2*, https://doi.org/10.1007/978-3-030-49683-8

Bolter, Jay D., 13 Bradshaw, Peter, 166–168 Browning, Tod, 5, 8 Bruchard, Dorothée de, 82, 83 Bruhn, Jørgen, 61, 122, 123, 137, 146n4, 166, 215, 219, 222, 227

#### **C**

Caplan, Lucy, 54n7 Cardoso, José Arlei Rodrigues, 215, 217, 219, 222, 226 Chapple, Freda, 27n1 Chion, Michel, 9 Cinema, v, 11, 20, 66, 81, 82, 92, 104, 105, 121, 127, 150 *See also* Film Cognitive import, vii, 3, 62, 64, 67, 92 Cohn, Dorrit, 153, 162 Comics, 27n5, 46, 79–112, 166, 214, 215, 217, 222, 226 Communication extra-communicational domain, 187 intracommunicational domain, 28n14, 193 Conway, Jack, 9 Crossley, Mark, 202, 214 Crowther, Paul, 60, 65

#### **D**

Dahlin, Åsa, 67 Diegesis diagetic screen, 28n14, 169 extradiegetic, 80, 164, 169 intradiegetic, 156, 157 metadiegetic, 150, 169 Digital, 4, 66, 74n7, 74n12, 98, 104, 110, 118, 131, 134, 135, 185, 192, 200, 204, 205n2, 215 *See also* Analogue Ding, Ersu, 228

Domingos, Ana Cláudia Munari, 215, 217, 219, 222, 226 Dumas, Alexandre, 162 Dundas, Deborah, 46, 54n2 Dunne, Daisy, 130, 132, 145

#### **E**

Eco, Umberto, 85 Ecocriticism, 117–146, 215 Education, xi, 218 *See also* Learning; Teaching Eisner, Will, 81, 86, 87, 92, 96, 105, 113n9 Ekphrasis, 40, 60, 61, 64, 69–72, 74n5, 186 *See also* Media representation Elkins, James, 69, 70 Elleström, Lars, v–vii, 3–5, 14, 24, 28n8, 28n13, 28n14, 29n16, 35, 36, 52, 53, 54n4, 60, 62, 63, 65–67, 72, 73, 80, 82, 88–90, 92, 99, 107, 112n2, 119, 122, 127, 128, 138, 146n3, 149–152, 155, 158, 169, 171, 177, 181–183, 187, 189, 190, 192–194, 198, 203, 204, 213, 215–218, 220, 225, 228 Elliott, Kamilla, 216 Email, 229 Erlingsson, Benedikt, 166 Estok, Simon C., 121 Extracommunicational domain, *see* Communication

#### **F**

Fain, Tim, 18–22, 26 Fiction, 62, 63, 71, 80, 88, 89, 118–120, 122, 138–140, 142, 145, 151–153, 156, 157, 159, 186–188, 193, 202, 204 Fictionality, 153

Film, 4–26, 27n4, 27n6, 28n9, 28n10, 28n11, 28n12, 35, 48, 50, 51, 61, 91, 112n6, 117–120, 123, 139, 146n2, 149, 155, 156, 158–170, 178, 193, 196, 201, 202, 214, 216–218, 221, 222 *See also* Cinema Fludernik, Monika, 71, 154, 162 Focalization, 70–72, 74n13 Forster, Marc, 159

#### **G**

Gabrys, Jennifer, 125 Garrard, Greg, 120, 121 Genette, Gérard, 70, 74n13, 150, 151, 154–158, 161, 163, 164, 169, 219 Genre, 5, 25, 80, 122 *See also* Submedium Gesture, 20, 40, 44, 48, 49, 87, 181, 182, 185, 186, 191, 224, 226 Gibson, James J., 176 Gjelsvik, Anne, 122 Glaser, Stephanie A., 59, 66, 68, 70, 71 Graphic novel, 42–48, 54n2, 87, 88, 113n9, 214, 217, 222 Grusin, Richard, 13 Guibert, Emmanuel, 92, 99, 101, 109

#### **H**

Hall, Stuart, 125 Hamon, Philippe, 60, 61, 63, 74n10 Harries, Karsten, 60, 69 Hatfeld, Zack, 50 Have, Iben, 176, 214, 222, 227 Hearing, sense of, 6, 9, 15, 156, 165, 182, 223 Heise, Ursula K., 125, 126 Heteromediality, 217 Hutcheon, Linda, 123, 217 Hypermediacy, 202

**I** Iconicity, 42, 63, 107, 182, 223–225, 227, 228, 230 *See also* Icons; Sign types Iconography, 33–53 Icons, *see* Sign types Illustration, 22, 40–48, 83, 84, 93, 94, 96–98, 101–104, 106–108, 110, 111, 112n1, 132, 134, 135, 137, 214, 222 Images, v, 4, 9, 10, 12–15, 18–23, 25, 27n4, 33, 34, 40, 42, 44–47, 50, 51, 54n5, 60, 64, 68, 69, 74n12, 80, 81, 83, 84, 86–89, 91–94, 96, 97, 99, 103, 104, 106, 107, 110, 111, 112n1, 123, 130, 133, 134, 136, 158, 167, 182, 184, 199, 202, 217, 221, 222, 224, 226, 229 *See also* Icons; Sign types Indexicality, 63, 79–112, 223–225, 227, 230 *See also* Indices; Sign types Indices, *see* Sign types Internet, 98, 107, 127, 129, 130 Interpretant, *see* Sign constituents Intracommunicational domain, *see* Communication

#### **J**

Journalism, 15, 79–112, 135, 214, 215 Joyce, James, 103, 104 Joyce, Lucia, 103, 107

#### **K**

Kalinak, Kathryn, 167 Kattenbelt, Chiel, 27n1, 28n7, 67, 73 Klimek, Sonja, 151, 161, 164 Kress, Gunther, 123 Kukkonen, Karin, 151, 224

#### **L**

Laguionie, Jean-François, 161 Lang, Sabine, 151, 153, 161 Language, vii, 4, 18–26, 29n16, 72, 80–82, 125, 131, 137, 145, 150–152, 155, 158, 165, 166, 176, 178–180, 183, 191, 197, 199, 214, 215, 222, 223, 225–230 *See also* Sign types; Symbols; Texts; Verbal Lanzius, Evelyn, 12–15 Lavender, Andy, 17, 28n13, 214, 221, 227 Learning, 214, 225 *See also* Education; Teaching Lefèvre, Didier, 97, 99, 101, 109 Lemercier, Frédéric, 99, 101, 109 Lessing, Gotthold Ephraim, 190 Linguistics, 23, 24, 64, 137, 150, 151, 155, 183, 228 Literacy, 62 Literature, 34, 59, 60, 62, 63, 66, 67, 70, 72, 73, 74n10, 80–82, 95, 96, 113n9, 120, 127, 132, 137, 138, 143, 146n1, 146n2, 149–160, 165, 170, 171, 186, 214–218, 222 Lotherington, Heather, 214, 225, 229 Louvel, Liliane, 70 Lutas, Liviu, 202, 215, 217, 219, 221, 222, 227 Lyman, David, 53n2, 54n2 Lynch, David, 195

#### **M**

Maps, viii, 101, 130, 131, 133, 134, 141, 143, 177, 178, 180–190, 192–194, 197–200, 203, 204 Marcellino, Fred, 40, 46, 50

Materiality, viii, xii, 9, 36, 38, 45, 52, 64, 66, 67, 72, 198, 225, 229, 230 McCloud, Scott, 81 Media borders, xii, xiii, 35, 119, 124, 126, 146, 183, 192, 217 Media characteristics, 60, 62, 72, 90, 91, 152, 153, 158–160, 170, 204, 216, 218–220 Media integration, xii, 123, 128, 133, 138, 139 Media modalities material modality, 64, 72, 158, 189, 198, 220 semiotic modality, 36, 72, 80, 88, 89, 152, 163, 178, 189, 223 sensorial modality, 11, 65, 152, 178, 198–199, 223 spatiotemporal modality, 63, 65, 66, 82, 89, 163, 176, 189, 221 Media representation, 79–112, 158 Mediation, v–ix, 3, 5, 26, 36–38, 63, 80, 89, 92, 121, 197 Media transformation, xii, 90, 104, 107, 123–124, 128, 133, 138, 139, 146n3, 192–194, 198, 206n15 Media translation, 4, 69, 73n1, 107 Media types, xiii, 80–93, 106, 111, 112n2, 121–123, 141, 145, 181, 183, 214, 217, 221, 230 basic media types, xiii, 88, 122, 183, 223, 224, 230 qualifed media types, 81–88, 90–93, 96, 106, 111, 112n2, 119, 121, 122, 128, 141, 143, 181, 214, 217, 221, 230 Medium specifcity, v, 122–124, 142, 144 Melnick, Ross, 12, 15 Méndez-Leite, Fernando, 155

Metalepsis, 149–171, 202, 215 Miller, Bruce, 51 Mind perceiver's mind, 4, 36 producer's mind, 3, 36 Mitchell, W. J. T., 122 Modalities, v, 4, 6, 9, 12, 14, 15, 17, 25, 34, 36, 49, 51, 53, 60, 63, 66, 72, 73, 80, 82, 91, 122, 124, 133, 145, 151–153, 169, 171, 189, 198, 224 *See also* Media modalities Modality modes, 60, 61, 73, 88, 104, 215, 220, 225 Modelling, 122, 177, 192–196, 198, 204, 206n15, 220 Modes, 5, 26, 59–73, 80, 89, 91, 92, 104, 123, 152, 156, 158, 160, 168–171, 176, 190, 198, 203, 205n13, 220, 221, 223, 224, 229 *See also* Modality modes Morton, Timothy, 125, 146n4 Moya, Álvaro de, 112n7 Mulder, Jan W. F., 226 Munari Domingos, Ana Cláudia, 215, 217, 219, 222, 226 Music, 13, 16, 18, 20, 22, 24, 26, 28n11, 37, 38, 48, 90, 144, 146n2, 150, 158, 165–169, 171, 182, 202, 214, 217

#### **N**

Narration narrative, 4–7, 9, 11, 13, 15, 17, 20, 22–26, 27n1, 28n8, 35, 37, 44, 48, 50, 60, 61, 71, 80–87, 89, 90, 92, 93, 96–99, 102–104, 108, 120, 121, 126, 127, 133, 134, 137–139, 145, 149–170, 184, 191, 217–225

narrator, 18, 22, 37–42, 44–49, 53, 68, 70, 71, 73, 87, 88, 93, 94, 98, 102, 140, 141, 150, 153–163, 165, 168–170 Nault, Renée, 43, 45–48, 54n2 Newell, Kate, 6, 42, 215, 217, 219, 222, 227 Nixon, Rob, 124, 125 Norberg-Schulz, Christian, 64

#### **O**

Object, *see* Sign constituents Ong, Walter J., 181 Opera, 4, 13, 15–17, 28n10, 50, 54n6, 61, 214, 217, 221 Orality, 181 *See also* Speech

#### **P**

Participation, 99 Pedersen, Birgitte Stougaard, 176, 214 Peirce, Charles Sanders, 63, 90, 226 Pelletier, Louise, 68–69 Penna, Gustavo, 60, 73n1, 73n2 Perception, xii, 28n7, 28n14, 35–37, 49, 64, 67–69, 80, 81, 194–196, 198, 221, 227 Pérez-Gómez, Alberto, 68, 69 Performance, 4, 7, 8, 10, 12–18, 20, 21, 24–27, 27n2, 28n13, 48–50, 61, 66, 67, 73, 90, 107, 166–168, 171, 181, 186, 195, 206n13, 214, 221 Performer, 5, 12–19, 21, 24, 49, 205n7, 214 *See also* Actor; Actress Pier, John, 162–164 Plath, Sylvia, 107, 108 Print, 4, 15, 38, 81–83

#### **Q**

Qualifed media, *see* Media types Qualifying aspects contextual qualifying aspects, 63 operational qualifying aspects, 63

#### **R**

Radio, 4, 12–15, 21, 27n6, 36, 93, 138, 158, 181, 214, 217, 222 Rajewsky, Irina O., 5, 27n1, 35, 53, 54n3, 98, 149, 158 Rasmussen, Steen E., 61, 62, 66–68, 73 Reading, 28n11, 37, 52, 54n7, 80, 81, 86, 93, 104, 107, 119, 130, 131, 134, 137, 153, 156, 176, 178, 179, 184, 187, 189, 217, 223, 226 Remediation, 13 Representamen, *see* Sign constituents Representation, 10, 16, 26, 41, 53, 61–63, 66–69, 72, 74n10, 74n12, 83, 84, 86, 88–99, 101–104, 106, 110–112, 113n12, 120–126, 131, 132, 135, 139, 141, 144, 150–152, 154, 155, 158, 166, 168, 170, 182, 194–196, 203, 214, 216, 221 Rodrigues Cardoso, José Arlei, 215, 217, 219, 222, 226 Rothafel, Samuel 'Roxy,' 12–14, 16, 26, 27n6 Roxy, *see* Rothafel, Samuel 'Roxy' Rust, Stephen, 121 Ryan, Marie-Laure, 154, 162, 165, 187, 218, 224

#### **S**

Sacco, Joe, 86–88, 92, 97–99, 101, 102, 113n10 Satrapi, Marjane, 92, 94–96, 104, 106 Saussure, Ferdinand de, 228 Schlöndorff, Volker, 50 Scholes, Robert, 225 Schumacher, Patrik, 61, 74n4 Science, 60, 69, 118–120, 125, 129, 130, 132, 133, 135, 136, 138, 140, 142, 143, 145, 146, 198, 202 Screens, 6, 8–10, 13, 15–21, 23, 28n14, 36, 38, 44, 98, 99, 110, 127, 130, 132, 158, 159, 163, 168, 169, 178, 194, 202 Semiotics presemiotic modalities, 63, 67, 89, 216, 220, 224, 225, 228 semiosis, 64, 89, 216, 227 semiotic modality, 36, 63, 72, 80, 88, 89, 152, 163, 178, 189, 223 Sequentiality, 43, 178, 203 Sight, sense of, viii, xiii, 15, 44, 45, 49, 50, 64, 132, 189, 223, 230 Sign constituents, 3, 4, 13, 14, 17, 28n8, 35, 37–39, 42, 63, 67, 68, 70, 74n11, 75n13, 80, 89, 90, 107, 123, 155, 158, 189, 191–193, 196, 198, 217, 219, 221, 222, 224, 228 object, 17, 107 representamen, 221, 222, 224 Sign types, 20, 22, 25, 39, 41, 46, 52, 178, 179, 182, 188, 199, 222, 223, 226, 227 icons, 53 indices, 97, 111 symbols, 128, 131, 133, 137, 225–230 Silent flm, 4, 5, 9, 11–15, 17, 18, 21–26, 27n6, 28n10 Simonson, Mary, 28n10, 183, 192, 197, 201, 215, 217, 219, 221, 222, 227 Smell, sense of, 15, 38, 65, 183, 223

Social media, 225

Social semiotics, 224 Song, 15–17, 28n11, 103, 167, 206n14 Sousa, Jorge Pedro, 83, 84, 112n5 Space three-dimensional space, 67, 70 two-dimensional space, 130 *See also* Spatiotemporality Spatiotemporality, 189 Spectator, 27n4, 60, 67, 69, 70, 73 Speech, 6, 7, 11, 24, 27n5, 37, 38, 48, 70, 145, 190, 191, 222, 225 *See also* Orality Spiegelman, Art, 86, 87 Story, 9, 17, 18, 20, 21, 23, 25, 36, 46, 50, 60, 61, 81, 85–88, 90, 91, 93, 96, 98, 99, 101–104, 107, 110–112, 113n9, 117, 134, 136, 137, 151, 153, 154, 157, 160, 161, 185, 186, 190, 192, 197, 198, 214, 218–220 Stougaard Pedersen, Birgitte, 176, 214 Submedium, 61, 62, 72, 74n7, 74n8 *See also* Genre Symbolicity, 63, 223–225, 227 *See also* Sign types, symbols Symbols, *see* Sign types

#### **T**

Talbot, Bryan, 92, 94, 103, 107, 108, 113n14 Talbot, Mary M., 92, 94, 103, 107, 108, 113n14 Taste, sense of, 65, 223, 224 Taylor, Holly A., 71 Teaching, 20 *See also* Education; Learning Technical media, 5, 18, 92–94, 97, 103, 111, 122, 127, 134, 194, 197 Technology, 3, 12, 27n4, 37, 38, 64, 74n12, 82

Television (TV), v, 5, 36, 49, 50, 90, 92–95, 97, 98, 102, 103, 111, 160, 169, 181, 214, 217 Texts, 11, 27n5, 37, 47, 48, 60, 63, 70, 71, 82, 89, 91, 92, 98, 119, 123, 124, 127, 128, 130–135, 138, 140, 141, 143, 145, 155, 158, 165, 177–194, 197, 198, 200, 202–204, 205n5, 205n10, 217, 220–222, 224 *See also* Language, Symbols; Sign types; Verbal Theatre (theater), 4, 12–19, 24, 26, 27n6, 82, 91, 92, 150, 196, 201, 202, 214, 216, 217, 221, 226, 229 Theme, 25, 34, 35, 42, 43, 45, 49, 84, 85, 87, 89, 123, 138, 159 Time, v, 4–6, 11, 15, 18, 21–23, 25–27, 27n6, 34, 37, 44, 46, 49, 53n2, 54n7, 63, 65–67, 69, 79, 80, 84–87, 92, 101, 102, 104, 111, 125, 130, 131, 134–136, 139, 142, 150, 159, 160, 166, 168–170, 181–183, 186, 188–190, 192, 194, 199, 203, 205n7, 206n17, 216, 217, 219, 223, 225, 227, 230 *See also* Spatiotemporality Tolkien, J. R. R., 179, 193, 205n11 Töpffer, Rodolphe, 81 Touch, sense of, 38, 64, 137 Translation, *see* Media translation Transmediality, 90, 149–151, 165, 215, 217, 220 Transmediation, 37, 61, 64, 73, 79–112, 112n1, 158, 190, 216–218, 226 Truthfulness, 93, 96, 193, 214 Tseng, Chiao-I, 188, 214, 221 Tsukada, Saori, 18–26, 28n13 Tversky, Barbara, 71, 176, 203

#### **U**

Unamuno, Miguel de, 153–160, 165, 170

#### **V**

Verbal, 40, 48, 62–64, 66, 71, 72, 80, 81, 86, 88, 89, 95, 99, 107, 128, 131, 133, 137, 163, 166, 214, 217, 220–222, 224–230 *See also* Language; Symbols; Sign types; Texts Vieira, Miriam, 60, 75n14, 181, 183, 186, 196, 215, 219, 221, 222, 227 Virginás, Andrea, 20, 28n14, 169, 195, 214, 221 Virtual reality (VR), 177–179, 188, 189, 193–205, 206n17, 206n18 Virtual sphere, 216, 218, 219, 222, 224 *See also* Intracommunicational domain Vision, *see* Sight, sense of Voice narrative voice, 48, 150, 156, 159–170 recorded voice, 38, 130, 131 singing voice, 15, 16

ventriloquist's voice, 4–7, 9–11 voiceover, 23, 155, 156, 158–161, 168

#### **W**

Wedig, Noelle, 50 Weik von Mossner, Alexa, 120, 127 Weitze, Charlotte, 136–137, 141, 145 Wertham, Frederic, 85 Wolf, Werner, 54n3, 151, 158, 166, 167 Writing, xii, 15, 40, 48, 50, 52, 84, 97, 107, 108, 117, 123, 128, 130, 131, 133–135, 137–139, 143–145, 150, 153, 155, 157, 158, 160, 177–180, 185, 186, 189–191, 193, 203, 204, 205n2, 214, 216, 217, 220–222, 226–230

#### **Y**

Yarde, Alex, 52 Yusoff, Kathryn, 125

#### **Z**

Zutter, Natalie, 36