Locating the Semiotic Power of Multimodality

  title={Locating the Semiotic Power of Multimodality},
  author={Glynda Hull and Mark Evan Nelson},
  journal={Written Communication},
  pages={224 - 261}
This article reports research that attempts to characterize what is powerful about digital multimodal texts. Building from recent theoretical work on understanding the workings and implications of multimodal communication, the authors call for a continuing empirical investigation into the roles that digital multimodal texts play in real-world contexts, and they offer one example of how such investigations might be approached. Drawing on data from the practice of multimedia digital storytelling… 

Figures from this paper

Multimodal Synthesis and the Voice of the Multimedia Author in a Japanese EFL Context
It is argued that in the current semiotic climate, characterised by the increasing availability and complexity of communication tools and ready appropriation of available designs, practices of multimedia authorship truly can evince expression that is authentically multiplicative.
This study of digital storytelling attempts to apply Kress's (2003) notions of synaesthesia, transformation, and transduction to the analysis of four undergraduate L2 writers' multimedia text creation processes to show how synaesthetically derived meaning may be a natural part of the process of creating multimodal texts.
Challenges of Multimedia Self-Presentation
One privilege enjoyed by new-media authors is the opportunity to realize representations of Self that are rich textual worlds in themselves and also to engage the wider world, with a voice, a smile,
Beyond elocution: Multimodal narrative discourse analysis of L2 storytelling
Abstract Drawing upon research on narrative and speech styles and on digital and multimodal communication, the author proposes multimodal narrative discourse analysis (MNDA) with associated
The intersection of multimodality and critical perspective: Multimodality as subversion
The findings reveal that, on the one hand, the students were able to identify the power relations between characters as well as how particular characters are marginalized in the media texts they viewed by paying attention to the multiple modes that make up the text’s representation.
mStories: exploring semiotics and praxis of user-generated mobile stories
A semiotic analysis found that the user-generated content demonstrated complex and sophisticated multimodal sense relations, but control over the textual or compositional meta-function of the text was determined largely by the computer interface, with users habituated to relinquishing authorial control over this element.
Developing Voice in Digital Storytelling Through Creativity, Narrative and Multimodality
For Simon, who is nine years old, reading and writing are obstacles rather than tools. However, when offered the opportunity to create digital stories, Simon becomes deeply engaged. He creates
The sliding of the signified: multimodal sign operations in a youth-created experimental digital video
This study offers conceptualizations of multimodal discursive units that can be used to interpret and analyze the interactions of visual and spoken signifiers in films and digital videos.
Drawing bodies and spaces in telecollaboration: a view of research potential in synaesthesia and multimodality, from the outside
While much scholarship on the multisensory and transmodal phenomenon of synaesthesia seeks to uncover its psychophysiological and neurological bases, recent research in multimodal literacy and


Expanding the web of meaning: Thought and emotion in an intergenerational reading and writing program
This study explores informants' experiences in a reading/writing program that paired eighth-grade language arts students with elderly volunteers. Twenty-three pairs were followed over an academic
Discourses of Power, The Dialectics of Understanding, The Power of Literacy *
The study of literacy has recently been made a major focus of attention in socio-, psycho-, and anthropological linguistics. Our understanding of how literacy affects both consciousness and social
Travels in hypermodality
A semiotic scheme for the analysis of composite verbal-visual meanings and some discussions of the semiotic politics of visual communication in hypermedia design are offered.
A pedagogy of Multiliteracies Designing Social Futures
THE NEW LONDON GROUP 1 In this article, the New London Group presents a theoretical overoiew of the connec­ tions between the changing social environment facing students and teachers and a new
The Social construction of literacy
This book presents a social perspective on literacy and on its acquisition. The term 'literacy' is notoriously vague, and in the opening chapter of this collection, Jenny Cook-Gumperz examines the
Forms of talk
Forms of Talk extends Erving Goffman's interactional analyses of face-to-face communication to ordinary conversations and vebal exchanges. In this, his most sociolinguistic work, Goffman relates to
On the relationship between old and new technologies
Literacy in the New Media Age
The futures of literacy: modes, logics and affordances, Literacy and multimodality: a theoretical framework, and some items for an agenda of further thinking.
Understanding Media: The Extensions of Man
This reissue of Understanding Media marks the thirtieth anniversary (1964-1994) of Marshall McLuhan's classic expose on the state of the then emerging phenomenon of mass media. Terms and phrases such
Handbook of Literacy and Technology : Transformations in A Post-typographic World
This chapter discusses the transformation of Literacy in a Post-Typographic World through the design of new technologies for Literacy and Learning in Teachers' Instructional Needs and the role of technology in this transformation.