“Comic Books: An Evolving Multimodal Literacy”

by Taylor Quimby | Xchanges 7.1

2. The Speech Balloon, the Panel, and Multimodality

Have you ever tried to describe a funny comic strip to a friend, only to discover that the joke loses its humor when read aloud? In order to convey the meaning of the strip, you cannot simply read the speech balloons, you also have to describe the characters and relate the peripheral action. These log-jammed retellings tend to ruin the pace of the strip.

The reason comics don’t translate verbally has everything to do with how they communicate information. You don’t just read the words; you also read the image. James Paul Gee uses textbooks as an example of how word/image texts are more complicated than we imagine. In many modern textbooks, images not only take up more space, they carry meanings that are independent of the words in the text. If you can’t read the images, you will not be able to recover their meanings from the words in the text alone. In such multimodal texts (texts that mix words and images), the images often communicate different things from the words, and the combination of the two modes communicates things that neither of the modes does separately (Gee 17-18). When a comic inserts dialogue (or any other symbolic language) into its visual narrative, it becomes a multimodal text.

Let’s look at an example (Fig. 1). In this strip from Calvin and Hobbes, Calvin complains about New Year’s resolutions to Hobbes while they are sledding. The humor is not merely physical; it’s multimodal. Calvin is seemingly unaware of his own fragile physicality, as we can see by the unconcerned expression on his face before they crash into the tree. Hobbes, on the other hand, is rooted to the action. As they pick themselves up from the frigid river in the last panel, Hobbes implies that perhaps Calvin should resolve to watch where he is going, but Calvin unequivocally rejects the notion that he has any faults.

Calvin’s refusal to take part in the visual world around him prevents him from learning from his own philosophical discussion. Calvin’s comedic hypocrisy in this strip has everything to do with the relationship between the visual and the linguistic. In the Calvin and Hobbes 10th Anniversary Collection, Bill Watterson describes how the sled is a simple device to add physical comedy to the strip, most often used when Calvin gets longwinded or philosophical — the action lends a silly counterpoint to the text, and sometimes the wagon ride even acts as a visual metaphor for Calvin’s topic of discussion (104). Notice how Watterson first acknowledges the individual elements of the strip, but then describes how meaning is formed in the relationship between them. This relationship is part of what Gee calls an internal design grammar, or “the principles and patterns in terms of which one can recognize what is and what is not acceptable or typical content in a semiotic domain” (28). A semiotic domain, Gee explains, is “an area or set of activities where people think, act, and value in certain ways” (19). In this case the semiotic domain is comic books, and the visual/linguistic relationship is part of its unique design grammar. There are other multimodal relationships at work, too.