16 English Journal 105.5 (2016): 16–22

S tudents are sprawled across the classroom, all seemingly absorbed in the work of creating a multi-modal text: importing videos, re-

mixing sound clips, manipulating photographs, composing text, creating new graphics, and adding transitions. Except for Terrell. He sits in front of a sight familiar to every student, writer, artist, and English teacher: the blank screen.

“How are you doing, Terrell?” I ask. “Is there anything I can help you with?”

He shakes his head. “I thought this story stuff would be, like, easier than writing. But there’s all this stuff I gotta put in— like music and my pic-tures and then I gotta make sure that it’s not too short or too long or boring, so that people want to watch it. It’s a lot, Ms. J. A lot. But I got this. Don’t worry.”

Terrell eventually did “get it,” and his obser-vation about the difficulties of creating multimodal texts was spot- on. As students compose with a va-riety of different modes, such as images, sound, music, and writing, they have to think deeply about the ways in which meaning is constructed, con-veyed, and eventually interpreted by an audience. This article documents my attempts to explore the possibilities (and complexities) of multimodal meaning- making in the digital dimension, using

examples from the Neighborhood Stories Project, in which a group of fifth- grade students and I inves-tigated what it means to tell a multimodal story. As part of an extended summer workshop on literacy and multimodal composition, we “read” a series of visual, musical, digital, and multimedia texts, just as we read poems and stories, about the creators’ experiences of living in a particular time and place. Then, students took cameras into their homes and communities to record photographs and video foot-age that told their own stories of living, playing, and growing up within their neighborhoods.

At the heart of our work was the idea that composing in a digital, multimodal landscape can present “not just a new way to make meaning, but a different kind of meaning” (Hull and Nelson 225). As students created their neighborhood stories— occasionally with words, but more often with pho-tographs, video, sound, music, color, special effects, visual effects, writing, and live acting— they de-veloped critical understandings of how symbols and modes come together to make meaning. But even more importantly, as they worked within and across modalities, they became “more proficient at representing their ideological beliefs” (Albers 7). In other words, as students, like Terrell, became multimodal artists, as they pushed back against more traditional forms of representation, they also

This piece explores the possibilities (and complexities) of multimodal composition, using examples from a project in which fifth-grade students used digital media to create unexpected stories about their communities.

Telling Unexpected Stories: Students as Multimodal Artists

Robin Jocius

The role of imagination is not to resolve, not to point the way, not to improve. It is to awaken, to disclose the ordinarily unseen, unheard, and unexpected.

—Maxine Greene, Releasing the Imagination: Essays on Education, the Arts, and Social Change

EJ_May_2016_B.indd 16 4/23/16 4:45 PM

17English Journal

Robin Jocius

characterization, theme, and text structure. En-couraged by the success of the students and the project, I built more and more digital, multimodal projects into the curriculum. It was years before I realized that we were only scratching the surface of multimodal possibility.

Principles for Designing Transformative Multimodal Learning Experiences

In many cases, multimodal, digital tools have been used primarily to engage and motivate students, to give them opportunities to cross the boundaries be-tween their lives in and outside of the classroom. While this is a worthy goal, it isn’t ambitious enough, not when the interactive characteristics of digital media can change both the form and function of academic literacy experiences. A switch in the form of an academic project (e.g., students create digital “family portraits” in place of writing a characteriza-tion essay) involves shift-ing the medium students are using to create (digital composing software instead of paper and pencil). This offers opportunities for readers and writers with di-verse needs: students with strengths in alternative forms of representation can capitalize on the digi-tal skills they developed in nonschool contexts to experience academic success. However, while proj-ects that represent changes in form alone ask stu-dents to translate their learning across modalities, they don’t maximize the full interactive potential of digital tools. Likewise, a change in the function of an academic project (e.g., asking students to pro-vide feedback on their peers’ characterization es-says using a digital message board) capitalizes on new forums for student communication. However, changes in function alone fail to take advantage of the semiotic materials that digital media have af-forded to multimodal composers.

Ideally, all multimodal classroom activities would represent shifts in both form and function. For example, students could collaboratively cre-ate argumentative videos that incorporate images, music, special effects, acting, movement, color, and

developed the ability— and confidence— to tell sto-ries often unseen and unheard.

An Initial Multimodal Experiment

Researchers have suggested that digital tools, such as cameras and iPads, can be used to motivate young readers and writers who, for one reason or another, have become disengaged from classroom literacy practices. As Donna Alvermann argues, “adolescents who appear most ‘at risk’ of failure in the academic literacy arena are sometimes the most adept at (and interested in) understanding how media texts work, and in particular, how meaning gets produced and consumed” (200). So, for stu-dents, teachers, and schools, new and digital media have presented new possibilities to collaborate, create, and share information. For example, social forums for reading and writing, such as Twitter, blogs, YouTube channels, fanfiction sites, and other online forums, have enabled the often instantaneous dissemination of information across school and nonschool spaces.

Almost ten years ago, as a classroom teacher working with “at- risk” adolescent readers, I de-cided to test Alvermann’s hypothesis: Would the introduction of multimodal texts and composing activities engage my disengaged students? Would students, many of whom had grown accustomed to academic failure, have the chance to experience literacy success? As we explored how music, color, sound, and images convey meaning, would we find new avenues to critique and interpret texts?

In some ways, the multimodal experiment was successful. During one of the earliest proj-ects, in which students produced visual interpre-tations of unimodal (print) texts, one of my most disengaged students crafted an intricate family portrait, spending hours and hours reading and rereading passages from The Outsiders to ensure that his computer- assisted drawing accounted for the most precise details within the text. Another group painstakingly re- created battle scenes from Animal Farm, complete with faux gunpowder and realistic pig costumes. In a critique session, as stu-dent creators proudly shared their work with their peers, I saw these “at- risk” students ask powerful questions, challenge interpretations, and produce sophisticated analyses of literary elements such as

Would the introduction

of multimodal texts and

composing activities

engage my disengaged

students? Would

students have the chance

to experience literacy

success?

EJ_May_2016_B.indd 17 4/23/16 4:45 PM

Telling Unexpected Stories: Students as Multimodal Artists

18 May 2016

writing. Then, using a tool such as VoiceThread, students could provide synchronous, annotated feedback to their peers. VoiceThread is an interac-tive platform that allows teachers and students to post videos, images, pieces of text, PowerPoint pre-sentations, and other digital artifacts to a common forum. Users can then work with each other and the artifact as they create and share their video, audio, written, visual, and annotated responses. In this case, digital, multimodal composers have the op-portunity to share and discuss their work, employ different signs and combinations to craft meaning, and collaborate with others across space and place.

Digital and multimodal pedagogies also pro-vide new avenues for students to critically exam-ine how texts can serve to reinforce or negate social concerns. For example, I recently observed a group of middle school students who constructed a social network using Google Classroom. Once they had developed the criteria for social interactions within the digital space, the students selected characters from different novels (The Giver, Number the Stars, and Wonder), created multimodal representations of their characters, and then interacted with each other as “friends” or “enemies.” As the students embod-ied their chosen characters, they began to discuss

overarching social and political issues in this set of texts (i.e., social justice, activism, and the ways in which institutions can silence individual voices).

As readers and writers navigate multiple modes in digital contexts, they also “gain sub-stantively in meta- cognitive and meta- linguistic abilities and in their ability to reflect critically on complex systems and their interactions” (New Lon-don Group 69). Throughout the Neighborhood Stories Project, I challenged my students (and my-self) to create multimodal interactions that were truly transformative, that went beyond replicating a written activity in a different modality, and that represented shifts in both form and function. To meet that challenge, I developed a set of three prin-ciples of transformative multimodal practices that guided my instructional design. With the princi-ples, I include a set of guiding questions, as well as instructional considerations (see Figure 1).

Multimodal Analysis: Students as Text Critics

As Stephen King reminds us, “If you want to be a writer, you must do two things above all others: read a lot and write a lot” (145). It follows, then,

FIgURE 1. Principles of Transformative Multimodal Practices

Principle guiding Questions Instructional Considerations

1. Digital, multimodal practices should present a hybrid model of print and digital learning, individual and collaborative understandings, and personal and critical meaning- making.

• Does this practice include path-ways for students to develop skills in print- based and digital contexts?

• Does it provide opportunities for students to develop skills in inter-pretation, analysis, and criticism that they need to navigate the 21st- century world?

• Provide opportunities for students to move among print, digital, and new media

• Encourage personal and critical responses to texts; students can and should question the authority of the author and imagine the text from an alternative point of view

2. Instructional practices involving personal and critical responses to multimodal texts need to account for the multiple and multifaceted identities that ado-lescents bring to the classrooms (as students, as readers, as com-posers, as critics, and so on).

• Does this practice recognize and honor students’ personal and cul-tural identities, as well as their identities as students?

• Does it allow students to move beyond individual and personal responses to critique social systems and representations?

• Choose texts carefully; offer mirrors, which reflect the cultural norms and values of the reader, and windows, which juxtapose the familiar with the unfamiliar (Bishop ix)

• Research texts, authors, and social contexts to choose texts that reflect students’ cultural, racial, socioeco-nomic, and linguistic backgrounds

3. Multimodal pedagogies and classroom practices should strive to represent shifts in both form and function.

• Does this practice represent both a change in form and function?

• Does this practice allow students to move in and among sign systems to construct and interpret meaning?

• Present opportunities for students to use the digital knowledge and exper-tise developed in nonschool contexts

• Maximize the potential of digital tools by having students share their compositions with distributed online audiences and forums

EJ_May_2016_B.indd 18 4/23/16 4:45 PM

19English Journal

Robin Jocius

that to become a multimodal artist, you need to “read” and write a variety of multimodal texts. The analysis of multimodal texts involves developing a comprehensive knowledge of many of the same con-ventions as print texts— to interpret, for example, students must often examine setting, theme, char-acter, climax, and resolution. However, and perhaps more importantly, multimodal textual analysis can push students toward the fundamental knowledge that every text— photographs, poems, paintings, films, and music alike— tells a story from a particu-lar perspective and for a particular purpose.

Texts don’t simply come into being, absent of particular ideas or perspectives; they are ideologi-cal, emerging from the collective personal experi-ences of the creator. While a work of art is often a vehicle for emotional expression, it is also a reflec-tion of the social world in which the creator lived. To “read” a painting, for instance, students must explore how color and line come together to form the composition, but they must also ask critical questions about the cultural and historical context in which that particular painting was created.

During the Neighborhood Stories Project, my students and I read a series of multimodal texts in which the creators told stories about their neigh-borhoods. As we analyzed poems, paintings, sculp-tures, videos, prose, and multimodal news stories, students began asking questions about content— “Why did he do orange faces? That doesn’t make sense,” and “Why are there so many people in here?” As the teacher, my role was to guide students to ask questions about the content of each text, but I also wanted to push them to delve deeply into particu-lar cultural and historical contexts. As we watched news stories and read editorials about violence within the neighborhood in which my students lived, for example, we researched the journalists and authors and discussed how their backgrounds might have shaped the stories they told. I tried to abide by Maxine Greene’s words: “For teachers, the obligation is to teach persons how to notice what is to be noticed without imposing alien readings or interpretations” (“The Art” 133).

For example, one of the last multimodal texts that we read as a group was Palmer Hayden’s “A Midsummer Night in Harlem” (1936), which de-picts a Black community in Harlem through vivid colors, a lively composition filled with dozens of

small groups of people, and an exaggerated paint-ing style. Hayden, an African American artist, has been a controversial figure in art criticism, generat-ing heated discussion around whether his portraits perpetuate or resist dominant stereotypes (Wolf-skill 343). My fifth- grade students engaged in a similar debate; when Terrell, who had become an active and engaged multimodal “expert,” suggested that Hayden might have been “trying to say some-thing” by presenting his subjects in caricature, an-other student, Eric, vehemently disagreed, arguing that the figures looked “racist.”

After students learned more about Hayden’s race and background, the debate continued— with arguments for both sides. Finally, a student who had been mostly silent throughout the discussion posed yet another alternative: “He’s just not very good at painting.” We concluded the discussion by agreeing to disagree; in other words, and as Terrell pointed out, “It’s complicated.” The takeaway for students was that knowing more about the context in which a text was created, as well as personal de-tails about the creator (in this case, Hayden’s race, education, and statements about art), influenced the way in which they came to “read” texts. As stu-dents began to craft their Neighborhood Stories, they began to believe that they, too, had stories to tell— that their experiences could contribute valu-able perspectives.

Students as Multimodal Artists

At the beginning of the Neighborhood Stories Proj-ect, students were given digital cameras to record audio and video footage in their homes and neigh-borhoods. In some cases, the types of photographs and video that students captured were closely connected to the texts we studied as a group; for example, after we “read” the Palmer Hayden paint-ing, several students, including Terrell, elected to document groups of people sitting outside within their own communities. In other cases, though, stu-dents’ choices in terms of subjects and the ways in which their subjects were portrayed bore little re-semblance to the other neighborhood stories that we had studied.

After we spent several days analyzing other creators’ multimodal compositions as a group, I asked students to collect their photographs, videos,

EJ_May_2016_B.indd 19 4/23/16 4:45 PM

Telling Unexpected Stories: Students as Multimodal Artists

20 May 2016

and artifacts in preparation for composing. By this point in the larger workshop, students had spent more than a month working with a variety of mul-timodal composition tools (PowerPoint, Movie-Maker, Audacity, image manipulation software) to compose a series of projects, such as “All About Me” digital videos, PowerPoint collages document-ing “found objects,” and “music videos” featuring onomatopoeia.

Students’ familiarity with various software tools allowed for a flexible composing structure— scaffolding, models, explicit step- by- step instruc-tion, which were an integral part of students’ initial attempts to compose multimodally, were replaced by more opportunities to weave in between and among modes in ways that felt most natural to stu-dents. Like writing, multimodal composition does not proceed in a linear path (Krashen 17), and as students become more experienced multimodal composers, they need the chance to translate, trans-form, and transduce modes. While some students chose to begin their compositions by laying down music tracks, others engaged in a lengthy planning process, and still others began by focusing first on the visual presentation of their ideas.

As students became more experienced in rep-resentation using multimodal, digital tools, they also developed valuable skills in communicating their cultural, social, and political beliefs. When we had viewed the news stories about the neigh-borhood in which students lived, many students had expressed disagreement about the represen-tation of their community. Students were eager to tell their own stories— about their families, friends, and places that were most familiar. In fact, many students’ Neighborhood Stories ultimately presented counternarratives— texts that examine, critique, and counter more dominant narratives (Solorzano and Yosso 23)— which contradicted commonly held stereotypes about the urban area in which they lived.

For example, Eric, a Black male who wanted to become an ophthalmologist, focused on the dif-ferences between interior and exteriors. In the third frame of his Neighborhood Story (see Figure 2), he juxtaposed images from the fence outside the hous-ing projects in his neighborhood with a filtered ver-sion of a chessboard within his home. “You don’t see the inside,” he told the audience as he presented his video. “There’s the fence and the stoop,” and it’s

FIgURE 2. Screenshot from Eric’s Neighborhood Poem

EJ_May_2016_B.indd 20 4/23/16 4:45 PM

21English Journal

Robin Jocius

“dirty and nasty, and that’s all everyone thinks it is.” As Eric argued, the image representing chess, a game that he was just learning to play, showed that his home was something more than its exterior, and that despite all of the negative perceptions, there was indeed something “beautiful” about his neigh-borhood. Terrell, who had vigorously nodded his head as Eric spoke to the group, chimed in, “He has it right, with the pictures and the sounds. That be how it is— how it feels to be on the grass. To live there. It ugly sometimes, but sometimes not— sometimes everything is just . . . right.”

The Neighborhood Stories Project allowed students to consider how a text is created, how per-spectives are represented using a variety of modes, and how digital and multimodal tools might allow them to tell their own stories of unseen and unheard spaces. As time passes, students will surely forget much of what we accomplished together— middle school will fill their heads with new and essential knowledge, even as technologies such as PowerPoint and MovieMaker disappear and reappear, rendering many of their technology- specific skills obsolete. However, months after the project had been com-pleted, when I saw Terrell— still holding his cam-era, still photographing the world around him, and still creating stories that are not “too short or too long or boring”— I did hope that, as Greene puts it, my students would continue “to call, to say, to sing, and— using their imaginations, tapping their courage— to transform” (Releasing 198).

Reenvisioning the English Language Arts Classroom for the Digital Age

Ideally, all English language arts classrooms would produce spaces in which all students— struggling and thriving, members of nondominant and dom-inant cultural groups— engage meaningfully in the composition of texts of all forms and functions. Many have argued that digital tools make this type of pedagogical reimagining possible— that existing sites of education can be transformed to be “more inclusive of diverse literate identities and practices” (Vasudevan and Campano 331). However, despite emerging technological innovations and the in-creasing availability of digital tools within class-room spaces, print- based texts and well- established curricular practices (often, reading, writing, and

interpreting these texts) continue to dominate many classrooms.

This is perhaps not surprising; as long as we live in a world where success is determined by per-centile ranks and categorization, where education is inherently unequal, the enactment of digital and multimodal practices in schools will be circum-scribed by the availability of technological resources and the demands of preparing students to pass the test. However, if we view multimodal composing as a socially situated process that builds literacy skills in textual analysis, interpretation, and critical thinking, rather than a distraction that takes time away from test preparation, new opportunities for students to create and contest meaning arise.

For example, students like Terrell need more opportunities to use multiple semiotic resources in a collaborative, recursive composing process. As students juxtapose many different modes, they are forced to think about how these symbols work with, and possibly against, each other in a particu-lar representation. Also, giving readers and writers the tools to delve deeply into texts that directly re-late to their personal experiences and the cultural conflicts in their worlds may open their eyes to the ways in which texts position and reposition their identities and cultures, a key skill in our 21st- century world. Finally, allowing these students to bring their unschooled and often unsanctioned literacy practices into academic settings creates a culture in which unseen and unheard stories (from unexpected creators) can be told.

Works Cited

Albers, Peggy M. “Art Education and the Possibility of Social Change.” Art Education 52.4 (1999): 6– 11. Print.

Alvermann, Donna. “Effective Literacy Instruction for Ado-lescents.” Journal of Literacy Research 34.2 (2002): 189– 208. Print.

Bishop, Rudine Sims. “Mirrors, Windows, and Sliding Glass Doors.” Perspectives: Choosing and Using Books for the Classroom 6.3 (1990): ix–xi. Print.

Greene, Maxine. “The Art of Being Present: Educating for Aesthetic Encounters.” Journal of Education 166.2 (1984): 123– 35. Print.

— — — . Releasing the Imagination: Essays on Education, the Arts, and Social Change. San Francisco: Jossey, 1995. Print.

Hull, Glynda A., and Mark Evan Nelson. “Locating the Semiotic Power of Multimodality.” Written Communi-cation 22.2 (2005): 224– 61. Print.

King, Stephen. On Writing. New York: Simon, 2002. Print.

EJ_May_2016_B.indd 21 4/23/16 4:45 PM

Telling Unexpected Stories: Students as Multimodal Artists

22 May 2016

Krashen, Stephen. Writing, Research, Theory, and Applica-tions. New York: Prentice, 1984. Print.

New London Group. “A Pedagogy of Multiliteracies: Designing Social Futures.” Harvard Educational Review 66.1 (1996): 60– 92. Print.

Solorzano, Daniel G., and Tara J. Yosso. “Critical Race Methodology: Counter- Storytelling as an Analytical Framework for Education Research.” Qualitative Inquiry 8.1 (2002): 23– 44. Print.

Vasudevan, Lalitha, and Gerald Campano. “The Social Pro-duction of Adolescent Risk and the Promise of Ado-lescent Literacies.” Review of Research in Education 33.1 (2009): 310– 53. Print.

Wolfskill, Phoebe. “Caricature and the New Negro in the Work of Archibald Motley Jr. and Palmer Hayden.” The Art Bulletin 91.3 (2009): 343– 65. Print.

Robin Jocius (rjocius@citadel.edu), who first joined NCTE in 2007, is an assistant professor of literacy education at The Cita-del; her teaching and research interests include adolescent literacy, digital media, culturally diverse students, critical literacy, and multimodal composition.

READWRITETHINK CONNECTION Lisa Storm Fink, RWT

In this unit, students write autobiographies, illustrate them, and set them to music. Music is a powerful tool to evoke emotion, and students will carefully select songs to accompany the stories from their lives. Students brain-storm lists of important events in their lives, along with images and music that represent those events. They then create storyboards in preparation for the final multimodal project. After making revisions, they present their final projects to their peers in class. http://bit.ly/1HaEaN5

Close Reading

I heard a fly buzz inthe Athenian woodsof Hoffman’s midsummernight’s dream.

While studentsworked silently to note the lighting, the transitions, the camera’s perspective,I wondered if that flystill lived, and if in hisbrevity he pondered theeternity he achieved because he happenedto interpose himselfbetween Bottom andthe lens

—or, maybe hehad been trained to buzz, tocross the stage, released tofret his brief immortal flight,well rehearsed for his exit backstage (inside the crown);an ass, methinks, to toil sowhen simpleness andduty are the comicrelief writ in the textof the world.

Nota bene,my little players, to laugh to battle away the tortuous hours! Your play is preferred!

But—do not close your books;rather, hold them close—so close that your heart hears them—

—Sally Ventura© 2016 by Sally Ventura

Sally Ventura (sventura@sbu.edu) is an English teacher at Olean High School and an adjunct professor at St. Bonaventure University. She serves on the New York State English Council board and has recently published essays in English Record and College English Association Critic. She has been a member of NCTE since 2005.

EJ_May_2016_B.indd 22 4/23/16 4:45 PM

Reproduced with permission of the copyright owner. Further reproduction prohibited withoutpermission.