Multimodal Discourse Analysis for teaching English as a Second Language

New technological developments have boosted the use of different modes or semiotic resources; social changes and developments, on the other hand, have changed the process of meaning making because discourse shapes and is shaped by social practices. Semiotic resources are used in communication (language, sound, gestures, facial expressions ... etc) and this has impact and reflections on the methods of teaching. Literacy is not only about reading and writing, it rather means the ability to communicate through multiple modes. Hence, it is important to embed multimodality (the study of using multiple modes) in educational settings.

stays for a longer term. Gilakjani et al (2011) assert that "a multidisciplinary approach is needed to understand the social, cognitive, cultural and linguistic variables involved in the process of language learning". They also conclude that language learners can adapt with the multimodal medium of communication if the designers focus on the meaning making potentials of all the modes used. Each of the various presentation modes appeals to students' different sensory modalities.
Eva M. Mestre-Mestre (2015) studied communication in the particular setting of the second language classroom, and two different modes in combination were analysed in order to describe the types of intersemiotic resources used by the students. The representation of two dissimilar semantic groups were chosen for analysis. The results show that students chose different strategies combining texts and images to construct their meaning, depending on the type of concept they were trying to communicate. In many cases, images are only used to support the text, whereas in other cases most images and texts perform the same functions and confirm each other.
Marchetti and Cullen (2016) shed light on a multimodal approach in the classroom as a "creative" domain for learners and teachers. They argued that a multimodal approach can increase the interaction and engagement between the teacher, students, materials, and topics. They concluded that "students perceived enhanced learning experience through the association of images and external audio to spoken interaction" (p.47).
Grapin (2018) compared the weak and strong version of multimodality as they have different perspectives on language. He concluded that "in the weak version, nonlinguistic modes are scaffolds need to be removed once ELs develop proficiency with oral and written language. In the strong version, multiple modes are the semiotic resources necessary for engaging in disciplinary practsies and are especially beneficial to ELs" (p. 20). he also states that students (need to) consider the limitations and affordances of all modes and what to include and exclude from their design.

Multimodal approach to communication
Communication has always been multimodal in nature as different semiotic resources are resorted to in meaning making (meaning production, transference and interpretation). The writings/drawings on cave walls and clay tablets which, at least some of them, date back to eternity are evidence that communication has always been multimodal in a sense.
For instance, when people speak, they use words, intonation, gestures, facial expressions and other modes of communication to cross over their intended message. The addressee needs to depend on all the sign making assemble to interpret the message. The sign maker (speaker) chooses the modes or semiotic resources according to his/her interest (Halliday, 1978;Kress, 2010); but s/he needs to take the addressee into account. In other words, the speaker has to use semiotic resources that are recognized and can be interpreted by the addressee to the best of his/her knowledge. Multimodality focuses on the choices people make amongst a group of available modes and the socio-cultural reflections and effects of these choices on meaning production and interpretation.
In recent years, communication has shifted drastically due to technological advancements and globalization. The contemporary usages of abbreviations in texting and emojis in texts and internet communication sites (Facebook, Messenger, Instigram, Twitter … etc) change the way we perceive communication. Language is no longer the sole bearer of information and knowledge. Visual, aural, spatial and gestural aspects affect how meaning is made and perceived.
Also, the forces of globalization and manifest local diversity increasingly juxtaposed modes of meaning making that were sharply different from each other. The challenge of learning to communicate in this new environment was to navigate the differences, rather than to learn to communicate in the same ways (Cope and Kalatzis, 2015, p.2). Wong and May (2019) drew the effects of social context in meaning making and their data were mostly visual to shed light on the importance of visual mode in different discourses like advertisements, postage stamps and diary keeping.
There are three assumptions which act as prerequisites to communication. The first assumption is concerned with the mediums and systems used for articulating meaning. The second assumption regards communication as multi-faceted that requires multiple modes operate together and this co-operation among the modes makes meaning despite the meaning of each of the modes. The third assumption is about the instability of the meaning potentials of modes and the derivation and exclusion of meaning potentials in different communicative acts and settings (Kress et al,p.43). In other words, communication is carried out through the use of meaning making systems and modes that operate together and simultaneously, they acquire and lose meaning potentials in different social contexts.
In the Sausserean schema, two interlocutors are linked in a dyadic structure. One initiates a message, the diagram and the theory both suggest that it originates from within the interlocutor's "head"; there it is shaped into speech, seemingly; it is uttered; the other participant receives this (spoken) message; and in that interlocutor's 'head' it becomes the basis of a response (Kress, 2010, p.33-34).
The aim of multimodality is to use different multimedia to represent the content or information to appeal to different learning preferences that students with different learning styles can enjoy. Communication happens between participants through a medium which is usually a language, in a socio-cultural spectrum. However, language is no longer the sole medium for or the only carrier of meaning in communication. Multiple modes are and can be put to use in meaning making process. The modes function simultaneously; each is having its own role and function in meaning production and interpretation.
Hyland and Paltridge (2012) state that "the meaning potential of words, sounds, and images are sets of interrelated systems and structures" (p.122). In other words, when multiple semiotic resources are used for meaning making, each adds up to the overall meaning of the discourse and the meaning potentials of each mode are dependent on each other.
Multimodality challenges the predominance of language as the primary means of communication. It entails that communication happens across multiple modes, like visual, spatial, aural, gestural … aspects of the communication context (Jewitt, 2013;Kress, 2010). Language (spoken and written) are studied in multimodal discourse analysis approach to communication, but it is analyzed within the framework of the multimodal ensemble encompassing all the other modes that accompany it. The modes are of equal status and important for both the meaning producer and interpreter(s).
In multimodal theory, Kress and Jewitt (2003) identify four aspects that compromise one's representation of meaning. The four aspects are materiality (the materials and resources used), framing (the way in which elements of a visual composition operate together), design (how people make use of the materials and resources that are available to them at a particular moment to create their representation), and production (the creation and organization of the representation).
Carey Jewitt (2009) proposes four principles or assumptions that encompass multimodal discourse analysis. The first assumption considers language as one component in an ensemble of equally important modes. Second, the modes complement each other and each one provides a part of the overall meaning. The third assumes people have various modes available to them and they utilize the selected modes to configure meaning. The fourth and final assumption considers multimodal discourses social (Paltridge, 2012, p.171).
Multimodality (production and interpretation) stems from semiotics which is, roughly, defined as "the study of signs" (Crystal, 2008) and signs are the carriers of meaning encompassing the signifier (the word) and the signified (the entity or object referred to by the word). Communicators or social actors need to abide to fixed sets of principles for a successful communication with one another in a social context. Gunther Kress (2010, p.10) proposes three most important principles of sign making, which are: 1. Signs are motivated conjunctions of form and meaning 2. That conjunction is based on the interest of the sign maker 3. Using culturally available resources. Furthermore, communication and learning happen across various modes. Thus, all the modes should be tackled and involved in the process of teaching. Nowadays, due to technological advancement, literacy has shifted gears and language per se is no longer the sole requirement and criterion for being a literate person. Words on pages and images on screen create and require different reading paths and affordances. Multimodal discourse analysis can motivate students to get engaged more in class and might reduce the risk of forgetting information by students due to its use of various senses.

Learning styles and multimodality:
People have individual and collective identities based on how they see themselves and their preferences, on one hand, and based on how people see them and identify them. Students have preferences and identities. The term ''learning styles'' refers to the way in which learners concentrate on, process, and retain new information (Dunn and Dunn, 1992;1993;1999). Teaching, by the same token, should be adapted to the different learning styles of students.
Teaching has to be adapted to ''the way in which each learner begins to concentrate on, process, absorb, and retain new and difficult information'' (Dunn & Dunn's framework; International Learning Styles Network, 2008), (b) the learner's preferred modes of perception and processing (Kolb's, 1984, 1985, framework), or (c) ''the fit between [people's] learning style and the kind of learning experience they face'' (Hay Group, n.d., p. 11).
Omrod (2008) wrote, ''Some cognitive styles and dispositions do seem to influence how and what students learn. . . . Some students seem to learn better when information is presented through words (verbal learners), whereas others seem to learn better when it's presented through pictures (visual learners)'' (p. 160, italics in original). Furthermore, "Factors that can influence learning styles include culture, school climate, expectations, teaching style and classroom practices" (Reid, 2005, p. 51).
Honey and Mumford (1986) proposed another model for the inventory of learning styles. They are Activists (prefer to learn by doing), Reflectors (observe and like to collect information before decision making), Theorists (work towards adding new learning into existing frameworks by questioning and assessing the possible ways that new information might fit into their existing frameworks of understanding), and Pragmatists (look for the practical implications of any new ideas or theories before making judgments on their value).
There are different models of learning style assessment, but they all support the idea that we do not learn in the same way. There are certain physiological, psychological, cultural and environmental elements that affect the way people acquire, process and retrieve information and data.

Multimodality and teaching/learning
Communication has been multimodal for long. However, what has changed is the sociological interaction at large, and these technological developments have challenged communication forms and mediums along with the status of language in interactive communication. Language is a psychological and social phenomenon that stems from the mind and is represented and reinforced through its social use. Undoubtedly, the technological developments have impacts on language production and interpretation which will, subsequently, alters ways of learning and teaching.
Language is widely taken to be the dominant mode of communication in learning and teaching. Image, gesture and action are generally considered illustrative supports to the 'real thing'. Our observation of teaching and learning in the science classroom casts doubt on this assumption (Kress,et al,p.42).
McLaughlin (1987) in his book (Theories of Second Language Learning) lays down five theories for Second Language Acquisition: the monitor model, interlanguage theory, linguistic universal, acculturation/pidginization theory, and cognitive theory.
Information and communication technology provide academics with an opportunity to create rich learning environments for their students, enhanced by the wealth of information and resources on the internet, as well as the inclusion of a range of multimedia-based learning elements.
Multimodal teaching is a style in which students learn material through a number of different sensory modalities. For example, a teacher will create a lesson in which students learn through auditory and visual methods, or visual and tactile methods. Teachers can use any combination of learning modalities; however, in multimodal teaching, a teacher must utilize more than one. This successful teaching style implements many strategies to ensure students understand and retain information.
The visual and verbal modes complement each other to realise an intersemiotically coherent multimodal text, I also suggest that the intersemiotic resources used to realise this complementarity can be readily explored for pedagogical purposes.
Multimodal learning environments allow instructional elements to be presented in more than one sensory mode (visual, aural, written). In turn, materials that are presented in a variety of presentation modes may lead learners to perceive that it is easier to learn and improve attention, thus leading to improved learning performance; in particular for lower-achieving students (Chen & Fu, 2003; Moreno & Mayer, 2007; Zywno 2003). Mayer (2003) contends that students learn more deeply from a combination of words and pictures than from words alone; known as the "multimedia effect". Further, Shah and Freedman (2003) discuss a number of benefits of using visualisations in learning environments, including: (1) promoting learning by providing an external representation of the information; (2) deeper processing of information; and (3) maintaining learner attention by making the information more attractive and motivating, hence making complex information easier to comprehend. Fadel (2008) found that, students engaged in learning that incorporates multimodal designs, on average, outperform students who learn using traditional approaches with single modes" (p. 13).
Communication through computer technology has increased the intermingling of text, audio, video, and images in meaning making to the point that Kress (2000) argues that it "is now impossible to make sense of texts, even of their linguistic parts alone, without having a clear idea of what these other features might be contributing to the meaning of a text" (p. 337). Elaborating on this assertion, Kress and van Leeuwen (1996) posit a comprehensive "grammar" of visual design and discuss the development of visual literacy and its educational implications.
Multimodal perspectives on teaching and learning build on the basic assumption that meanings are made (as well as distributed, interpreted, and remade) through many forms and resources of which language is but oneimage, gesture, gaze, body posture, sound, writing, music, speech, and so on (Kress & van Leeuwen, 2001;Jewitt, 2009).
The major benefit, as identified by Picciano (2009), is that "it allows students to experience learning in ways in which they are most comfortable, while challenging them to experience and learn in other ways as well" (p. 13). Consequently, students may become more self-directed, interacting with the various elements housed in these environments. So, depending upon their predominant learning style, students may self-select the learning object, or representation that best suits their modal preference (Doolittle, McNeill, Terry & Scheer, 2005). In other words, "different modes of instruction might be optimal for different people because different modes of presentation exploit the specific perceptual and cognitive strengths of different individuals" (Pashler et al. 2008, p. 109).
Such daily activities advocate the need for a transformation in the teaching and learning of ESL lessons in order to promote students' capabilities in making meaning of different literacy texts which students come across in their ESL learning activities. Thus, it is hypothesized that: 1. Students have physiological, psychological, environmental preferences that affect the way they perceive, process, and retrieve information. Some of the learning styles of students are highly neglected in class as they are the minority or may take a longer time to accommodate for by the teachers.
2. Time and lack of facilities are the two main obstacles of using different modes of representation by teachers. Accommodating for different learning styles requires preparation of slideshows and careful and thorough lesson planning that takes a long time. Furthermore, representation of multimedia requires some technological advancement that a lot of places (colleges and departments) cannot afford and are deprived from.

Methodology
To see how English language lecturers perceive the importance, utility and challenges of the other modes in their classes, a questionnaire was designed and distributed. Some respondents filled in the questionnaire in the Google Form. The others answered the printed form of the questionnaire. Besides the questionnaire, various classes were observed by the researchers to validate what is going on in class. In total, (22) teachers of College of Education/English Department from Salahaddin University-Erbil filled in the questionnaire. This questionnaire was distributed among the above mentioned teachers to find out their teaching preferences and styles.
Another questionnaire was designed based on Kolb's learning style assessment to identify the learning preferences of students. Third year students in English department at College of Education (Salahaddin University -Erbil) for the academic year (2019-2020) were taken as samples. The population was 76 participants. The samples were 69as 8 participants were not present at the time of administrating the survey. They were (24 male) and (45 female) students. Group interview was used with the students to identify their learning styles. First, the learning styles are explained for the students and then some questions were asked as leading questions. After that, the students were asked to identify the style they feel most comfortable with by raising their hands. The whole experience (explanation and group interview) took about one hour.

Results and discussion
Based on the questionnaire taken by English language teachers at English departments at Salahaddin University-Erbil, most of them are using different modes of representation in their classes. The use or not using PowerPoint slides in class may have to do with the nature of the class, though the creativity of the teachers can help them to use slides in every class to their advantage. Slides give a chance to teachers to save time and use different modes of representation.
Most teachers (76%) think that writing is more effective than images. Images require a different kind of literacy. People who can read and write in a particular language are not necessarily able to read images. Hence, teachers need to analyze images for students to give them a chance to see pictures as a whole of a set of separate components.
When we talk about multimodality, it is not only about using picture and videos. Some teachers (25%) highlight important concepts in their PowerPoint slides with bold (a word that appears darker than the rest) or with the use of a different color than the rest.
Most of the students (78%) considered themselves to be visual learners as they thought they enjoy learning through visual representation more. This has a lot of pedagogical implications and teachers need to take that into consideration and use more visual elements in their slides and classes like videos and pictures.
The study of meaning making as a social act that happens across a number of semiotic sources has vital implications for education in general and teaching in particular. The following pedagogical implications are by no means exhaustive: First, communication is all about choices and interests. Teachers, for example, have a set of choices (sources) available at their disposal from which they can get benefit for their teaching styles. One advantage of using different modes in class by teachers is the reduction of the difficulty that arises from the different learning styles that students have. For instance, visual learners prefer to learn through pictures and images while auditory learners enjoy listening to the teacher and audios.
Second, students are constantly encouraged to be autonomous which means to depend on themselves more. They can resort to any interpretation they deem suitable. As mentioned in the study, meaning making is not a static process, it is constantly changing. Hence, the creative thinking of students can be activated as the same mode might have a set of interpretations (choices). Each student can be motivated to interpret the topic in any way they find suitable. Then, they can compare their interpretations to see from different angles.
Third, curriculum designers can take advantage from social semiotic model of representation and communication and multimodality. Multimodality emphasize that communication happens across a multiple of modes. Each mode fulfills a limited portion of the overall message. Furthermore, texts and images are the most common modes used in educational textbooks. Hence, it is important for textbook designers to be familiar with the relations between images and texts, in particular. The dissertation has provided a set of options from which they can choose the relation they think can serve their objectives in the best possible way.
Fourth, as a first step, the analysis of images based on Kress and van Leuween's model of Reading Images (1996;2006) can be taught to students in communication classes. Then the integration to images and captions in social media posts can be analyzed as a relation between image and text. They extended Halliday's Systemic Functional grammar and social semiotics to account for the functions performed by images. Halliday (1978) proposed that language is only a means of communication and other systems can be used to "design interpersonal meaning, present the world in specific ways, and to realize coherence" (Jewitt, 2017, p.32). Like ideational, interpersonal and textual metafunctions of language, images can perform representational, interactive and compositional metafunctions. Furthermore, images can be analyzed into smaller components as a language chunk can be analyzed.
Fifth, Martinec and Salway's model (2005) of image/text relations is a good starting point to understand the relation better. Each text and image relation can have a status and a logico-semantic relation. The former refers to the status of the two modes, whether they are equal or one is subordinate and the other is superordinate. The latter, on the other hand, is about expansion or projection relations, which means they both present the same information or one expands the overall meaning of the other.

Conclusion
Literacy is no longer about the ability of writing and reading only. Communication through a multiple modes of representation like pictures, writing and videos is changing the way people look at literacy nowadays. Teachers and educators need to enable students to convey messages through and understand multimedia correctly. The bottom up processing of multimodal artifacts means each mode is analyzed in isolation alone then the relation and interaction between the accompanied modes convey meaning too. For example, writing can support, expand or project a picture in social media sites. Multimodality can address the issue of different learning preferences of students. Most students are visual learners while most teachers do not use pictures and videos in their classes whether it is because the use of different modes of representation is time consuming or because the module does not help the use of different modes.