Multimodality in human interaction

Abstract

Human action is built by actively combining materials with intrinsically different properties into situated contextual configurations where they can mutually elaborate each other to create a whole that is both different from, and greater than, any of its constitutive parts. This has a range of consequences for the organization of language, action, knowledge and embodiment in situated interaction. Two phenomena that depend upon such distributed organization of action will be investigated here. First, Chil, a man who suffered severe damage to the left hemisphere of his brain that left him with a three word vocabulary, Yes, No and And, was nonetheless able to act as a powerful speaker in conversation. He did this by operating on the talk of others to lead them to produce the words he needed but couldn’t say himself, and also by using gesture to incorporate meaningful phenomena in this surrounding environment into the organization of his utterance. Second, the processes through which archaeologists acquire the ability to see relevant structure in the dirt they are excavating, and construct the documents, such as maps, that animate the discourse of their profession are investigated. The way in which action is built through the simultaneous use of materials with diverse properties makes it possible for experienced archaeologists to calibrate the professional vision, practice and embodied knowledge of novices, and thus to interactively construct within situated interaction the cognition, ways of seeing and embodied practices of new archaeologists. Both Chil’s ability to act as a speaker and the social organization of the embodied knowledge and perception required to act as a member of a scientific community are made possible through the way in which alternatively placed social actors contribute with different kinds of materials to a common course of action.

Key words: multimodality, aphasia, professional vision, gesture, pointing, talk-in-interaction.