Appendix A
The Multimodality of Communicative Events

In the semiotic task of revealing more precisely the place of the built environment—or any other system of signs—in communication, the analysis of communicative events in their multimodal totality has acquired today a fundamental urgency and importance. In the present section I would like to discuss the complexities inherent in such events as well as our current abilities to adequately model such complexities.

Communication, in the broadest sense, involves the transmission of information regarding the perception of similarities and differences. Any semiotic system is a complexly-ordered device for the cueing of such perceptions in given sensory channels and in conventionally-delimited media.

A communicative act such as a verbal utterance does not normally exist in vacuuo (except perhaps in the fictitious atmosphere of certain recently fashionable linguistic models); rather, speech acts are invariably co-occurrent with communicative acts in distinct signalling media. This state of affairs is neither accidental or circumstantial, for on the basis of internal evidence alone, it is increasingly evident that each of the isolable sign systems evolved by humans has been designed from the outset to function both semi-autonomously and in deictic concert with other sign systems.

But beyond an understanding of certain formative entities in the linguistic code—whose meaningfulness, as in the case of ‘shifters’, can only be disambiguated through cross-modal indexing—we remain at a serious loss to account for the extraordinarily complex systematicities of normal (i.e., multimodal) communication in daily life.

It has been clear for a long time that an adequate account of communicative events demands of us more than a mechanical summation of the organizational properties of particular codes as analytically isolable, and more than is currently offered by the hybrid heuristics of sociolinguistics and ‘pragmatics’, which, while admirable for their remarkable rediscovery of the wider world in which verbal language is embedded, nevertheless rarely escape an implicit verbocentrism. One can only stretch paralinguistics so far.

In the ongoing semiotic bricolage of daily life, we orchestrate and intercalate anything and everything at our disposal to create and maintain a significant world, or simply to get a message across. It is clear that the attempt to understand such complexities through the scientistic super-imposition of design features, analytic methods and even data language drawn from the study of one of its embedded components—for example verbal language—upon other significative modalities has, by and large, been a failure. While it is true that much has been learned by such activity, it must be decisively admitted that the ultimate expected illumination has tended to be rather dim and fleeting in comparison to the energies expended—or, as more often has happened, the mute stones and gestures have remained mute.

Of course this is not to deny the importance and relevance of a semiotics of the code—whether architectonic, gustatory, linguistic, or somatotopic; rather it becomes increasingly urgent to reaffirm the status of such models as selective, partial and synechdochal fictions.

If we are to augment the ongoing multiplication of semiotic models of specific codes in given cultural contexts beyond the trivial reductionisms of currently available ‘semiotics of culture’, our focus must be held tenaciously upon the actualities of semiosis in daily life which implicate and combine varieties of significative formations drawn from distinct signalling media. Moreover, general sign theory itself must push beyond the ultimate propositional logics which, perhaps not so curiously, seem to privilege the perspective (verbal or visual) of the given analyst.

None of this criticism is particularly new or original. It is raised again, and must be continued, until our picture of the extraordinarily complex nature of normal human semiosis begins to be clarified in a non-trivial fashion. Clearly, this is not intended as an indictment of the semiotic enterprise itself, rather only its dominant priorities. If semiotics is to realize its potential as a principled, insightful and radical contribution to the problem of meaning in human life, it must remain absolutely clear about the relative urgencies of its priorities.

Additionally, if semiotics is to be more than merely a new formalism, I think that we must be prepared to admit that it may be at least theoretically possible for semiotics to learn from the experience of other perspectives which in the past or at present have attempted to wrestle with equivalent problems. There is nothing more vacuous than a semiotics of art (for example) which is less well-informed or insightful than the received art history.

Again, I think these issues are self-evident, and rather than continue to persue them in the abstract, I would like to begin to address the question of the implications of a holistic and multimodal approach to semiosis.

It is clear that communicative acts in a given medium are normally co-occurrent with acts in other modalities which may or may not implicate or address distinct sensory channels, and I think it is also clear that communicative events resemble complex, dynamically equilibrated spatiotemporal arrays of such acts, of which the basic primate display is a simple, but (in the human line) radically apotheosized analogue.

Furthermore, it is evident that the analysis of transmissions in any one of a series of copresent modalities in a communicative event may not necessarily result in an entirely complete, coherent, or homogeneous semantic domain. Much of the information simultaneously broadcast is often redundant and perceptually augmentative, and some of it may be contradictory. Some of it may be supplementary in providing collocational semantic markings with respect to information in another modality.

I think it is a reasonable assumption that each of the various sign systems employed by humans in social communication has been designed or evolved to operate in concert with all others. Cross-modal indexing, redundancy, complementarity, and supplementarity are properties of any code, perhaps to a greater degree than we may have been willing to admit in the past. Any human sign system is de facto not merely an open system, but is an asymmetrical and dynamic system: it possesses both dynamism in its synchronicity and stability in its diachrony.

This is an extremely complex state of affairs, since it situates human sign systems somewhere between mechanistic well-formedness and idiosyncratic bricolage; but I will also take it as a reasonable assumption that this state of affairs has not only been highly adaptive in the evolution of the human line, but that it has, in a variety of imaginable ways, been responsible for what it is we have become as a species.

The peculiar internal nature of human sign systems is both a concomitant of and a contributor to our characteristically cross-modal behavior. Moreover, the human grade of intelligence is such that when faced with a choice, we invent a third possibility, or we answer a question with a question. We contrast with our nearest primate relatives not merely by the possession of any one code (whether verbal or visual), but rather by the globality of our intelligence in all modalities.

The chief task of semiotics—its highest priority—is the clarification of the multidimensional geometries of relationship underlying this multimodal behavior, which is manifest in even the most simple communicative events in daily life. The question is, what is minimally implicated in any such event?

If we situate ourselves at the locus of verbal language, it will be evident that any speech act is co-occurrent, minimally, with the following:

Each of these modalities—which, depending upon the conventions of a given culture, may themselves incorporate more than one ‘code’ as such—may be said to broadcast simultaneously with a particular speech act. In concert, these intercalated transmissions define and delimit a communicative event.

As important as it may be to identify and define the modal components of a given communicative event, it is no less important to stress and explicate their salient contrastive structural properties, for it is not the case that we are dealing with topologically identical cells in a matrix, or merely different shapes in a jigsaw puzzle. Nor for that matter will the relative contributions of copresent signings in different modalities necessarily be equivalent; nor will it be necessarily the case that they will always be hierarchicalized in any one linear direction, wherein any one set of signings invariably stands as a ‘figure’ to the ‘ground’ of other signings in other modalities.

There are significant structural differences among the copresent signings in a communicative event, which in large part are concomitants of the particular sensory channels implicated in a transmission. Vision is directional; audition is omnidirectional. Moreover, there are important differences with respect to relative permanence of broadcast: speech signals decay instantaneously; buildings and their infrastructures remain perceptually available for use across a multitude of different speech acts. Clearly, there is an ascending scale of relative ‘object-permanence’ is the list of modalities given above.

Thus, any nontrivial understanding of the inherent multimodality of communicative events must reject a simple summation or commutative relationship of its components, for we are dealing with different kinds of components which have been evolved to do partially different things.

The cases where a speech act occurs outside this multimodal context are relatively minimal. But the situation is not necessarily symmetrical with other modalities, and speech acts may be absent in communicative events, whether or not they are replaced by surrogate or complementary signings in other modalities. Various kinds of gestural signings may serve in such a capacity, whether derived metaphorically or synechdochally from verbal structure, or arising independently of a linguistic code. Some such systems may be truly ‘deponent’, in the sense of being ancillary or paralinguistic, operating in rhythmic synchrony with speech, while others will be capable of semantic disambiguation without the copresence of verbal transmission.

There is an enormous amount of redundancy in even the simplest communicative event, and a great deal of redundancy is built into the operant behavior of any code. But this phenomena is not only infra-modal, it is cross-modal, and information supplied by one modality may be augmented by distinct formations in other codes. It may well be that the particular effability of any one code is to a certain degree a concomitant of its embeddedness in arrays of copresent codes, both actually and potentially. In visual communication, the symbolic and significative potential of a given gestalt is a necessary coefficient of its ground, in mutual reciprocity. From the perspective of communicative events, this codeterminate ‘ground’ involves both inframodal and cross-modal contextualization.

The real question here is the extent to which we are capable of modelling such complexities. Taking merely the four modal activities noted above—namely, verbal utterances, dermal patterning, somatotopic signing, and architectonic appropriation—then our task will be to clarify the relationships defined by the operant behaviors of copresent signings in these domains. It seems evident that these relationships will not only be linearly syntagmatic, metonymical or commutative—since however useful it may be to model such phenomena as if they were ‘texts’ or complicated ‘rebuses’, they are more than these. Nor are they merely paradigmatically associated, or metaphorically related: a communicative event is necessarily more complex than a linguistic unit such as a phoneme, defined by the intersective copresence of a simultaneous bundle of distinctive features.

The relationships in question will be both metonymical or synechdochal, and metaphorical—that is to say, both syntagmatic and paradigmatic. Furthermore, whether a given relationship between signings in any two modalities is paradigmatic or syntagmatic may well be a function of the stance of the analysis. In other words, the relationship may be non-transitive or irreversible: a metaphorical relationship from one direction may be synechdochal when modelled from the other direction. All of this may be over and above the patent syntagmatic-paradigmatic oscillation among units in a particular code at different levels in its hierarchy of sign formations. It may very well be the case that not only will such relationships be asymmetrical, they may be differently asymmetrical from the standpoint of different modalities.

But there are two further complications. The first concerns the phenomenon of markedness which pervades any code: to what extent do we really understand markedness relations as applied cross-modality? I am not aware of any study which addresses this problem either directly or indirectly, although the work on palaeolithic symbol systems by the Soviet writer Toporov¹ and the American anthropologist Marshack² may lead to insights into this problem. It is most clearly understood in work in linguistic semiotics, notably in the work of Jakobson and others,³ where its relationship to metaphor and metonymy is distinctly specified.

The second complication regarding the nature of cross-modal relationships in communicative events has to do with the problem of the relative dominance of various functions in a transmission in a given signing, and its copresent associations in other modalities. In other words, will it necessarily be the case that a focus upon conative, or phatic, or aesthetic functions in a given signing will be equilibrated with an equivalent focus in another modality? I suspect that here also the situational possibilities are quite complex, for it is evident that from the perspective of a communicative event in its totality, distinct modalities may contribute different weightings in functional dominance: otherwise equivalent verbal utterances with a dominance on one function may acquire different transmissive foci in different settings. Once again, this fact suggests that a semiotics of the code per se is a selective fiction in isolation from its multimodal communicative context. And taken together, the various complexities just outlined suggest that the relationship between a semiotics of the code and general sign theory is necessarily not metaphorical, but inevitably synechdochal.

The calibration of these possible geometries of relationship is precisely what is the most urgent task facing semiotics. As it is, it is exceedingly complex in merely dealing with the four modal domains abstractly and generically discussed here, let alone in the actual pluralistic conditions of semiosis in daily life, which involves the intercalation of signings drawn from many distinct codes,—both those inherited from our palaeolithic past, such as verbal language and the architectonic code (the latter of which is now evidenced as early as 300,000 B.C. in its present form), as well as those assembled yesterday.

The situation is precisely this: if it is the case that each of the various semiotic codes evolved by humans are irreducible with respect to each other—in other words, that the contents expressed by complex nonverbal units cannot be translated into one or more verbal units, and vice-versa, except by weak approximation-then the hope to find some uniformitarian common denominator, some ‘meta’ language, some non-trivial and non-reductionist general theory of semiosis, is an illusion except as a selective and synechdochal fiction. There are no metalanguages; rather (and much more interestingly) only selective infralanguages which are part and parcel of given codes. And since any infralanguage by its very nature adds to the body of a given code itself, and thereby alters its topology, so in any attempt to see itself as an object it must undoubtedly act so as to make itself distinct from, and thereby false to, itself. In this condition it will always partially elude itself. The kitten will always chase its own tail.

I suspect it is in the very nature of any human sign system to partially elude itself, and herein lies the very effability of the semiotic codes which we have evolved. But if there is no truly synoptic picture of semiosis in a value-neutral sense, no one perspective which subsumes all others, we are left with something which in the long run is inherently much more useful—namely a mutable focus on communicative events which affords a temporal and syntagmatic cascade of perspectives which selectively illuminate a situation in a stereoscopic and overlapping fashion.

It is precisely the operant nature of our multimodal understanding which privileges each perspective selectively and successively. Since any analysis is a function of the purposes to which it is put, it is in the nature of any analysis to be provisional, for there are many different functions and purposes, some of which are contradictory and irreducible, even if they may be copresent in the same analysis to varying degrees of dominance.

Consequently, it will be necessary to be explicit regarding the inevitable teleological determinants in any semiotic analysis, even if this implies not only an abandonment of a semiotics of the code except as a provisional fiction, but also an abandonment of a uniformitarian theory of semiosis itself in favor of a holographically-overlapped matrix of generic and irreducible semiotic theories.

Of course this is not to deny the necessary operational paradox that any code can be employed, in communication, as a provisional metalanguage. Nor is it to deny the evident fact that even though many codes are mutually irreducible in a strict sense, they may reveal correlative processes of formation and transmission, as we have argued in the present study. But whatever they share is shared by virtue of their status as human sign systems with partly-overlapping and mutually-implicative functions. Codes are necessarily correlative rather than isomorphic. The role of semiotics is to provide a clearer understanding of how and why each copresent system provides its own particularly powerful perspective on the totalities of human experience, and the ways in which each such perspective necessarily implicates all others. The most urgent task awaiting semiotics is precisely a principled attention to the directional geometries of this implication.

And in addressing these implicational relationships our analyses will be most productive when the essential and fundamental difference between meaning and reference are clearly borne in mind. Meaning is the specification of an ordered trace of relationships which a given sign or matrix of signs prescribes with respect to other signs within the same code, whereas reference is not an indexical relationship which a given sign bears to formations outside of semiotic systems, toward some fictive ‘real’ world, but rather involves cross-modal implications. In verbal language, for example, ‘shifters’ are cross-modally indexical with respect to significative formations in a somatotopic modality (i.e., the relative placements and perspectival positionings of addressers and addressees in communicative events). In an identical fashion, the ‘meaning’ of a given painting or environmental construct is internal to its own code, whereas the ‘reference’ of a mediaeval religious composition (or any other) may implicate a culturally copresent set of texts, doctrines or beliefs, which themselves comprise significative formations in their own right in adjacent codes. And the relationships among all these may be metonymie or indexical, or metaphoric.

I believe strongly that a clarification of these issues can only lead to a salient enhancement of the semiotic enterprise, shunting our focus more tenaciously upon the nature of relationship itself, which after all is what semiotics is all about, from its conceptual foundations to the fine grain of its ongoing analyses. I have tried to suggest here that the most productive direction such analyses can take today is in the direction of the disambiguation of the cross-modal relationships manifest in their totality primarily in the complexities of communicative events. In addition, it is only in this way that our understanding of the internal structural nature of individual codes can be made less fictive.