I

DETERMINING THE RELEVANT
FEATURES OF A GESTURAL
EXPRESSION

AN INTRACULTURAL EXPERIMENTAL STUDY

Goal and method

The aim of this study (Calbris 1980) was to verify the understanding of complete French gestures (including facial expressions) out of context, and to study their signifying structure by determining the respective contributions of the gesture alone and of the facial expression. The physical expressions, out of context and without speech, were filmed by two cameras simultaneously. The first camera filmed the gesture alone (G); the second filmed the entire physical expression, including the face (FE:G) (Illustration 1, a and b). In the test, subjects are asked to associate the filmed image, partial or complete, with a signified from a list of utterances. The contribution of the facial expression can be deduced from the difference in responses given for the two images, i.e., FE:G – G.

To make the film, we selected several dynamic expressions with gestures involving various vehicles—thumb, thumb plus other digit(s), forefinger, fist, hand, etc.—and situated at different levels of the body, from the waist, stomach, chest, shoulder, chin, mouth, nose or cheek, eye or temple, and forehead, to above the head. In all, thirty-four expressions were chosen (Table 1). How can one isolate, from the full signifier (F), the partial signifier (p) assumed relevant? For a physical expression involving movement at chest-level, camera #1 films only the chest, while camera #2 films the whole body. When the gesture occurs near the face, camera #1 takes the expression from the back (if G is on the side of the head, or goes from one side to the other) or three-quarters from behind (if G is in the middle of the face) in order to isolate the gesture from the facial expression.

Table 1. Descriptions of filmed gestures and their corresponding clichés

___________________

¹ Approximate equivalents of attitude are given, without regard to the French imagery.

² The bracketed expressions, explicitly illustrated by the gestures and replaced in the experiments by more general utterances, are translated literally.

Full views of filmed gestures corresponding to Lists One and Two of Table 1 are shown on pages 4-7. Certain movements are shown in complementary drawings by Zaü (for Calbris and Montredon 1986, by permission of CLE International)

Illustration 1. Partial (a) and full (b) views of the emblem ‘C’est mon petit doigt qui me l’a dit’

Test

The common utterances listed below posed a methodological problem. Any logical person can associate by deduction the thumb-twiddle gesture with the utterance ‘He twiddles his thumbs,’ which describes it, without having to know its signification. To replace such descriptive utterances, equivalent ones are given which specify the signifier by attitude or situation (in italics):

Of course, the subjects were informed that such substitutions had been made. I used the example of the thumb and forefinger pinching the lips to signify ‘My lips are sealed’ (Fr. Motus et bouche cousue); in this case, the utterance would have been replaced by a non-descriptive one such as ‘Top secret.’

The subjects were to choose one utterance for each filmed sequence. Since the complete list of 34 utterances was too long, it was divided in two. I separated closely related items such as Ras le bol! and La barbe! and tried to keep the same headings of signifieds in the same order to facilitate memorization (see Lists 1 and 2 in Table 1). In addition, it was not possible to show all of the partial and full views (68 in all) to a given subject. I therefore opted for two equivalent lists of utterances (Lists 1 and 2), two equivalent series of filmed physical expressions, each with a partial (p) and full (F) view, presented in random order, and two equivalent groups of subjects (Groups A and B). The distribution chosen, given in Table 2, eliminates factors that might skew the results (e.g., individual subject, order of lists, order of views). Each subject from Group A sees the full set of gestures, with as many partial views as full views but with a single view per gesture. The same holds for Group В subjects. In order to compare the partial and complete signifiers for a given attitude—each signifier being seen by one group—it suffices to eliminate the influence of the group by performing the experiment with groups of roughly thirty subjects each.

All of the subjects had to be French and to have lived in France (place of birth and of secondary school or university). These requirements were fulfilled with the generous help of the English language staff at the Centre de Linguistique Appliquée in Besançon who, during the summer of 1976, convinced their students to volunteer for a study on French gestures as part of the research of an ATP (Action Thématique Programmée) of the CNRS (Centre National de Recherche Scientifique). Of the 47 French volunteers, divided into two groups (A and B) with 22 and 25 subjects respectively, 53% were students, 33% teachers, and 14% from other professions. There were more men than women: 60% vs. 40%. While their ages ranged from 17 to 49, they were generally young: 84% were under 30, and the average age was 25 years. The majority of them came from the Paris area or Franche-Comté.

Table 2. Distribution of verbal and nonverbal items

To facilitate the statistical analysis the test was multiple choice, but in developing it we took into account the results of a previous test in which subjects were shown photographs of gestures and gave free answers. The subject has to associate an utterance to each filmed gesture. He has a list of seventeen sentences and is to check the one he thinks best corresponds to the gesture shown in partial or full view on film. For example, one expects the Group A subjects, seeing someone from the back draw a line over his head, to associate this isolated gesture (p) with the utterance Ras le bol! (‘I’m fed up!’), just as do the Group В subjects who see the full, front view (F) of the gesture. The A subjects see the partial view in the first series (14th image), while the В subjects see the corresponding full view in the second series (29th image).

The 16mm film is projected on a large screen.² After each numbered image, the film is stopped to allow each of the 25 subjects present to check the box corresponding to the chosen utterance on his or her test sheet.

Results

For each group, the raw results were recorded on a grid of gestural sequences vs. utterances. For each view, partial (p) or full (F), the numbers of correct and incorrect answers are given in the corresponding box (Tables 3 and 4 for Group A, Tables 5 and 6 for Group B). The identification of filmed gestures was found to be clearly significant according to the χ² test: p > 0.001. Next, the results from A and В were paired, and the percentage values were tabulated by list. Thus, for each gesture, we have the percentage of comprehension for the partial view and for the full view. See Tables 7 and 8 for the raw percentage results for Lists 1 and 2.

The overall average recognition is quite good: 84%. The gestures assumed relevant proved to be in fact very relevant; they are necessary and sufficient, since the same score is obtained for the isolated gestures and for the full face-and-body views: 83% and 85%. From this near equality between the scores for the relevant partial signifiers and for the full signifiers, must we conclude that the complementary signifiers of facial expression are insignificant? Not at all; we will show below that their positive and negative effects simply cancel each other out.

Table 3. Group A, Series 1: List 1 (22 subjects)

Table 4. Group A, Series 2: List 2 (22 subjects)

Table 5. Group В, Series 1: List 2 (25 subjects)

Table 6. Group B, Series 2: List 1 (25 subjects)

Table 7. List One - Percentage results for Group A, Series 1 and Group Β, Series 2.

Group A, Series 1, items 1-18 (22 subjects)

Group B, Series 2, items 18-24 (24 subjects)

* View repeated for reliability check.

Table 8. List Two - Percentage results for Group A, Series 2 and Group Β, Series 1.

Group A, Series 2: items 18-34 (22 subjects)

Group B, Series 1, items 1-18 (25 subjects)

* View repeated for reliability check.

___________________

2.The film is 16mm black and white (10 minutes), filmed by François Luxureau and François Didio at the studio of the SERDDAV (Service d’Etude, de Réalisation et de Diffusion de Documents Audio-Visuels).

Positive or negative role of the complementary facial signifier

When the principal signifier is relevant, that is, recognized by two-thirds of the subjects (which is the case 94% of the time), the role of the complementary signifier may be null (44%), positive (25%) or negative (25%). In other words, for roughly half of the cases, it plays no role, while for the others it is as much negative (1/4 of the cases) as positive (1/4 of the cases). One can also say that when the principal signifier is relevant, it carries in half of the cases as much information as the full expression; a quarter of the time it carries less; a quarter of the time it is better perceived than the full expression because the other signifier, rather than being complementary, is superfluous and hinders the perception of the essential signifier.

The complementary signifier plays a positive role when the gestural signifier has several meanings and the facial signifier destroys the polysemy by confirming one of the possible significations. For example, the UPWARD CHOP gesture (‘Let’s get out of here’) was confused by some subjects with the PHALLIC FOREARM JERK when isolated, but the facial expression of calling confirmed the idea of a suggestion to leave. It should be noted that the facial signifier often eliminates the ambiguity of the gestural signifier simply by indicating whether the gesture should be interpreted in a positive manner (smiling face) or negative manner (unsmiling face or grimace). For example, the thumb and forefinger joined in a circle express both nothingness and ‘delicious’ perfection, the latter of which would be confirmed by adding a smiling face.

The complementary signifier is more likely to play a negative role when the gestural signifier is particularly relevant and well interpreted:

(1) A superfluous facial expression can modify the meaning of a gesture. For the gestural sign of fear, the FINGER PURSE, with 100% isolated recognition or the admiring THUMB UP gesture, also 100% recognized, the superposition of a superfluous exclamatory facial expression lowers the scores to 90% and 88% respectively, with the other subjects choosing the utterance Oh là là, il est gonflé! (‘He’s got quite a nerve!’). The addition of the exclamatory facial expression becomes synonymous in one case with ‘there is reason to be afraid’ and in the other case with ‘that’s worth taking your hat off to.’ Hence 9% of the subjects chose, for the gesture of fear, the formulation ‘He’s got quite a nerve (in his place, I would have been scared stiff),’ and for the gesture of admiration, 12% chose the same utterance, ‘He’s got quite a nerve (hats off to him).’

(2) A mischievous face above the FINGER SHAKE (for ‘Watch out, boy’) led some subjects (12%) to select an attitude whose formulation recalls mischief: the roguish attitude Je t’ai eu! (‘Got you there!’). But how, you may ask, is the gesture integrated into this new interpretation? As a supplementary explanation for the success, ‘You should have paid attention; see how I got you!’

(3) For 85% of the subjects, flicking the thumbnail off an upper front tooth and projecting it forward (THUMBNAIL FLICK) signifies that one has no money and consequently has not a bit to eat: Que dalle! (‘Nothing!’). But squinting and laughing eyes introduce a shade of amusement which led half of the subjects to choose the attitude of vengeful triumph Et vlan, dans le baba! (‘Wham, right in the ass!’) which is a possible extension of refusal, ‘Not a red cent! You’ll get nothing—that’s what you deserve, so there!’

Let us return to the previous cases. To the extent that the gestural signifier allows a very good identification, the complementary signifier becomes useless. It then serves to actualize the gestural signifier, and the added nuance sometimes turns a subject away from the essential meaning. What should be complementary becomes superfluous and leads to errors. This means, logically, that it is easier to get a consensus on the essential signification of a ‘signifying’ gesture than on its actualization by the facial expression. It also means that the emotional shading carried by the face sometimes overrides the essential signification carried by the gesture: mischief over warning, triumph over refusal (cases 2 and 3). Some subjects will be more attentive to the emotional form of the expression than to the information underlying it. However, their choice follows not from eliminating the element deemed less important, but from giving precedence to an element judged more important. This preference produces a new attitude which integrates the two elements in a different way. Consider again example 2, of mischievous warning. The importance placed on the facial expression (here a positive element) provokes an inversion of the positive and negative elements and hence a new signifying configuration in which pre-eminence is no longer given to the negative element carried by the gesture (warning), shaded by a positive attitude (mischief), but is given rather to the positive element of mischief with regard to someone who has failed to take the warning.

What was the point of having introduced positive elements when the goal was to verify the identification of essentially negative expressions? A penchant for risk? Methodological refinement? No, the reason was psychological. If I eased several unpleasant attitudes by adding a positive touch, it was because I wanted as far as possible to present a pleasant face in order to compensate for the number of denigrating physical expressions which the subjects had to view. By their choices, certain subjects insisted on a logical point of view: the complementary signifier must serve to reinforce the meaning of the principal signifier, rather than actualizing it in a particular situation, which in any case is difficult to convey exhaustively. In other words, if the principal signifier indicates a negative attitude of threat, failure, or refusal, it is logical that the complementary signifier do the same or at least be consistent with the negative aspect of the attitude. In short, my error pointed out the significant structuring of gestural elements according to a ‘psycho-logical’ choice by the receiver.

In summary, the complementary signifier plays a negative role if it carries a diversifying nuance, if it is in contradiction with the principal signifier, or if it generally supplements a different principal signifier. The facial expression of vengeful repartee superposed on the gesture of privation (‘Nothing!’) evokes vengeful refusal: ‘(Not a red cent) Wham! Right in the ass!’

Errors due to the principal signifier

These errors can be arranged under three headings distinguishing between those due to ambiguous signification, overly general signification, or partial decoding of the gesture.

Here are two examples of ambiguous signification. (1) The fist knocks several times on the forehead to signify that it is difficult to get ideas in, either because the subject resists (Têtu comme une mule: 86%) or because he is stupid (Mon Dieu, qu’il est bête: 14%). (2) When the nail of the forefinger touches the thumbnail to express the idea of precision or perfection worthy of admiration (Délicieux: 80%), the fingers form a circle which is also used to express the idea of nothingness (Plus rien: 14%).

Sometimes the gesture evokes a general notion whose interpretation depends on the situation imagined. For example, the gesture of the forefinger sliding under the nose depicts the seme passer sous le nez (lit. pass under nose = slip through fingers). The verb can be conjugated in the past or future, with a first, second, or third person pronoun qualifying the object. In the past, with a first person qualifier, it is the retelling of failure, ‘Too late, it slipped through my fingers.’ In the future with a second person qualifier, it becomes a refusal, ‘You won’t get anything; it will slip through your fingers,’ and by extension a rude refusal, ‘He can get lost.’ Note that in parallel there is a reversal of role, one person’s refusal being another person’s failure.

To illustrate the third heading, partial decoding of the principal signifier, we give examples of mismatches arising from features shared by two relevant signifiers. (1) The isolated UPWARD PALM CHOP (‘Let’s get out of here’) was understood by 20% of the subjects as ‘Fuck you!’ (PHALLIC FOREARM JERK). Indeed, both movements involve the left hand on the right forearm. (2) The FIST SHAKE over one’s head in a sign of victory was interpreted by one subject as vulgar, ‘Fucker!’ It is true that in both gestures the forearm is raised and the hand makes a fist. Let us look at what differentiates the three gestures: the UPWARD PALM CHOP is distinguished from the FOREARM JERK by the hand being flat and in line with the forearm, while the FOREARM JERK is distinguished from the FIST SHAKE by the presence of the left hand on the forearm.

With regard to errors arising from the complementary signifier, see the negative role of the facial expression discussed below.

Errors due to the signified

It must be remembered that the errors observed are limited to the utterances compared within each list of 17 items. Errors from the signified are most often noted for the entire expression, i.e., gesture and facial expression. An error is more revealing if it is also found for the partial signifier. Thus each of the physical expressions below, seen partially (p) or fully (F), was incorrectly associated (→) by some subjects with the verbal expression opposite.

The study of mismatches on the level of the signified leads to a certain classification. The proximity of two signifieds is related to various phenomena of equivalence, ambivalence, or a shift from one signified to another via a third which is more general and which includes the other two (al→A→a2). Moreover, we observe the phenomenon of extension, that is, a shift between two signifieds within an imagined situation, often made possible by a cause and effect relationship. For example, ‘I got you there!’ because you are an ‘idiot’. Here are illustrations of the various phenomena noted:

EQUIVALENCE. The reciprocal mismatch between the two expressions Qu’il est bête (‘He’s so stupid’) and Crétin (‘Idiot’) is indicated above.

AMBIVALENCE. a+ ←A→ a–. The fact that the VERTICAL HAND SHAKE gesture (Oh là là, il est gonflé—‘He’s got quite a nerve’: 74%) was understood in a negative manner (Faut se le farcir—‘What a pain’: 21%) or in a positive manner (Chapeau!—‘Well done!’: 4%) is related to the ambivalence of this expression, an ambivalence which probably holds for all exclamations. The same mismatches occur, less frequently, for the full view.

SHIFT THROUGH A MORE GENERAL INTERMEDIATE SIGNIFIED: al→A→a2.

Table 9 gives four examples:

(1) The utterances Ras le bol! (‘I’m fed up!’) and Faut se le farcir! (‘What a pain!’) both express a feeling of annoyance, of saturation (cf. the FED UP gesture which evokes Faut se le farcir! for 9% of the subjects).

(2) The utterances Qu’il aille se faire voir (‘He can get lost’: 72%), Que dalle (‘Nothing’: 9%), and Je ne te crois pas (‘I don’t believe you’: 13%) have in common the idea of refusal.

(3) Pretending to feel money by rubbing the thumb against the forefinger and middle finger evokes the idea of money that is required: Hé, faut payer (‘Hey, pay up!’) or even refused: Que dalle, pas un sou (‘Nothing! Not a red cent’).

(4) If the teasing CHIN SCRAPE gesture for Na na na (‘So there!’) evokes the utterance Dans le baba (‘In your ass!’) for 18% of the subjects, it is because both utterances imply a triumph at someone’s expense.

Table 9. Shift in meaning: a1→A→a2

EXTENSION OF A SITUATION. The THUMB UP gesture corresponding to the signified ‘Well done!’ is associated with the item Oh là là, il est gonflé (‘He’s got quite a nerve’). The TONGUE OUT gesture signifying ‘Got you there!’ is associated with the item Crétin (idiot). The FINGER SHAKE, a sign of warning, ‘Watch out, boy’, is associated with the item Têtu (stubborn). These three mismatches suggest the following idea: to allow a shift between signifieds, the subject may imagine a particular situation which includes a paraphrase of or complement to the physical expression viewed, e.g., ‘Well done, he’s got quite a nerve,’ ‘Got you there, idiot,’ ‘Watch it, you stubborn brat.’ The complement is often explanatory. Thus the word ‘stubborn’ explains the threat signified by the shaken forefinger, and the exclamation Oh là là, while reinforcing the admiration signified by the raised thumb, gives the reason for it.

The subject places himself in a real situation in which the gestural information and verbal information complement one another. Here the gestural information is manifest and is complemented by verbal information (Chapter 8, Economy of information).

DIVERSIFYING THE SITUATION BY CHANGING AN ACTOR. Suppose that a signified is synonymous with a particular action. This signified is maintained and at the same time varies according to the ways in which the action in question is performed (Chapter 6, ‘Rather like an epidemic . . .’). The subject, object or moment of the action can change:

• Subject of the action: role reversal. Since the gesture is symbolic, I can mime an action as if it applied to me, my listener, or a third person not present. I cannot mime the actualization or the context of the action. For example, I would mime ‘Slip under your nose’ as ‘Slip under my nose,’ sliding my finger under my own nose and not under my listener’s! The same gesture expresses failure or the possibility of failure on the part of oneself or someone else. Thus we observe a certain number of role reversals.

• Object of the action. The clichés which accompany the gesture of rude refusal (HAND TOSS) such as qu’il aille se faire cuire un oeuf (lit. let him go have an egg cooked) or des clous (nails) or else ouais, à d’autres! (yeah, to others) involve different objects of refusal: refusing to do, quil aille se faire . . . , recognized by 72% of the subjects; refusing to give, or giving ‘nails,’ which is implicit in the choice of pas un sou (‘not a red cent’) by 9% of the subjects; and refusing to believe (‘try someone else’), which 13% of the subjects made explicit by choosing the utterance of disbelief. Hence, with the same rudeness (HAND TOSS) one can refuse to do (72%) to give (9%) or to believe (13%).

• Tense of the action. (1) The same action ‘slip under the nose’ in the past tense (slipped under the nose) or future tense (will slip under the nose) corresponds to very different situations: failure, ‘Too late. We missed it,’ or refusal, ‘(You’ll get nothing) Not a red cent.’ (2) In the same way, ‘screw’ in the past tense gives ‘Got you there (I screwed you),’ and in the future, ‘Fuck you (Go get screwed).’ This explains how the PHALLIC FOREARM JERK evoked the utterance Je t’ai eu (‘Got you there’) for one subject. (3) We move without knowing it from threatening to doing, as when 12% of the subjects shift from the item Gare à toi (‘Watch out’) to Je t’ai eu! (‘Got you there!’).

In summary, the errors due to the signified can be explained by the phenomena of equivalence, ambivalence, or the mediation of a third signified that is either more general or shared. We also observe cases in which an attitude is extended to another within an imagined situation. Finally, certain mismatches can be explained by what I would call a diversifying treatment of a signified.

Semantic data deduced from mismatches

Semantic components of a sign

We can base our study of the components of a signified on the phenomena of convergence and dispersion. If elements are seen to converge toward a point, and if the same elements diverge from the same point, we are led to the conclusion that together the elements constitute this center of convergence and divergence; they are its components. Here is an example for the signified Que dalle! pas un sou! (‘Nothing! Not a red cent!’). In Table 7, the spread in meaning from this signified appears on the horizontal line opposite the partial and total gestural signifiers, while the convergence appears in the column under the phrase Que dalle (‘Nothing! Not a red cent!’). Below we give a list of sememes which can explain the situation (Table 10). They indicate the type of action, its object, the reason for it, and the reaction of others. The mismatches indicate that the verbal or gestural expression NOTHING implies that the speaker triumphs (c); he takes revenge (e) by refusing (a) money (b) to the listener, who thus suffers defeat (d). Since ‘Que dalle!’ also signifies ‘not even that; a nothing!,’ a convergent mismatch with the utterance ‘Nothing left!’ (f) would doubtless have been observed if the utterance had been on the same list. The separation of utterances into two lists precludes proof.

Table 10: Components of the signified ‘Que dalle’: Nothing

Hypersign

The study of mismatches brings out the shared aspects of signifieds or the connections between them. To the extent that the corresponding signifiers are analogous, one can establish a correspondence between a hypersignified and a hypersignifier. To put it differently, let us imagine several gestural signifiers and their corresponding signifieds. If one common element appears on the level of the signifiers and another on the level of the signifieds, it seems logical to associate the two elements, one being the signified of the other.

The signified ‘fucker’ is mimed by raising the right forearm, with a closed fist, and stopping it with the left hand, representing an erection (PHALLIC FOREARM JERK). Some subjects associated this gesture with the utterance Je t’ai eu (‘Got you there’) which is a metaphor of physical possession (to get, to have someone). And this latter utterance is displayed by sticking out one’s tongue! The two gestural signifiers (raised right forearm and protruding tongue) are phallic symbols, and the two signifieds point in parallel to the idea of physical possession: ‘I had you . . . , fucker!’ A symbolic erection signifies metaphorical possession.

While sticking out one’s tongue can mark a triumph over someone else, projecting the thumb outward, as it is scraped under the chin, is a derisive sign of triumph, Na na na! (‘So there!’). The signifieds of these two attitudes are similar, as are the signifiers: projecting the thumb out under the chin appears to be equivalent to sticking out one’s tongue if we consider the aggressive version ‘Bien fait pour toi’ (You got what you deserved) in which a child pushes his thumb out once while sticking out his tongue.

In addition, projecting the thumb out from under the chin (Na na na!) and flicking it off a tooth—possible sign of vengeful refusal (Que dalle!)—were both associated with the utterance of vengeful repartee (Et vlan!) signified by jabbing the fist out from the stomach. This means that vengeful repartee (Et vlan!) and refusal (Que dalle!) share the aspect of vengeance, while the repartee and the derisive attitude (Na na na!) both have a triumphant quality. One cannot help but see an analogy between these signifiers, all of which involve a projection, a sudden protrusion of some part of the body: the tongue (Je t’ai eu!), the thumb off a front tooth (Que dalle!), or from under the chin (Na na na!), and the fist out from the stomach (Et vlan!). One is all the more tempted to group these signifiers since the various mismatches reveal a common element on the level of the signifieds: the four utterances are manifestations of triumph over someone, whether it be vengeful refusal, a repartee that hits the mark, or a bet won. Does this mean that the sudden protrusion, evoking an erection, is a symbol of affirming oneself over someone else?

Summarizing, the thirty-four physical expressions chosen were recognized out of context by 85.5% of the subjects. Their emblematic function is experimentally verified, as are the respective contributions of the gestural and facial signifiers. The isolated gesture is found to be highly relevant (recognized by 83% of the subjects), while the role of the facial expression is ambiguous.

The observation of mismatches allowed both a microscopic and a macroscopic study of signs (components and hypersigns). Through these mismatches, we note that the gesturally indicated signified can vary, actualized differently according to the subject, object or tense of the action in question. This actualization leads to semantic derivatives (privation-failure-refusal). The mismatch analysis also uncovers the complementary role of gesture with respect to speech. While the test consists in finding the phrase ‘synonymous’ with a gesture, some subjects, by giving a verbal complement instead of an equivalent, seem to be placing themselves in real speaking situations. We have also learned that the pieces of information carried by the gestural and facial signifiers are not simply juxtaposed, independent entities. The combination, or signifying configuration, of the two changes according to the figure-ground relationship between them.