The art and science of emotions, part one

Lie to me

I've been reading about the newest controversy in "emotion science," the branch of psychology devoted to the study and analysis of how human emotions are revealed through facial expressions. At issue is whether these external indicators are in fact universal—as demonstrated by the research of Paul Ekman—or, as Northeastern University Professor Lisa Barrett argues, if "each of us constructs [emotions] in our own individual ways, from a diversity of sources: our internal sensations, our reactions to the environments we live in, our ever-evolving bodies of experience and learning, our cultures."

At the heart of the debate is Ekman's research, conducted in the 1960s (and cemented in the public consciousness by Malcolm Gladwell's Blink), in which he showed a series of faces expressing seven basic emotions (anger, contempt, disgust, surprise, sadness, happiness and fear) to people all over the world and found that, regardless of race, culture, language or social status, people could identify the emotions correctly. Thus, they concluded that emotions are universal and quantifiable (I'm not sure how they extrapolated the latter from the former, but then I haven't read the study, only the Gladwell).

Given the boost in prestige that comes with a mention in a runaway best-seller, the sudden interest of government agencies and Hollywood studios and the multimillion-dollar offers that follow, I suspect a lot of theories based on common sense and extrapolated from controlled laboratory experiments could suddenly be made to seem quantifiable. 50-odd years later, however, it seems there's a backlash.

The Department of Homeland Security, having spent $878 million putting Ekman's deception-detection techniques to the test in America's airports, recently reported that although thousands of arrests have been made since their adoption in 2004, the program hasn't caught a single terrorist. Also it's been running under constant criticism for being a thinly-disguised system of racial profiling based on dodgy science and which, if not actively harmful, is at best useless in practice (and here I thought they put the last nail in the coffin of that theory with the cancellation of Lie to Me).

In a Boston Magazine article, Professor Lisa Barrett contends that Ekman's original study doesn't prove what it claims to because when they showed people photos of the seven expressions, they provided a multiple-choice selection of emotions to label them. Participants were asked to match the emotions expressed in the photos with words like "happy" or "angry" or stories that were obviously sad or frightening, etc. Barrett recently restaged the same experiment with an isolated tribe in southern Africa, only without the labels, and found they were unable to replicate the results that had supposedly proven Ekman's hypothesis:

Quote:"...Smiling faces went into one pile, wide-eyed fearful faces went into another, and affectless faces went mostly into a third. But in the other three piles, the Himba mixed up angry scowls, disgusted grimaces, and sad frowns. Without any suggestive context, of the kind that Ekman had originally provided, they simply didn't recognize the differences that leap out so naturally to Westerners."

(Oh, for what it's worth, Barrett's researchers pared it down to six emotions, either discarding contempt or combining it with disgust, perhaps because they couldn't tell the difference.) So this is where I have to call bullshit. Do the differences between "angry scowls, disgusted grimaces, and sad frowns" really leap out so naturally to Westerners? I'm about as Western as you can get and, although I usually score better than average on emotional intelligence tests, I think it's highly disingenuous to suggest that to Westerners the difference between expressions of anger, disgust, contempt and sadness naturally leap out at us. Setting aside the "epidemic" levels of autism and the 1-2% of the population that may be psychopathic and therefore clinically incapable of gauging other people's emotions, the very question sets up a false—or at any rate, questionable—dichotomy between what are in reality fluid, often interchangeable—and even more often coexistent—states of mind.

Add to that ambiguity the fact that most of the time when we witness a genuine expression of one of those negative, antisocial emotions, there's going to be some simultaneous attempt on the part of the person expressing it, to hide or disguise their true feelings. Your boss may be furious when you openly disagree with her in a team meeting but the office code dictates that she treat you with a modicum of professionalism, so you may only catch a fleeting glimpse of her anger. Naked, unadulterated expressions of pure anger, disgust, sadness and contempt—despite decades of reality TV—are rarely directly expressed in polite society without a dozen other emotions crowding the edges, competing for face-time, as it were, and muddying the waters.

The Himba tribe categorized happy and frightened faces correctly, same with most of the "neutral" faces, which corroborates the original experiment. In the early 60s, there was no World Wide Web connecting friends and coworkers to like-minded individuals around the world, no "translate this page" enabling teenagers to read each other's blogs and find their innermost thoughts echoed by teenagers in the most distant and different of places. When Ekman began his study, the US had yet to pass the Civil Rights and Voting Rights Acts that mandated equality and legal protection for minorities.

What seems to us unquestionable, that we are all one human family and that race is merely an artifact of ancient human migration patterns, a set of classifications more useful in emphasizing superficial differences and justifying the oppression of various groups for political gain than anything to do with science or identity, were not yet truths that everyone held self-evident—not by a long shot.

The Civil Rights era was also the peak decade of the Cold War, when science was being used to fortify Western nations against other Western nations, resulting in a climate of such fear and mistrust that some 20 years later Sting would feel the need to remind Americans that "we share the same biology regardless of ideology" (and even he stopped short of an unqualified pronouncement, saying only, "I hope the Russians love their children too").

So in its context, Ekman's research came along at a crucial time in history, proving that emotions were the universal currency of human interaction and assuring people that a smile in Denmark was the same as a smile in Zimbabwe. While it may seem obvious to us—even a bit too easy—at the time, that assurance and others like it probably helped lay the foundations for the socially connected, color blind world we like to believe we inhabit today.

Back to the idea that Westerners are better equipped to differentiate sad faces from angry faces. It goes without saying that we in the modern "West" exist in perpetual immersion in other people's emotions. Every day we're surrounded by their mind-numbing proliferation, from the constant overshare of our friends' mental states on Facebook to the exploitation of celebrity dysfunction as the media's prevailing business model, and somewhere in between is the vast emotional dumping ground that is reality television.

What is she feeling?

Take the face above; I think we can agree she's not happy. There are elements of anger and disappointment, which is related to sadness, just a more accusatory form of it. In some cultures, that might be enough but not in the West. So would you label this contempt? Disgust? What if you hadn't been given the multiple choice; would you categorize this as derision, disdain, disbelief, revulsion, rejection, sneering, scorn or just adolescence? You know how the Eskimos have 24 words for snow..?

This is purely speculation on my part, but isn't it possible that the Himba simply haven't yet developed the specific social context in which it would be necessary to understand the difference between an expression that says, "I have to be home by 10:30? God, this is so. unfair. None of my friends have a curfew. Why do I have to be the only one whose parents are total prehistoric assholes?" from an expression that says, "Omg, Kevin told Jason to tell you that he likes me? Isn't he that weirdo who wears a parka even when it's like 80 degrees out? I am totally going to die. That is so. gross."

What if the researchers came to you and said there are six categories of human emotion; tell us what they are and sort this pile of 100 expressions into your six categories. Would you have named the same six? Where would you put a confused expression? A seductive, come-hither gaze? How about a look of awe or astonishment? It's obvious when you give it just a bit of thought that emotions are not discrete modes of functioning or isolated mental states that can be quantified, treated or judged for whatever purpose is footing the bills.

A while ago, I stumbled across this amazing chart of facial expressions created by Joumana Medlej as a resource for fellow artists. Its stated purpose is nothing grander or more subversive than demonstrating how the vast, ephemeral range of human emotions can be distilled and conveyed convincingly by a stylized combination of economical lines.

Emotion chart

Anyone who has taken a life drawing class knows that just as important as the marks you make on the page is the radical simplification of visual data that your brain must perform in order to render your subject in two dimensions. As your hand creates the simulation, your eye is constantly taking in information which you must decide whether to incorporate or ignore; discarding perceptions of volume, distance and depth while documenting the range of light and shadow to convincingly convey edges, curves and foreshortening.

I remember one moment in my first year drawing class when this concept demonstrated itself to me in the form of a model's profile. She was facing the window and as I started to trace the outline of her face I realized that what I was attempting to render in black charcoal had the exact opposite appearance in real life. The "edge" that so clearly defined her features against the backdrop of the classroom behind her was actually the brightest part of her face, reflecting the light of the sun the same way the glowing curve of the moon stands out against the night sky. I stared at her profile for quite a while that morning, as if seeing the human face for the first time, before finally turning back to the page and consciously committing to the radical simplification I had always performed automatically until that moment.

As Joumana Medlej explains, "In drawing facial expressions, we have to bypass 'what a sad face would look like' and aim for 'how we would recognize a face as being sad.' This is because an illustration must make up for the loss of other clues that we have in real life to decipher someone's mood, many of which we perceive unconsciously. In real life, we just know..."

In the process of simplification, some lines carry more weight than others—take eyebrows, for example. As every makeup artist will tell you, no feature bears a greater burden when it comes to communicating a person's emotional state. If eyes are the windows to the soul, then eyebrows are the sign on the door that can instantly transmit messages in all caps, like KEEP OUT or HELP.

Expressive eyebrows

This is demonstrated in the sparse but completely effective emotions expressed through the stylized curve of a single eyebrow in the chart above, and it's communicated with brilliant brevity by cartoonists on the faces of characters from Cruela De Vil to Sponge Bob Square Pants. Kids can easily tell the difference between the lonely, wistful Wall-E and the lovestruck, slightly intimidated Wall-E and he doesn't even have a face.

Wall-E

Of course emotions are complex and confounding, a set of variables shared by all but interpreted differently by each individual to the point that even siblings and old married couples who've grown to look alike over the decades, can't read the unspoken, enigmatic truth in each other's faces 100% of the time. Despite that, and in spite of the continuing efforts of emotion scientists periodically splintering off into warring factions unable to see they're merely describing none-too-distant points on the same continuum, don't most of us just intuitively know this stuff?

I think emotion science failed when it tried to make the cognitive leap from asserting that broad, basic categories of emotions—your happiness, anger, fear, sadness—are universal (which all the evidence indicates they are) to the claim that you must therefore be able to tell when someone is lying—and not just someone, everyone, or a statistically significant percentage of people most of the time. Lying isn't an emotion, first of all. It isn't even consistently associated with the same emotions for everybody in a given culture.

Also, everybody lies. Even if you were somehow able to isolate "dishonesty" from the myriad competing expressions, circumstances and motivations playing across our faces at any moment, what then? Detecting a lie doesn't point you to the truth. If you pulled 100 "liars" out of the Departures line every hour you would then be faced with the impossible task of ferreting out 100 different reasons for lying. Many would be personal, some would be subconscious, and a few might be so deeply private that the very act of probing could provoke a defensive, even violent response. And maybe—probably, evidently—none would have anything to do with terrorism.

But just as culture shapes our range of emotional perceptions, it creates its own blind spots as well. Imagine the generation of young men shipped off to war in Europe who endured years of separation from their wives and girlfriends during the critical emotional bonding phase of their relationships. Communication back home from the front was limited to brief, infrequent calls made from not-so private phone booths, written letters and postcards that were probably screened by military censors but definitely self-censored because letters from overseas tended to get shared around, and who wanted to imagine the entire extended family reading his letters and being exposed to intimate details meant only for his girl?

Betty Draper, Mad Men

The combination of emotional distance and the abundance of prostitutes overseas (if we're being honest) bred a whole generation of men who couldn't tell the difference between an expression of loving contentment and one of bored tolerance in exchange for financial security—and the cultural fallout from this inability became evident in a generation of Betty Drapers, the housewives in gilded cages immortalized by The Rolling Stones in "Mothers Little Helper."

I suppose it would be fair to ask if the parents of today are breeding a generation of children who will lack the ability to differentiate between an expression of genuine interest and one of annoyed anticipation of getting back to Gears of War or America's Top Model. But it's probably too soon to say.

A final note on the inherent limitations of classification systems: In a short passage from Otras Inquisiciones, Jorge Luis Borges quotes an apocryphal taxonomy from "a certain Chinese dictionary entitled The Celestial Emporium of Benevolent Knowledge. In its remote pages it is written that animals can be divided into (a) those belonging to the Emperor, (b) those that are embalmed, (c) those that are tame, (d) pigs, (e) sirens, (f) imaginary animals, (g) wild dogs, (h) those included in this classification, (i) those that are crazy-acting (j), those that are uncountable (k) those painted with the finest brush made of camel hair, (l) miscellaneous, (m) those which have just broken a vase, and (n) those which, from a distance, look like flies."

In part two, I'll look at how AI is changing the landscape of emotion science.