What Face-Reading Computer Software Can Tell Us About Our Emotions


Is it possible for computer software to understand the human face? After 10 years of research, Fernando de la Torre and his team of computer scientists, engineers, and psychologists at Carnegie Mellon University’s Human Sensing Laboratory (HSL) believe they can finally say "yes."

This spring, the HSL released a piece of software they call IntraFace to the public. Anyone with an iPhone or Android can use this tool to characterize facial features through IntraFace-powered mobile and desktop applications. For several years, the software has been tested in a wide variety of applications, including autism, depression, and driver distractedness.

“Facial expression provides cues about emotion, intention, alertness, pain and personality,” de la Torre tells mental_floss. “We wanted to make artificial intelligence and algorithm-trained computers learn to understand expression and emotion. That was the ultimate goal."


Carnegie Mellon University’s Human Sensing Laboratory

Scientists have been trying to create automated facial recognition technology as early as 1964, when scientists Woody Bledsoe, Helen Chan Wolf, and Charles Bisson first started programming a computer to identify specific coordinates of facial features taken from photographs. According to the International Journal of Computer Science and Information [PDF], Bledsoe said the unique difficulties involved with facial recognition included a "great variability in head rotation and tilt, lighting intensity and angle, facial expression, aging, etc."

The team at Carnegie Mellon University’s Human Sensing Laboratory made their breakthrough roughly two to three years ago, when the lab first identified detection of the points of the face.

"If we don’t know here the mouth or eyes are, we can’t understand anything about expression," de le Torre says. In order to create IntraFace, the HSL’s team of computer scientists had to develop algorithms to interpret changes in facial expressions in real-time while compensating for deviations in angles, positions, and image quality.

That's why, he says, their work "is a breakthrough—a big revelation in facial image analysis. The first step in detection is the image: locating the eyes, nose and mouth. The second step is classification: identifying whether the person is smiling, frowning, male, female, etc. How does the computer know that? We learn from examples. All that we do to understand faces is from examples. We use image samples, label them, and train the computers through algorithms.”

Wen-Shang Chu is an IntraFace developer and computer scientist who is developing the algorithms for understanding these expressions. “From our demo alone, we developed face tracking, where we localized facial landmarks automatically,” Chu tells mental_floss. “We taught the computers to read the faces through 49 defined points on the faces.”

Equipped with the ability to identify facial features, the program was trained to interpret them using videos of facial expressions that were manually labeled by experts, collected from data sets available through CMU and several other universities. Thousands of images and hundreds of subjects—a mix of people of Asian, Caucasian, and African descent—were part of the data set, with more increasing over time. The researchers tested and refined the software’s abilities through the images, which could be generated at 30 images per second.

“We learned that registration and facial landmark detection is an important step for facial expression analysis,” de la Torre says. “Also, we learned that is better to train with more images of different people rather than many images of the same subject to improve generalization.”


“Evolutionarily, we [humans] recognize faces and emotions on other human beings,” de la Torre says. Between the 1950s and 1990s, psychologist Paul Ekman found a set of expressions used by people all over the world. The subtle motions and placements that define facial expression were divided into the upper and lower parts of the face and associated with major muscle groups called "facial action units." Ekman developed a taxonomy for facial expression called the Facial Action Coding System (FACS), and it is often used by psychologists today.

IntraFace's algorithms are taught to use Ekman's system as well as data from newer research conducted by Du Shichuan and Aleix Martinez about compound emotions (as opposed to single, internally felt emotions, such as the happy surprise we feel at a surprise birthday party). They identified 17 compound expressions [PDF], and Intraface takes these into account.


“With algorithms we can build emotionally aware machines that will be instrumental in many domains, from healthcare to autonomous driving,” de la Torre says, and a variety of companies and organizations are interested in using facial recognition technology.

For example, an automobile company IntraFace is working with (which they declined to identify) wants to incorporate IntraFace technology into the front panel screens of cars to extract information about the driver’s expression. IntraFace can monitor if the driver is distracted and detect fatigue; an intelligent car can compensate by alerting the driver and taking control when the driver is distracted.

The developers see potential commercial uses for their technology, such as market research analysis. For example, a company would be able to monitor focus groups in a noninvasive way for previously undetectable features such as subtle smiles, attentiveness, and microfacial expressions.

But it's IntraFace's potential in the world of medicine that has the researchers most excited.


In collaboration with the Physical Medicine Group in New York City, the HSL has a proposal under review with the National Institute of Health so that IntraFace can be used in the measurement of intensity and dynamics of pain in patients.

IntraFace was also used in a clinical trial for the treatment of depression, and it was applied to help better understand the role of emotion in depression. So far, IntraFace’s interpretation of facial features can account for 30 to 40 percent of the variance in the Hamilton Depression Rating Scale, the industry standard for depression severity measurement.

In addition, the researchers in the clinical trial were able to uncover information about depression that had not yet been discovered. Predominantly, people with depression had decreased positive moods, which was expected. IntraFace helped researchers uncover that depressed patients exhibited increased expressions of anger, disgust, and contempt but decreased expressions of sadness. People with less severe depression expressed less anger and disgust, but more sadness. This study was published [PDF] in 2014 in the journal Image and Vision Computing.

“Sadness is about affiliation; expressing sadness is a way of asking others for help,” Jeffrey Cohn, a professor of psychology and psychiatry at the University of Pittsburgh and an adjunct professor in CMU’s Robotics Institute, explains to mental_floss. “That, for me, is even more exciting than being able to detect depression or severity; we’re using [IntraFace] to really learn something about the disorder.”

IntraFace is also being used to develop and test treatments for post-traumatic stress disorder, and, in fall 2015, IntraFace’s facial feature detection technology was incorporated into an iOS application called Autism & Beyond using ResearchKit, an open source framework that enables an iOS app to become an application for medical research.

Autism & Beyond was created by a team of researchers and software developers from Duke University. “We have developed and patented technology that includes the [IntraFace] design on video stimuli to create certain emotions and expressions in children, and then correlate those emotions with developmental disorders,” Guillermo Sapiro, a professor of electrical and computer engineering at Duke University, tells mental_floss. The app can potentially be used by parents to screen young children for autism and mental health challenges, such as anxiety or tantrums.

The HSL team hopes the public release of the program will spark even more uses. De la Torre is convinced that others will build on his team’s product. (The source code, however, is not distributed.)

“We want to bring this technology to the people,” de la Torre said. “We have limited resources in our studies and students. We want to bring it out there and see what kind of interesting applications people will find with IntraFace.”

Public Domain, Wikimedia Commons
13 Electrifying Nikola Tesla Quotes
Public Domain, Wikimedia Commons
Public Domain, Wikimedia Commons

The greatest geek who ever lived had more than just science on the brain. While he was alive, Nikola Tesla’s advancements were frequently and famously attributed to others. But history has shown us the magnitude of his work, a sentiment best expressed by Fiorello LaGuardia’s eulogy: “Tesla is not really dead. Only his poor wasted body has been stilled. The real, the important part of Tesla lives in his achievement which is great, almost beyond calculation, an integral part of our civilization, of our daily lives.” Here are 13 electric quotes from the legendary scientist/engineer/inventor.


“... The female mind has demonstrated a capacity for all the mental acquirements and achievements of men, and as generations ensue that capacity will be expanded; the average woman will be as well educated as the average man, and then better educated, for the dormant faculties of her brain will be stimulated to an activity that will be all the more intense and powerful because of centuries of repose. Woman will ignore precedent and startle civilization with their progress.”

—From a 1926 interview by John B. Kennedy, “When Woman Is Boss"


“... The papers, which 30 years ago conferred upon me the honor of American citizenship, are always kept in a safe, while my orders, diplomas, degrees, gold medals and other distinctions are packed away in old trunks.”

—From “My Inventions V – The Magnifying Transmitter," 1919


“There is something within me that might be illusion as it is often case with young delighted people, but if I would be fortunate to achieve some of my ideals, it would be on the behalf of the whole of humanity. If those hopes would become fulfilled, the most exciting thought would be that it is a deed of a Serb.”

—From an address at the Belgrade train station, 1892


Blue Portrait of Nikola Tesla, the only painting Tesla posed for
Public Domain, Wikimedia Commons

"We begin to think cosmically. Our sympathetic feelers reach out into the dim distance. The bacteria of the 'Weltschmerz' are upon us. So far, however, universal harmony has been attained only in a single sphere of international relationship. That is the postal service. Its mechanism is working satisfactorily, but—how remote are we still from that scrupulous respect of the sanctity of the mail bag!"

—From “The Transmission of Electrical Energy Without Wires as a Means for Furthering Peace,” 1905


“What the result of these investigations will be the future will tell; but whatever they may be, and to whatever this principle may lead, I shall be sufficiently recompensed if later it will be admitted that I have contributed a share, however small, to the advancement of science.”

—From “The Tesla Alternate Current Motor,” 1888


“That is the trouble with many inventors; they lack patience. They lack the willingness to work a thing out slowly and clearly and sharply in their mind, so that they can actually 'feel it work.' They want to try their first idea right off; and the result is they use up lots of money and lots of good material, only to find eventually that they are working in the wrong direction. We all make mistakes, and it is better to make them before we begin.”

—From “Tesla, Man and Inventor,” 1895


“Most certainly, some planets are not inhabited, but others are, and among these there must exist life under all conditions and phases of development.”

—From “How to Signal to Mars,” 1910


"When we speak of man, we have a conception of humanity as a whole, and before applying scientific methods to the investigation of his movement, we must accept this as a physical fact. But can anyone doubt to-day that all the millions of individuals and all the innumerable types and characters constitute an entity, a unit? Though free to think and act, we are held together, like the stars in the firmament, with ties inseparable. These ties cannot be seen, but we can feel them. I cut myself in the finger, and it pains me: this finger is a part of me. I see a friend hurt, and it hurts me, too: my friend and I are one. And now I see stricken down an enemy, a lump of matter which, of all the lumps of matter in the universe, I care least for, and it still grieves me. Does this not prove that each of us is only part of a whole?"

—From “The Problem of Increasing Human Energy,” 1900


Nikola Tesla, with Rudjer Boscovich's book "Theoria Philosophiae Naturalis", in front of the spiral coil of his high-voltage Tesla coil transformer at his East Houston St., New York, laboratory.
Public Domain, Wikimedia Commons

“We build but to tear down. Most of our work and resource is squandered. Our onward march is marked by devastation. Everywhere there is an appalling loss of time, effort and life. A cheerless view, but true.”

—From “What Science May Achieve this Year,” 1910


“Everyone should consider his body as a priceless gift from one whom he loves above all, a marvelous work of art, of indescribable beauty, and mystery beyond human conception, and so delicate that a word, a breath, a look, nay, a thought may injure it. Uncleanliness, which breeds disease and death, is not only a self-destructive but highly immoral habit.”

—From “The Problem of Increasing Human Energy," 1900


"It will soon be possible to transmit wireless messages around the world so simply that any individual can carry and operate his own apparatus."

From Popular Mechanics via the New York Times, October 1909


"Let the future tell the truth and evaluate each one according to his work and accomplishments. The present is theirs; the future, for which I really worked, is mine."

—As quoted in Tesla: Man Out of Time, by Margaret Cheney, 2001


“Life is and will ever remain an equation incapable of solution, but it contains certain known factors.”

—From “A Machine to End War,” 1935 [PDF]

Gut Bacteria Could Be Keeping You Up at Night

The bacteria in your gut do far more than help digest food. In recent years, scientists have discovered that they play an important role in myriad bodily processes, from mood and mental health to obesity and gastrointestinal disease. According to recent research, the trillions of microbes in your gut could also impact how you sleep, The Guardian reports.

Though investigation into the links between sleep and intestinal bacteria is just beginning, scientists already know that lack of sleep takes a toll on the body beyond just causing fatigue. It may contribute to the risk of obesity and developing type 2 diabetes. However, digestive processes may themselves affect sleep, scientists now suggest. "There is no question in my mind that gut health is linked to sleep health, although we do not have the studies to prove it yet," psychologist Michael Breus told The Guardian.

A study in Frontiers in Behavioral Neuroscience found that rats fed a prebiotic diet (consisting of fiber that gut bacteria can feed on) had better-quality sleep than rats fed a control diet. The researchers linked this better sleep to increases in the gut bacteria Lactobacillus rhamnosus, a popular probiotic strain. The rats spent more time in REM sleep even when they were subjected to stress, which has been linked to insomnia issues.

To demonstrate how the microbiome affects sleep, though, researchers will likely have to untangle it from the many other ways that the microbiome affects our health, mental and otherwise. Imbalances in gut bacteria might influence depression, which in turn disrupts sleep. Other studies have suggested that poor-quality sleep affects the microbiome, rather than the other way around. Given how much impact the microbiome has on our health, it makes sense that there could be links between major health issues like insomnia and our bacterial colonies. The nature of those links, though, will require much more research to tease out.

[h/t The Guardian]


More from mental floss studios