Google says its new AI models can recognize emotions – and that’s worrying experts

Google says its new family of AI models has a curious feature: the ability to “identify” emotions.

Announced Thursday, the PaliGemma 2 family of models can analyze images, allowing AI to create captions and answer questions about people it “sees” in photos.

“PaliGemma 2 generates detailed, contextually relevant captions,” Google wrote in a blog post shared with TechCrunch, “and goes beyond simple object identification to describe actions, emotions, and the overall narrative of the scene.”

Emotion detection does not work out of the box and PaliGemma 2 needs to be fine-tuned for this purpose. Still, experts TechCrunch spoke to were concerned about the prospect of a publicly available emotion detector.

“This worries me very much,” Sandra Wachter, a professor of data ethics and AI at the Oxford Internet Institute, told TechCrunch. “I find it problematic to assume that we can ‘read’ people’s emotions. It’s like asking a Magic 8 Ball for advice.”

For years, both startups and tech giants have been trying to develop AI that can recognize emotions for everything from sales training to accident prevention. Some claim to have achieved it, but science stands on shaky empirical foundations.

Most emotion detectors are modeled on the early work of Paul Ekman, a psychologist who theorized that people share six basic emotions: anger, surprise, disgust, joy, fear and sadness. However, subsequent studies challenge Ekman’s hypothesis and show that there are major differences in the way people from different backgrounds express their feelings.

“Emotion recognition is generally not possible because people experience emotions in complex ways,” Mike Cook, a research fellow at Queen Mary University who specializes in AI, told TechCrunch. “Of course we believe we can tell what other people are feeling by looking at them, and many people have tried over the years, such as spy agencies or marketing companies. I’m sure that in some cases it’s certainly possible to recognize some generic signifiers, but we can never fully “solve” this.

The unsurprising consequence is that emotion recognition systems tend to be unreliable and influenced by the assumptions of their developers. In a 2020 MIT study, researchers showed that facial analysis models can develop unintended preferences for certain expressions, such as smiling. Recent work suggests that emotional analysis models attribute more negative emotions to the faces of black people than to the faces of white people.

Google says it conducted “extensive testing” to assess demographic bias in PaliGemma 2 and found “low levels of toxicity and profanity” compared to industry benchmarks. However, the company did not provide the full list of benchmarks used, nor did it specify what types of tests were performed.

The only benchmark published by Google is FairFace, a series of portrait photos of tens of thousands of people. The company says PaliGemma 2 performed well at FairFace. However, some researchers criticized the benchmark as a bias measure, noting that FairFace only represents a handful of racial groups.

“Interpreting emotions is a fairly subjective matter that goes beyond the use of visual aids and is deeply embedded in a personal and cultural context,” said Heidy Khlaaf, senior AI scientist at the AI ​​Now Institute, a nonprofit organization that influences society Effects of artificial emotions examine intelligence. “Aside from AI, research has shown that we cannot infer emotions from facial features alone.”

Emotion recognition systems have drawn the ire of regulators abroad who have sought to limit the technology’s use in high-risk contexts. The AI ​​Act, the most important AI law in the EU, prohibits schools and employers from using emotion detectors (but not law enforcement agencies).

The biggest fear with open models like PaliGemma 2, available from a number of hosts including AI development platform Hugging Face, is that they will be misused or abused, which could lead to real-world harm.

“If this so-called ’emotional identification’ is built on pseudoscientific assumptions, this has significant implications for how this ability can be used to further – and falsely – discriminate against marginalized groups, for example in law enforcement, recruiting, border management, etc so on,” Khlaaf said.

Asked about the dangers of releasing PaliGemma 2 to the public, a Google spokesperson said the company stands behind its testing for “display damage” as it relates to visual question answering and captioning. “We have conducted robust assessments of the PaliGemma 2 models in terms of ethics and safety, including child safety and content safety,” they added.

Watcher isn’t convinced that’s enough.

“Responsible innovation means thinking about consequences from the first day you walk into your lab and continuing to do so throughout a product’s lifecycle,” she said. “I can imagine countless potential problems (with models like this) that could lead to a dystopian future in which your feelings determine whether you get the job, a loan and admission to college.”