Anima Anandkumar

Former Principal Scientist, Amazon Web Services

Anima Anandkumar served as principal scientist for Amazon Web Services from 2016 to 2018. She is currently the director of machine learning at NVIDIA and is a Bren Professor of Computing and Mathematical Sciences at California Institute of Technology.

Following are excerpts of an interview with Anima Anandkumar conducted by James Jacoby on August 27, 2019.

This interview appears in:

Amazon Empire

Text Interview:

Highlight text to share it

Ethical Concerns Around Facial Recognition Software

And what precipitated the letter that you signed onto about the use of facial recognition and selling the service to law enforcement?

So I signed the letter asking Amazon to stop selling face recognition and related software to law enforcement.This was after I left Amazon, and, you know, I was seeing how the landscape is evolving, right?Like, I said, in the early days, it was really up to companies to think about first building the applications and getting to the market, but as the technology matures, I think it’s also the responsibility for the companies to ensure that these are well designed, are getting used in the context that it’s meant to be—that it’s designed for.And what I saw was the study from MIT Media Lab called “Gender Shades,” and Joy [Buolamwini] and Deb Raji, who were the principal authors for the study, showed that as, you know, you made the skin tone darker in a person as well as changed the gender, especially for women, there was a huge drop in accuracy in face recognition and other facial attribute software—not only by Amazon, from many other cloud providers.And so that study showed that these applications that are launched to the public have many biases in-built.

Flaws?

Yeah, I would say flaws, because there is potentially many ways to fix these biases.You know, you could either now take in data that is more balanced, right?So a lot of reasons these face recognition software are biased is because the data we’ve collected is heavily imbalanced in terms of the minorities.You know, much of it is also historic.The earliest research papers that did face recognition used what we call a celebrity dataset, so these were faces just from the internet that you could easily, you know, obtain, and that’s heavily of the white race and not from the other races.And there’s also other biases, like women wearing makeup in those images versus in real life women not wearing makeup.So there’s bias in the existing datasets on which many of these algorithms are trained, and that’s being released and being used by law enforcement without I think enough accountability and auditing.

So when the MIT study came out, how was the Amazon response?

So the Amazon response, the official response by Amazon was to say that, you know, some aspects of the study were on facial analysis and not face recognition, and they are two separate applications on the Amazon cloud.And that was one of the reasons we wanted to clarify that in the letter that I signed along with many other researchers who said that the two are related, right?Even if it’s not the same application hosted on the cloud, but the performance of the two will be very related.

So were you frustrated by Amazon’s response to the study?

I was, because there’s a lot of scientific material available on how facial analysis and face recognition are related.For instance, the “Gender Shades” study showed that some of the famous female black celebrities were mistaken as male, right, like Michelle Obama, Serena Williams, and all these examples that show that there is a huge bias in the existing software.And Amazon’s response was that “Oh, that’s not face recognition; that’s gender analysis.”

And is that inaccurate?

So they are related, right?In fact, gender recognition is an easier task than recognizing the identity of the person.You know, you have to first get the gender right, and then only you can hope to get the identity of the person right.And so if it’s doing so poorly on an easier task, that shows that there is still a lot of room for improvement in terms of how the software performs on minority populations.

You've said that this—that they are releasing products, AI products, that are not battle-tested or battle-ready.What does that mean?What do you mean by that?

So I believe many of the AI applications released to the public is not battle-tested, right?By that I mean—especially applications that involve pre-trained models assume a certain kind of data distribution.They are trained on the input data that the public or the auditors do not see, so we do not understand what are the biases that are built into the system.We haven’t tested them under harsh conditions.Let’s say the law enforcement wants to use this on pictures that are taken with poor lighting or all kinds of challenges, but if the training data was based on say, celebrity data with a lot of well-lit pictures, we can’t expect that to perform as well on the real-world scenarios.And so it’s very important to understand in what context and what kind of environment will this application be deployed and then test that performance in that environment.

So what—what’s so problematic about law enforcement using this?

So there have been—so the use of AI applications by law enforcement is problematic, in my view, because it can make the historic biases even worse, right?If there is already biases against certain minorities in the system, and then we have AI applications that’s also trained with the similar biases, then overall the biases get further augmented, and that can be, in fact, having a huge cost on human lives in many cases.

… Were you surprised when Amazon, after the MIT study comes out, were you surprised that Amazon actually tried to kind of debunk it or dismiss it?Were you—were you surprised by that response?

I was disappointed by Amazon’s response to “Gender Shades” because in my view, it wasn’t grounded in solid science.Facial analysis and face recognition are indeed related.For instance, gender recognition is an easier task compared to face recognition, right, and if a system is doing poorly in gender recognition on black women, that implies that face recognition could also be having poorer—poor accuracy on such examples, so—

So was—was Amazon’s response to that study bogus?

So I would say that Amazon’s response to that study was not completely accurate.

And that upset you?

Indeed, yes.I—I had hoped for a response that would involve the scientists from Amazon and that would be following the scientific method, that would have performance numbers, right?That—that would have been a more appropriate response.

And did that prompt you to sign on to this letter to Jeff Bezos?

Indeed it did, because to be in my personal capacity, I think it’s my duty to inform the public about not just the benefits of AI, but also its dangers and potentially badly designed or flawed uses of AI.

Latest Interviews

Get our Newsletter

Stay Connected

Jon and Jo Ann Hagler on behalf of the Jon L. Hagler Foundation

Koo and Patricia Yuen

Funding for FRONTLINE is provided through the support of PBS viewers and by the Corporation for Public Broadcasting. Additional funding is provided by the Abrams Foundation; Park Foundation; the John D. and Catherine T. MacArthur Foundation; and the FRONTLINE Journalism Fund with major support from Jon and Jo Ann Hagler on behalf of the Jon L. Hagler Foundation, and additional support from Koo and Patricia Yuen. FRONTLINE is a registered trademark of WGBH Educational Foundation. Web Site Copyright ©1995-2025 WGBH Educational Foundation. PBS is a 501(c)(3) not-for-profit organization.