Support Provided ByLearn More
Tech + EngineeringTech & Engineering

Government Facial Recognition Plans May Be Hindered by Neuroscience

ByTim De ChantNOVA NextNOVA Next

Receive emails about upcoming NOVA programs and related content, as well as featured reporting about current events through a science lens.

The U.S. government is developing an advanced system to recognize people’s faces in a crowd. The plan, which was revealed in documents obtained by a Freedom of Information Act request, comes in the wake of National Security Administration controversy and has privacy advocates worried. Here’s Charlie Savage, writing for the New York Times:

Support Provided ByLearn More

Significant progress is already being made in automated face recognition using photographs taken under ideal conditions, like passport pictures and mug shots. The Federal Bureau of Investigation is spending $1 billion to roll out a Next Generation Identification system that will provide a national mug shot database to help local police departments verify identities.

But surveillance of crowds from a distance — in which lighting and shadows vary, and faces tend to be partly obscured or pointed in random directions — is still not reliable or fast enough. The BOSS research is intended to overcome those challenges by generating far more information for computers to analyze.

The current limitations of facial recognition were very publicly exposed this spring during the Boston Marathon. Suspects Tamerlan and Dzhokhar Tsarnaev were captured on grainy but passable surveillance footage. While humans had no problem identifying the individuals in the frames, computers couldn’t turn up a match.

The U.S. government is trying to develop a facial recognition system that's 80-90% accurate at 100 meters.

BOSS will be much more than a run-of-the-mill surveillance system, though. It will use visible light and infrared cameras at multiple angles along with distance sensors to create 3D renderings of people’s faces in a crowd. As proposed, these renderings would then be compared against a terrorist watch list. Privacy advocates are concerned that BOSS’s mission will be extended to include driver’s license photos, allowing law enforcement to track potentially hundreds of millions of individuals.

BOSS, however, doesn’t appear to be ready. At present, it can’t recognize faces at 80 to 90% accuracy from 100 meters, the proposal’s benchmark. The system is still a work in progress, and it’s entirely possible that it will take five years or more for it to be released. Advances in computing will certainly help, but there’s another, more subtle limitation that has been holding facial recognition back—we still don’t fully understand how humans recognize faces.

As I reported here on NOVA Next last April, neuroscientists and psychologists are still grappling with the nuances of human facial recognition:

Humans […] understand the face holistically, Gabrieli says. We don’t break it down into its component parts, analyzing a nose’s size, shape, and color, for example. In fact, we’re terrible when we try to do it that way. “If you show a nose on a face, people do way better at recognizing that nose when it comes back with a face as opposed to the nose by itself,” Gabrieli says. An analogy, he adds, is proofreading. When we’re reading over a passage we just wrote, we’ll often miss typos because we don’t read letter by letter—we understand the word as a whole, every letter simultaneously. “Faces are like that with a vengeance.”

Until we understand in more detail how humans pick a face out of a crowd, it’s entirely possible that computers will be unable to do the same, at least with a high degree of accuracy. The benchmarks that BOSS has to meet may too stringent for the program to succeed, at least in the near future.