[an error occurred while processing this directive]

[an error occurred while processing this directive] [an error occurred while processing this directive]
PBS Ombudsman

The Mailbag: Did Frontline Smudge Its Fingerprint?

On April 17, PBS's top investigative series, Frontline, took an hour-long look at other investigative series; not the fictional ones on television but rather the real ones that go on in courtrooms. The program was called "The Real CSI" and correspondent Lowell Bergman introduced it by saying, "CSI is the most watched drama series in the world, brimming with flash and glamour, where cutting-edge forensic technology always reveals the truth. But crime scene investigations in the real world are rarely so simple."

As a viewer, I liked that introduction and it pulled me into the program. I don't spend my days and nights watching television, but I do watch a handful of legal/crime dramas and one of the things that has bothered me for years now is that the old Sherlock Holmes or Inspector Columbo type of shoe-leather plus intellect sleuthing now gives way almost always to some awesome new technological tool that spares the writer the need to figure things out.

Fingerprints are hardly an awesome new technological tool. As Bergman quickly points out, they are "one of the most widely used of all the forensic techniques. For over a century, fingerprints have been used to identify criminals, from petty thieves to international terrorists." So far, so good. We all know that.

But are they fool-proof, infallible? Is there any real scientific basis for such claims? The viewer of this program can quickly tell that Frontline believes the answer to those questions is "no" and is going to make certain in the next 50 minutes that you get it.

I did not receive a great deal of mail about this program, but I did get a handful of long and detailed critiques from viewers who felt the program projected "an obvious bias against all forensic sciences (except for DNA) . . . deliberately chose a one-sided message . . . ignores more recent improvements . . . [and was] intent on tearing down the validity of the entire industry and scaring viewers."

A Warning to Those Who Are Not Fans of CSI

What follows is a very long and detailed exchange between viewers who take sharp exception to the Frontline reporting and presentation, and producers who, to their credit, took the time to respond to these charges, also in lengthy detail. If you are not really interested in this stuff, you may want to catch a Downton Abbey re-run or something else.

I thought this was a very good exchange and, hence, worth the time and space to present to readers of this column despite the length. My own sense of the program, as a viewer not a reviewer, was that you could quickly tell that it had an agenda; that you were going to find out that fingerprinting is not infallible, that its scientific foundation has been questioned, and hear about some dramatic cases (Brandon Mayfield in 2004 and Levon Brooks in 1990) where fingerprint matching and other forensic evidence — bite marks in the case of Brooks — went seriously wrong and contributed to big miscarriages of justice.

On the other hand, the program presents specialists, such as the executive director of the National District Attorney's Association, that point out that it is the rare case that goes wrong, that fingerprint evidence is a vital part of the criminal justice system and that there are hundreds of thousands of cases where it does work and where "good forensic scientists testify."

Indeed, the program concludes with a former Florida state attorney stating that the system is overwhelmingly accurate and warning that "to take a small number of cases or the opinions of a few people, and take that as an indictment of the system of forensic science, does a disservice to not only forensic science but to justice."

So this did not come across to me as an investigative piece without some balance, even though the overall impression it leaves is that this fingerprint technique that we all grew up believing was a sure gotcha for perps has at least some serious points to question. I don't think this is going to somehow make people less confident that fingerprinting remains a valuable and normally reliable crime-fighting tool. I think its content can be intelligently absorbed.

But it does raise a yellow caution flag about forensic evidence generally, aside from DNA, that, to me, felt like a real public service, illuminating cases that I was grateful to be reminded of and shining a light on a system that we take for granted but, like all systems we've gotten used to, will benefit from scrutiny.

Here Are the Letters

I am writing today to express my disappointment in the Frontline special regarding forensic sciences that aired in mid-April. I have always regarded PBS as one of the few stations that reports trustworthy news and educational programs. However, the Frontline special in question was anything but well-rounded or fair. An obvious bias against all forensic sciences (except for DNA) was detectable very early in the programming. In the case of fingerprint examination, for example, only one current fingerprint examiner was interviewed and anything she had to say in support of her industry was cut out completely.

Sweeping accusations such as 'fingerprint identification has no scientific validity' were allowed to be presented, and yet no one spoke of the hundreds of years of scientific research that HAS been done and DOES support fingerprint uniqueness. Not to mention the fact that such accusations were made by people who have no training in fingerprint identification, so therefore are simply uneducated about the topic.

In addition, Frontline spoke to students at UC Berkeley regarding what it takes to become a 'forensic expert.' One particular student paid a few hundred dollars, took an online test, and received a certificate as a Forensic Consultant. While it is true that people can accomplish such a feet from that particular school, Frontline never addressed the fact that 1) Most people in the forensic community had never heard of the school; 2) just because someone gets an online certificate doesn't mean they will be hired or allowed to testify in court; and 3) Criminalists who are actually employed with government-run laboratories have a Bachelor's Degree in a hard science, may have an additional 2 years of schooling just in forensics, and also typically complete a 1-2 year in-house training program.

The general public has a right to ask questions about forensics, and I think that they always should. When they do, I had hoped that stations such as PBS would offer a neutral standpoint that aimed at being educational. Instead, what came out was an overly biased program that was intent on tearing down the validity of the entire industry and scaring the viewers, all without ever conducting truly in-depth research into how the industry works.

San Mateo, CA

A Detailed Critique from Illinois

The Frontline show on forensics was certainly a good example of journalism that deliberately chose a one-sided message, that forensic science is untrustworthy, and then stuck to that message, consistently ignoring any evidence for an opposing viewpoint. There are many individuals who could present a realistic picture of forensics. Christophe Champod from Switzerland could have provided an academic authority outside of the US. The Scientific Working Group on Friction Ridge Analysis Study and Technology is a recognized group that meets regularly to make recommendations for latent print work. Learning about the SWG groups, the International Association for Identification, the American Academy of Forensic Sciences, and other organizations is easily accomplished via a Google search.

There is a story behind the Mayfield error. The error was directly responsible for several comprehensive investigations of the FBI's philosophy and procedures. The Mayfield error was discovered in May 2004. In June of 2004 an international panel of experts convened to examine the causes of the error. Following their report the FBI assembled eight internal review teams who produced 156 recommendations for change. The Office of the Inspector General began their own investigation in September 2004 and published a report containing many recommendations. In 2011 the OIG published a follow-up report called "A Review of the FBI's Progress in Responding to the Recommendations in the Office of the Inspector General Report on the Fingerprint Misidentification in the Brandon Mayfield Case."

Post-Mayfield Progress

This report reviews the progress and many changes that have occurred since Mayfield. This is a real newsworthy story that was ignored. These changes have extended beyond the FBI to the latent print community throughout the US as is evidenced in the recommendations of SWGFAST. The amount of time the program devoted to the Mayfield case suggested that huge egregious errors such as this happen often. The fact that thousands of cases are adjudicated successfully every day might indicate that the US has thousands of incarcerated innocents. Although the "ground truth" is not known in court cases, well done, peer-reviewed research studies on error rates in latent prints have been done. These were well designed and carried out. These studies use real latent print examiners in a realistic setting. They have been published in peer-reviewed journals and presented at professional meetings.

These show a very low error rate for erroneous identifications and a slightly higher rate for "misses" (failure to detect an identification). The FBI has a Counterterrorism and Forensic Science Research Unit [that] provided a study published in 2010 called the "Accuracy and Reliability of Forensic Latent Print Decisions." How could your research fail to come across this document and others? This well designed, large study found an error rate of 0.1% for false identifications and 7.5% for "misses" (failure to identify). These statistics do not fit the program's premise that all forensic scientists are charlatans, but would have provided some measure fairness to your subject.

This program chose a real-life experiment where a student was able to get a meaningless on-line certificate in "forensic consultancy." It is unclear what significance such a title carries because that information wasn't presented. The program suggested all forensic fields are bereft of legitimate professional organizations. The International Association for Identification provides rigorous certification tests for Latent Print Examiners and Crime Scene Analysts. No untrained individual would be able to pass these tests. The IAI website discusses the fail rate for test takers. For latent prints, the eight hour, three-part proctored exam has a pass rate of about 60%. Examiners who do fail the test typically do so because they can't complete it in the allotted time period, not because they have made erroneous identifications. Certification also requires re-testing at intervals and proof of continuing education.

The Letter Concludes

Dr. Dror's study of five individuals is one of the smallest studies ever conducted on latent print examiners. The premise of the study has no connection to everyday latent print examination. A sample size of five cannot be extrapolated to thousands of examiners. In a real life comparison, the examiner seldom has information on the individuals whose standards are being used for comparison. The question posed is: was the latent print made by the individual whose fingerprints appear on the standard? In Dror's study, that question is absent. The examiners were told that they are looking at the erroneous Mayfield latent. Four of the five examiners believed the information they were given and concluded the comparison was not identification. One examiner chose to ignore that information and identify the print. These results demonstrate only how much belief the examiner had in the information he/she was given.

This experiment is not even well designed to demonstrate bias. In an experiment in which bias is the treatment (the variable that is to be studied) there should be a control group. One group receives the biasing information and one group does not receive the biasing information. So in addition to lacking external reality, the sample size of the study is too small, and there is no control group. Dr. Dror seems to believe he has demonstrated a huge problem in bias. This can be an important issue in latent print work and his initial study was heuristic but certainly not well done or externally valid. Frontline used a poorly researched and biased report to generate ratings but has lost validity and legitimacy with all those who have more complete knowledge of this topic.

Mary McCarthy, Murphysboro, IL

And More . . .

As a latent print examiner, I can say that this program was entirely slanted towards the view that forensic science (and more specifically, latent print examination) is not valid science. There was no attempt to delve into or even address the scientific basis for fingerprint identification. None of the standardizing entities (the International Association for Identification, which certifies latent print examiners and crime scene investigators among others, and SWGFAST, which is attempting to set standards for friction ridge skin examination) was represented or even referenced. A cognitive psychologist who does not do latent print work was called an "authority on fingerprint analysis." It was an extremely slanted program. I expected better from Frontline.

Maryville, IL

~ ~ ~

I want to commend this week's Frontline on the topic of [sometimes] scientific evidence. Harry Edwards is the absolute best when it comes to clarity.

Detroit, MI

Frontline Responds

We have reviewed the concerns raised by some viewers regarding FRONTLINE's reporting on forensic sciences in the program entitled "The Real CSI" that aired on April 17, 2012. We'd address those concerns this way:

The viewer from San Mateo wrote that FRONTLINE made "sweeping accusations" such as "fingerprint identification has no scientific validity." FRONTLINE's reporting on fingerprints was guided by questions about the reliability of the identification process, rather than the scientific validity of the underlying assumptions. The closest the film comes to addressing the fingerprint science is in a question on camera from Lowell Bergman to Judge Donald Shelton about the science behind fingerprint analysis:

BERGMAN: "Wait a second — there's no scientific basis for matching, like a partial fingerprint?"

Judge Shelton's response was that he was unaware of any study demonstrating that no two people have the same fingerprint. He did not say that there is no scientific validity to fingerprint identification.

The NAS Report

FRONTLINE cited a National Academy of Sciences (NAS) report that found in general that many of the forensic sciences "have never been exposed to stringent scientific scrutiny" and "do not meet the fundamental requirement of science."

The NAS report does go on to raise doubt about fingerprint identification, stating: ". . . the scientific foundation of the fingerprint field has been questioned, and the suggestion has been made that latent fingerprint identifications may not be as reliable as previously assumed." It was this premise — that fingerprint identification may not be as reliable as previously assumed — that guided our investigation.

The idea of fingerprint uniqueness is the foundation of fingerprint analysis. It is what allows examiners to make positive identifications of individuals. FRONTLINE did not question the idea of uniqueness and instead focused on the process by which fingerprints are compared. Judge Shelton's response to Bergman explains:

SHELTON: "But even more important, how much alike do they have to be before you say that that fingerprint came from this person? What is the standard for how many points of comparison?"

This point was reiterated by Jennifer Mnookin of UCLA:

MNOOKIN: What matters here isn't: are your fingerprints really different from that guy over there? The real question is, is some part of your fingerprint sufficiently similar to some part of his that a competent examiner might mistake some part of your print for a part of somebody else's print.

The National Academy of Sciences also took the position that uniqueness is not the critical issue:

"The question is less a matter of whether each person's fingerprints are permanent and unique — uniqueness is commonly assumed — and more a matter of whether one can determine with adequate reliability that the finger that left an imperfect impression at a crime scene is the same finger that left an impression (with different imperfections) in a file of fingerprints."

Establishment Groups

The fingerprint examiner field is one of the largest and most established of any across the forensic disciplines. As the viewer from Murphysboro, Illinois, mentioned, two of the largest organizations are The Scientific Working Group on Friction Ridge Analysis Study and Technology (SWGFAST), the International Association for Identification (IAI).

FRONTLINE producers were familiar with these two organizations from their reporting. They found that the recommendations of the organizations closely match those of the Federal Bureau of Identification. We chose to focus on the FBI because it includes what is widely considered to be the top fingerprint laboratory in the country, which sets the standards in for the field.

This same viewer from Illinois also commented on our focus on the erroneous fingerprint analysis in the Brandon Mayfield case, writing: "The amount of time the program devoted to the Mayfield case suggested that huge egregious errors such as this happen often."

FRONTLINE dedicated a large portion of the program to the Mayfield case because it was a watershed moment that changed the way that the FBI operates. It also led SWGFAST and the IAI to change their recommendations on how to testify regarding fingerprint identification.

The frequency of similar mistakes is a matter of debate and was treated as such in the film. Scott Burns, of the National District Attorney's Association says:

BURNS: "The Mayfield case is the anomaly. It is the rare exception."

To which Itiel Dror, a neuroscientist who has studied fingerprint analysis, responds:

DROR: "What is an anomaly is that they found out. Not that they made a mistake."

FRONTLINE was careful not to report any degree of certainty about how often mistakes occur. Simon Cole, who has documented several dozen fingerprint errors, has suggested that mistakes only come to light because of extraordinary circumstances — the Mayfield error being the most extraordinary of all involving a separate international agency.

The viewer from Murphysboro also refers to the study "Accuracy and Reliability of Forensic Latent Print Decisions," which found an error rate of 0.1% for false identifications and 7.5% for false negatives. The viewer wrote: "these statistics do not fit the program's premise . . . but would have provided some measure of fairness to your subject."

We did consider including the study, but to make quite a different point. While a 0.1% false identification rate — meaning that there is a one in one thousand chance that a Mayfield mistake could occur — is relatively low, approximately 250,000 fingerprint identifications are made every year. False identifications could be happening every day.

The viewer from Murphysboro also writes that "the program suggested all forensic fields are bereft of legitimate professional organizations" and cites the IAI as having a rigorous exam with a 60% pass rate.

While it may be true that no untrained individual would be able to pass the IAI test, as the viewer notes, fingerprint examiners are not required to pass the exam to perform examinations. This holds true in laboratories across the country. In other words, while the IAI certification could be considered the minimum standard for fingerprint examiners, nearly half of those who take the test fail and it is not considered a requirement to practice. There are other forensic organizations similar to the IAI, but they are all voluntary and do not perform oversight of their members or the quality of their work. Judge Harry T. Edwards, the Chairman of the NAS committee and a primary author of the report, said this in the film:

EDWARDS: There are certifiers, but it's not what you and I are talking about, that is, real licensing programs, real certification programs that train, gives serious tests, and will revoke your license and affect your job and ability to testify in the event that you do something wrong or fail. No. That doesn't exist now.

As noted in the film, the NAS is recommending mandatory certification for all experts.

Finally, the viewer from San Mateo took issue with the example of Leah Bartos receiving a Certified Forensic Consultant certification from the American College of Forensics Examiners Institute (ACFEI). The viewer says that most people in the forensic community have never heard of the program and that "just because someone gets an online certificate doesn't mean they will be hired or allowed to testify in court."

ACFEI claims to be the largest forensic membership organization in the country. Although we could not verify this claim, they appear to have in excess of ten thousand members. If forensic scientists themselves are unaware of the largest organization in the field, how are judges, lawyers and juries expected to evaluate the legitimacy of the credentials they provide?

In the film, Judge Shelton explains the issue of credentials this way:

SHELTON: "It has an inordinate impact on jurors because jurors have no way of knowing whether this certifying body exacts scientific standards or is just a diploma mill."

Finally . . .

And while it is true, as the viewer states, that certificates do not guarantee that an individual will be allowed to testify in court (as the film makes clear, this is solely up to the discretion of the judge), credentials can create the appearance of expertise. Whether or not Leah Bartos, or anyone else with a certification from ACFEI, would be able to testify is not the central issue. What the example illustrates is that obtaining credentials does not require the level of study, experience or expertise that reasonable people might expect a credentialed authority to possess.

FRONTLINE and its producers spent nearly a year investigating the reliability of forensic science for this program, interviewing dozens of experts in the field and reviewing thousands of documents. We believe our reporting and presentation of the facts has been thorough, fair and accurate.

[an error occurred while processing this directive]