Ask The Headhunter: Big data in HR means big problems for job seekers

Relying on big data about job candidates doesn't make for a good hiring process, says Nick Corcodilos. Photo by Flickr user Infocux Technologies.

Relying on big data about job candidates doesn’t make for a good hiring process, says Nick Corcodilos. Photo by Flickr user Infocux Technologies.

Nick Corcodilos started headhunting in Silicon Valley in 1979, and has answered over 30,000 questions from the Ask The Headhunter community over the past decade.

In this special Making Sense edition of Ask The Headhunter, Nick shares insider advice and contrarian methods about winning and keeping the right job, on one condition: that you, dear Making Sense reader, send Nick your questions about your personal challenges with job hunting, interviewing, networking, resumes, job boards, or salary negotiations. No guarantees — just a promise to do his best to offer useful advice.

Last week, I appeared on Brian Lehrer’s CUNY-TV news show to discuss how human resources departments are misusing “big data” (aka, “people analytics”) to recruit, hire and watch employees. I think the topic deserves a lot more exploration and discussion.

Are you frustrated because employers reject your job application out of hand without even talking to you? Tired of online application forms kicking you out of consideration because you took too long to answer questions or because you failed to disclose your salary history?

Wait — America’s employment system is getting even more automated and algorithm-ized by applying “big data” to process you. According to a new report in The Atlantic (“They’re Watching You At Work” by Don Peck), the vice president of recruiting at Xerox Services warns that “We’re getting to the point where some of our hiring managers don’t even want to interview anymore.” According to the article, “they just want to hire the people with the highest scores.”

Does that worry you?

I’m worried about the claims made by vendors that promote the use of big data “tools” in the employment process — and about some of the conclusions Peck draws in his article. According to, big data refers to new methods of gathering, processing and analyzing information, primarily for marketing purposes. (See “For CMOs, ‘Big Data’ Is A Very Big Deal.” It’s perhaps no accident that big data is very big in marketing circles today.)

But I think there are some big problems with human resources (HR) departments using big data to assess people for jobs and on the job.

The Metrics Are Indirect
The vendors behind these “tools” don’t directly assess whether a person can do a job. Instead, they look at other things — indirect assessments of a person’s fit for a job. For example, they have you play a game and they measure your response times. From this, they try to predict success on the job. That determines whether you get interviewed.

The Problem: We’ve known for decades that this approach doesn’t work. Wharton researcher Peter Cappelli throws cold water on indirect assessments: “Nothing in the science of prediction and selection beats observing actual performance in an equivalent role.”

All that’s being thrown into the mix by these “assessment” vendors is big data — lots of extraneous factoids about people and their behavior that have nothing to do with the tasks required in a job. But more data about irrelevant behaviors don’t make better predictions. In fact, it makes things worse if the data are not valid predictors of success. It’s worse because indirect assessment leads to false negatives (employers reject potentially good candidates) and to false positives (they hire the wrong people for the wrong reasons).

The Conclusions Are Based on Correlations
These tools predict success based on whether certain characteristics of a person are similar to characteristics of a target sample of people. For example, Peck’s article says that “one solid predictor of strong coding [programming] is an affinity for a particular Japanese manga site.” (Manga are Japanese comics.)

Gild, the company behind this claim, says it’s just one correlation of many. But Gild admits there’s “no causal relationship” between all the big data it gathers about you and how you perform on the job.

In what can only be called a scientific non sequitur, Gild’s “chief scientist” says “the correlation, even if inexplicable, is quite clear.”

The Problem: A basic tenet of empirical research is that a correlation does not imply causality, or even an explanation of anything. Data tell us that people die in hospitals, and that correlates highly with the presence of doctors in hospitals. All jokes aside, that correlation doesn’t mean doctors kill people. Except, perhaps, in the world of big HR data: If you’re selling “people analytics,” then your product is a game that predicts how someone will do their job.

When we pile specious correlations on top of indirect assessments (interview questions like what animal would you be if you could be any animal?), we wind up with no good reasons to make hiring decisions, and with no basis for judgments of employees.

There’s a Hidden Lesson for Recruiters in Big Data
Hanging out at a manga website doesn’t improve anyone’s ability to write good code — nor does it predict their success at work. But it might mean that a recruiter can find some good coders on that manga site — the one reasonable conclusion and recruiting tactic that none of the people Peck interviewed seems to have thought of.

I don’t think Peck wrote this article to promote “people analytics” as the solution to the challenges that American companies face when hiring, but he does seem to think the Kool-Aid tastes pretty good. I think Peck over-reaches when he confuses useful data that employers collect about employee behavior to improve that behavior, with predictions based on silly big data assumptions.

Here are a couple of highlights in the article that blinded me. Well, the assumptions behind them were blinding, anyway:

  • Spying Tells Us a Lot
    In further support of indirect assessments of employees and job applicants, Peck cites the work of MIT researcher Sandy Pentland, who’s been putting electronic badges on employees to gather massive amounts of data about their daily interactions. In other words, Pentland follows them around electronically to see what they do.

    The badges capture all sorts of information about formal and informal conversations: their length; the tone of voice and gestures of the people involved; how much those people talk, listen, and interrupt; the degree to which they demonstrate empathy and extroversion; and more. Each badge generates about 100 data points a minute.

    That’s certainly big data. But Peck notes that these badges are not in routine use at any company.

  • It’s Just a Game
    A lot of the “breakthroughs” Peck writes about come from start-up test vendors like an outfit called Knack, which creates games “to suss out human potential.”

    Knack continues to seek venture funding, and the only Knack client mentioned in the article is Palo Alto High School, which is using Knack games to help students think about careers.

    Play one of [Knack’s games] for just 20 minutes, says Guy Halfteck, Knack’s founder, and you’ll generate several megabytes of data, exponentially more than what’s collected by the SAT or a personality test.

    Again, that’s a lot of big data. But does that improve the conclusions Knack makes? Let’s draw a comparison in the world of medicine. It’s an easy one: If more megabytes of game data can be used to generate more correlations, could doctors diagnose patients more effectively by collecting bigger urine samples? That’s the logic.

    No Sale
    I don’t buy it. As a headhunter, I want to know, can you do the job?

    Some big data about employee behavior can be analyzed to good effect. For example, Peck reports that Microsoft has gathered data showing employees with mentors are less likely to leave their jobs, so Microsoft gets mentors for them. But he seems to easily confuse legitimate metrics with goofy games of correlation. And the start-up companies he profiles don’t seem to be on any leading edge — they’re mostly trying to sell the idea that big data in the service of questionable correlations makes those correlations worth money.

    The “employment testing” business is rife with questionable practices. Dr. Erica Klein, an industrial psychologists and employment testing expert, writes in “Employment Tests: Get The Edge” (available in the Ask The Headhunter Bookstore):

    Unfortunately, there are no laws against tests being stupid or annoying or a bad idea. Sometimes organizations make poor use of tests and poor choices about test vendors. Those organizations are the eventual losers since they are not choosing the best employees and may expose themselves to discrimination lawsuits. The bottom line is that anyone can create and sell a test.

    Big Deal
    We know that what Cappelli says about the science of prediction is correct. But I think Arnold Glass, a leading researcher in cognitive psychology at Rutgers University, says it best:

    It has been known since Alfred Binet and Victor Henri constructed the original IQ Test in 1905 that the best predictor of job (or academic) performance is a test composed of the tasks that will be performed on the job. Therefore, the idea that collecting tons of extraneous facts about a person (big data!) and including them in some monster regression equation will improve its predictive value is laughable.

    Recruiting and hiring are no laughing matter — or any place for wildly speculative analysis of “all the data we can consume.” While HR departments experiment on humans who need jobs, managers are regressing into beings incapable of recruiting and judging job applicants. It seems that any technology that can “process” more and more applicants is better than a few skilled humans who can recognize just a few good workers.

    One could easily make the case that employers today are afraid of making judgments and hiring mistakes, and too easily seduced by big data in the service of plausible deniability.

    Dear Readers: Have you experienced “big HR data” or “people analytics” as a job seeker or as an employer? Is big data a big headache, or is it the path to 21st century recruiting and hiring?

    Nick Corcodilos invites Making Sense readers to subscribe to his free weekly Ask The Headhunter© Newsletter. His in-depth “how to” PDF books are available on his website: “How to Work With Headhunters…and how to make headhunters work for you,” “How Can I Change Careers?”, “Keep Your Salary Under Wraps” and “Fearless Job Hunting.”

    Send your questions to Nick, and join him for discussion every week here on Making Sense. Thanks for participating!

    Copyright © 2013 Nick Corcodilos. All rights reserved in all media. Ask the Headhunter® is a registered trademark.