Criminal Sentencing Algorithm No More Accurate Than Random People on the Internet

<section class="entry-content instapaper_body clearfix" itemprop="articleBody">
<div class="add-container clearfix" id="add-block-placer">
</div>
<p>
  An “unbiased” computer algorithm used for informing judicial decisions appears to be no better than the assessments of a random group of people, according to
  <a href="http://advances.sciencemag.org/content/4/1/eaao5580">
   a recent study.
  </a>
  What’s more, the algorithm appears to issue racially biased recommendations.
 </p>
<p>
  COMPAS, the software that many judges use to inform their sentencing decisions, tends to classify black people as higher risk and white people as lower risk, despite not including explicit information about race. In practice, this translates into more lenient rehabilitation suggestions for white defendant and more rigorous programs black defendant of the same recidivism risk.
 </p>
<figure class="richtext-image full-width">
<img alt="" class="richtext-image full-width" src="https://www.pbs.org/wgbh/nova/media/images/130821-f-eb935-051.width-990_rxZvJIM.jpg"/>
<figcaption>A study of 1,000 defendants suggests that an algorithm used by judges in several states may be wrong a third of the time.
        
    </figcaption>
</figure>
<p>
  The results—and the bias—were statistically indistinguishable from judgement calls made by human volunteers randomly selected over the internet.
 </p>
<p>
  Here’s
  <a href="https://arstechnica.com/science/2018/01/random-people-as-good-as-judicial-software-at-predicting-future-arrests/">
   John Timmer
  </a>
  writing at Ars Technica:
 </p>
<blockquote>
<p>
   The significance of that discrepancy is still the subject of some debate, but two Dartmouth College researchers have asked a more fundamental question: is the software any good? The answer they came up with is “not especially,” as its performance could be matched by recruiting people on Mechanical Turk or performing a simple analysis that only took two factors into account.
  </p>
</blockquote>
<p>
  The racial bias likely creeps into the COMPAS algorithm through data on arrest rates, which in some cities and counties are skewed. Equivalent, the developer of COMPAS, says it relies on 127 different data points when determining rehabilitation programs, but only six when assessing whether an individual is at risk of reoffending.
 </p>
<p>
  The new study builds on an investigation by Pro Publica, which analyzed COMPAS’s performance in Broward County, Florida, between 2013–2014. Researchers at Dartmouth College took data on age, sex, and criminal history for 1,000 defendants and handed it to volunteer “judges” who were recruited over the internet via Amazon’s Mechanical Turk service.
 </p>
<p>
  COMPAS was no better than the study’s participants in assessing the risk of a defendant reoffending. The study’s authors compared the recommendations made by COMPAS and the participants with real-world data on reoffenders. In practice, more white people who were predicted not to reoffend did (40.3% humans, 47.9% COMPAS) compared with black people (29.2% humans, 30.9% COMPAS). Moreover, a larger proportion of black criminals were
  <u>
   wrongly
  </u>
  predicted to reoffend (37.1% humans, 40.4% COMPAS) compared with white defendants (27.2% humans, 25.4% COMPAS).
 </p>
<figure class="richtext-image full-width">
<img alt="" class="richtext-image full-width" src="https://www.pbs.org/wgbh/nova/media/images/graph_ciminal_algorithm_white.width-990_KM2rgyN.png"/>
<figcaption>While COMPAS and human judgements were similar, failed judgement calls tended to favor white defendants and disadvantage black defendants. False positives are instances where criminals do not reoffend, but were predicted to. False negatives are instances where criminals were predicted to reform but did not.
        
    </figcaption>
</figure>
<p>
  The Dartmouth researchers also managed to reproduce the software’s predictions by consulting only 5% of the information the algorithm purportedly considers.
 </p>
<p>
  “The widely used commercial risk assessment software COMPAS is no more accurate or fair than predictions made by people with little or no criminal justice expertise,” write the study’s authors. “A simple linear predictor provided with only two features is nearly equivalent to COMPAS with its 137 features.”
 </p>
</section>
<p>
 Photo credit:
 <a href="http://www.jble.af.mil/News/Article-Display/Article/844448/">
  Staff Sgt. Nicholas RauReleased / Joint Base Langley-Eustis, Julia Dressel &amp; Hany Farid / Science Advances
 </a>
</p>

Extreme Diver Community Evolved Spleens Similar to Seals

<div class="related"></div>
<p>
</p>
<p>
 On an archipelago in Southeast Asia live the Bajau, an ancient people world-renowned for a unique cultural tradition. Every work day, the Bajau dive hundreds of feet below water to harvest fish and octopuses from the sea floor. To do so, they must
 <a href="https://www.theatlantic.com/science/archive/2018/04/bajau-sea-nomads-diving-evolution-spleen/558359/">
  hold their breath
 </a>
 on the order of minutes.
</p>
<p>
<a href="http://www.api-fellowships.org/body/international_ws_proceedings/2011-2012/P4-MBAL.pdf">
  Ancient epics
 </a>
 describe the Bajau originating from a land-based community dispatched onto the sea, stranded after a fruitless search for an abducted princess. These oral epics placed the Bajau’s origin at an uncertain point in time, and it was not until Magellan’s voyage around the world in 1521 that the Bajau were officially documented in historical records.
</p>
<p>
 Melissa Ilardo, of the Center for GeoGenetics, predicted that the Bajau had been seafaring long enough for diving advantages to emerge as heritable traits in the community. After all, the culture’s survival relies heavily upon harvesting food from the ocean’s depths; if a genetic diving advantage arose by chance, it would probably enhance survival and persist in the community for future generations. Ilardo
 <a href="http://www.api-fellowships.org/body/international_ws_proceedings/2011-2012/P4-MBAL.pdf">
  told Berkeley News
 </a>
 that “the closest thing to the Bajau in terms of underwater working time is sea otters; they are also spending about 60% of their time in the water. That is really remarkable, even compared to other professional or traditional divers.”
</p>
<p>
 Armed with a portable ultrasound machine and DNA isolation kit, Ilardo set out to measure traits unique to extreme diver physiology.
</p>
<div class="ad"></div>
<p>
 Although the spleen is mainly known for its blood-filtering properties, the spleen also acts like a reservoir, housing a large percentage of the body’s blood. Spleen enlargement is a major component of highly specialized mammalian divers, like Weddell seals and whales, and competition divers tend to have larger-than-average spleens. Large spleens assist in diving by contracting &amp; thereby increasing the number of red blood cells circulating oxygen throughout the body. Some whales have even evolved
 <a href="http://www.sciencemag.org/news/2014/09/some-whales-have-14-spleens">
  a multi-spleen system
 </a>
 to magnify the dive benefits, with 14 spleens reported in some species.
</p>
<p>
 Since the Bajau are superb divers, Ilardo postulated that they might have acquired an adaptation affecting spleen size. She measured spleen sizes of 59 Bajau, and then compared Bajau spleens to 34 people from Saluans, a neighboring village that does not historically dive.
</p>
<p>
 Ilardo discovered Bajau spleens were 50% larger than their land-loving counterparts. Even Bajau individuals who avoided diving altogether still presented increased spleen size—suggesting this trait is heritable, rather than an artifact of diving itself.
</p>
<p>
 Enticed by the penetrance of this physical feature, Ilardo wanted to know for sure whether the Bajau’s enlarged spleens had genetic underpinnings. Scouring Bajau genomes, Ilardo identified two unique protein variants very likely to contribute to the Bajau’s otherworldly diving skills. One unique gene variant, PDE10A, is related to spleen growth, while another, FAM178B, influences carbon dioxide levels in the blood.
</p>
<p>
 Indeed, the Bajau have developed heritable adaptations that imitate diving features of seals. This convening of traits, called convergent evolution, is an instance where two totally unrelated species (seals and humans) evolved separately towards the same beneficial feature (large spleen size).
</p>
<p>
 Overall, Ilardo’s research illuminates genetic features contributing to the Bajau’s unusual diving skills, and demonstrate how much there is to learn about how human evolution might intertwine with cultural trends. Ilardo plans to return to the archipelago, and continue working with the Bajau people, who are curious about their own genetic history. Ultimately, Ilardo hopes her work spreads awareness and appreciation for understudied, indigenous populations such as the Bajau.
</p>
<div class="newsletter"></div>
<p>
 Photo credit:
 <a href="https://commons.wikimedia.org/wiki/File:Badjao_kid_diving_for_coins.jpg">
  Ronnie Puckett / Wikimedia Commons (CC BY 2.0)
 </a>
</p>

<p>The Bajau people appear to have developed heritable adaptations that imitate the diving features of seals. </p>

Similar to herd-immunity for vaccinations, scientists propose that GMO crops act as a pest-absorbing sink.

GMO Crops Have an Unintended Side-Effect: Protecting Non-GMOs

<div class="related"></div>
<p>
 GMOs have a
 <a href="https://www.npr.org/sections/thesalt/2012/10/18/163034053/top-five-myths-of-genetically-modified-seeds-busted">
  bad reputation
 </a>
 , but 20 years after they were first planted in farm fields, they are having an unintended side effect: helping non-GMO crops.
</p>
<p>
 Many genetically modified organisms were engineered with insect-killing proteins to eliminate or reduce the use of pesticides. The most widely used insect-killing protein comes from a microbe called Bacillus thuringiensis (often abbreviated “Bt”), and the hazards associated with seem largely
 <a href="http://www.who.int/foodsafety/areas_work/food-technology/faq-genetically-modified-food/en/">
  limited to insects
 </a>
 . Since the first Bt crop was planted in 1996, GMOs have been immune to the corn borer and earworm pests-nicknamed the “scourges of corn.”
</p>
<p>
 To prevent Bt-resistance in these pests and to confine GMO genes to their intended crops, the EPA mandates farmers plant non-GMO plants around the periphery of GMO fields- a moat of sorts.
</p>
<div class="ad"></div>
<p>
 But the corn in those moats are also receiving the benefit of genetic modification, even if the plants themselves aren’t altered. In a new study, which examined the landscapes of pests radiating from GMO crops, researchers found that GMO crops are enabling non-GMOs to flourish with fewer pesticides. New Jersey’s non-GMO green beans and peppers crops, for example, reduced pesticide use by 85% since GMOs were introduced.
</p>
<p>
 Similar to herd-immunity for vaccinations, scientists propose that GMO crops act as a pest-absorbing sink. With the number of pests dialed down, farmers can spray fewer pesticides onto non-GMO foods. This unprecedented finding translates into fewer pesticides in the environment-and unlike GMOs, pesticides have well-known
 <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4947579/pdf/fpubh-04-00148.pdf">
  health and environmental hazards
 </a>
 .
</p>
<p>
 Farmers can apply this knowledge to maximize the benefits. For example, farmers could plant other vegetables in close proximity to Bt corn, reducing the need for pesticide applications. Other pest-prone produce, such as potatoes, could also be spared spoilage by being planted in close proximity to GMOs.
</p>
<div class="newsletter"></div>
  

  
      Photo credit: public domain

<p>In a new study, which examined the landscapes of pests radiating from GMO crops, researchers found that GMO crops are enabling non-GMOs to flourish with fewer pesticides.</p>

Black holes may not have event horizons, but fuzzy surfaces.

Researchers Find Black Hole 'Village' at Center of Milky Way

<div class="related"></div>
<p>
 For years, astronomers have predicted that any single supermassive black hole is a goliath nested in the center of a black hole village—but the field has failed to procure hard evidence. Finally, after years of prediction and accumulating information, astrophysicists from Columbia University have unveiled the first evidence of such a village—and it exists right in the middle of our galactic neighborhood.
</p>
<p>
 Supermassive black holes are abnormally large, and their origins have remained an astronomical elephant-sized mystery. Their outright size makes it unlikely for supermassive black holes to have grown out of a single, dying star. To explain their size, astrophysicists hypothesized star clusters merge in their death throes, accumulating enough mass to forge a single supermassive black hole, but when you crunch the numbers for typical star clusters in our universe, such a process could easily exceed the age of the universe. There’s no satisfactory explanation without a mechanism for speeding up the process.
</p>
<p>
 In contrast to discrete star clusters, galaxies are massive stellar hubs; their gravitational force could reasonably behave like a black hole pressure-cooker. Within the center of a galaxy, many dying stars could rapidly collect, form binaries, and merge into a single supermassive black hole. And indeed, most observable supermassive black holes reside at the center of galaxies—leading most astrophysicists to agree on this proposed model.
</p>
<p>
 Still, substantiating this pressure-cooker hypothesis requires hard evidence. A density cusp, a localized increase in the number of small (stellar-mass) black holes near the edge of a single supermassive black hole, would represent a believable snapshot of proposed models for supermassive black hole formation. However, black holes themselves are very difficult to detect, and complicating the situation is the tremendous noise surrounding galactic centers.
</p>
<p>
 Finally, after decades of searching, Columbia University has provided the first evidence of a density cusp—a remnant of supermassive black hole formation. Using archived data from NASA’s Chandra X-Ray Observatory, Chuck Hailey and colleagues determined at least a dozen potential black holes exist on the periphery of Sagittarius A*, the supermassive black hole residing at the center of our own Milky Way Galaxy.
</p>
<div class="ad"></div>
<p>
 Hailey admits that six of the twelve potential black holes are convincingly real, whereas the remaining six could be attributed to noise and statistical flukes—emissions from neighboring millisecond pulsars, for example.
</p>
<p>
 Nevertheless, the location of these black holes are what’s significant: To date, we have only identified nearly sixty black holes within our galaxy, but most are concentrated towards the galactic center. This study adds at least six authentic, another six predicted, black holes on the cusp of Sagittarius A*. Here’s Chuck Haily reporting for Scientific American:
</p>
<blockquote>
<p>
 These objects… provide a unique laboratory for learning about how big black holes interact with little ones, because we can’t readily study these processes in other, more distant galaxies.
</p>
</blockquote>
<p>
 The work has potentially supermassive implications. If density cusps exist within our own galactic neighborhood, the same principle could extend to other galaxies, waiting to be detected throughout the universe. It could also indicate that there are many more—a theoretical 20,000—black holes lurking within our own Galaxy.
</p>
<p>
 Watch a profile of astrophysicist Andrea Ghez, who helped discover the supermassive black hole at the center of our galaxy.
</p>
<div>
<iframe allow="autoplay; encrypted-media" allowfullscreen="" frameborder="0" height="270" src="https://www.youtube.com/embed/N6QIllAjUcQ?feature=oembed" width="480"></iframe>
</div>
<p>
 Don’t miss NOVA’s “
 <a href="http://www.pbs.org/wgbh/nova/space/black-hole-apocalypse.html">
  Black Hole Apocalypse
 </a>
 ,” available for streaming online.
</p>
<div class="newsletter"></div>
<p>
 Photo credit:
 <a href="https://commons.wikimedia.org/wiki/File:Black_Hole_Milkyway.jpg">
  Physics education group Kraus, Universität Hildesheim / Wikimedia Commons (CC BY-SA 2.5)
 </a>
</p>

<p>After years of prediction and accumulating information, astrophysicists have unveiled the first evidence of a black hole village—and it exists right in the middle of our galactic neighborhood.</p>

With people living longer than ever, can science give us a means to defy age, and keep our neurons and dendrites in good shape?

Single Gene May Dramatically Reverse Age-Related Mental Decline

<div class="related"></div>
<p>
 Our brains are marvelous memory machines. Repeated use of a specific brain circuit—reminiscing about your favorite pet, solving puzzles, or practicing the violin—encourages new circuits to stay with us for life. When we neglect to engage them, our circuits—and encoded memories—can wither into oblivion (reinforcing the old adage, “use it or lose it”).
</p>
<p>
 And yet, even when we take strides to keep our minds sharp, we don’t always have a say in the matter. As we age, our mental capacity to learn and retain information is
 <a href="https://www.cdc.gov/mmwr/pdf/wk/mm6218.pdf">
  at risk
 </a>
 to deteriorate rapidly with age.
</p>
<p>
<a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4015335/pdf/nihms569964.pdf">
  Age-related mental decline
 </a>
 doesn’t interfere with a person’s daily life to the extent of dementia, but it does compromise social interactions and can turn once-trivial mental tasks into unexpected bouts of frustration.
</p>
<p>
 With people living longer than ever, can science give us a means to defy age, and keep our minds intact?
</p>
<p>
 To do this, we need to understand our brains and how age augments them. In extreme cases, such as Alzheimer’s, cell by-products physically accumulate and clog neural pathways, creating a blockade in normal circuits. However, normal brain-aging changes are much more nuanced—several years ago, Phillip Landfield and his group of researchers at the University of Kentucky noticed old brain cells retain unusually high levels of calcium—one of several key ions in brain circuits.
</p>
<p>
 Taking a closer look, the scientists discovered a coinciding deficiency: a protein called FPBP1b was drastically low when compared with youthful brains.
</p>
<p>
 Attempting to reverse these age-related deficits,
 <a href="http://www.jneurosci.org/content/early/2017/12/18/JNEUROSCI.2234-17.2017">
  Landfield
 </a>
 and his group injected elderly mice with a virus engineered for cranking-up FKBP1b protein levels in their brains. These mice were then challenged with a memory-intensive exercise: navigating a water maze and learning where a hidden platform exists.
</p>
<div class="ad"></div>
<p>
 Much to their surprise, the mice not only outperformed their age-matched counterparts, but their neurons had drastically reverted back to more youthful characteristics. Examining the brain cells at a molecular level, some 800 genes unexpectedly began reverting back to “young brain” levels. With so many punctuated changes arising from a single protein, FKBP1b (and calcium levels it controls) is much more relevant to mental aging than previously thought.
</p>
<p>
 Many of the augmented genes were tied to important brain cells activities, such as rewiring connections and transmitting electrical signals. However, inflammation signatures were unchanged throughout the regenerative process, prompting scientists to pause and question how much inflammation impacts aging (currently a hot topic in the realm of brain research).
</p>
<p>
 Obviously, much more work is required to develop a medicinal equivalent to the “fountain of youth,” but this study is an encouraging first-step. It opens the door for understanding how certain key proteins, such as FKBP1b, keep our minds sharp as we venture into retirement.
</p>
<div class="newsletter"></div>
<p>
 Photo credit: Public Domain
</p>

<p>With people living longer than ever, can science give us a means to defy age, and keep our minds intact?</p>

Plant-based nanowood doesn’t cause allergic reactions or irritate lung tissues, unlike some other insulators.

Super-Powered Styrofoam Substitute Could Change How We Insulate Homes

<div class="related"></div>
<p>
 Household insulation comes in many forms. Asbestos was once a cheap wonder-material until it was deemed carcinogenic. Cotton candy-like fiberglass wool is common, but deceptively hazardous, laced with tiny fiberglass shards that can penetrate lungs. And while expandable Styrofoam films aren’t entirely detrimental to our health, they don’t decompose—posing chronic environmental problems down the road.
</p>
<p>
 What we need is an insulation alternative that is effective, non-hazardous, and environmentally friendly. A new material called
 <a href="http://advances.sciencemag.org/content/4/3/eaar3724.full">
  nanowood
 </a>
 seems to fit the bill.
</p>
<p>
 Although its name sounds futuristic, nanowood is completely derived from trees. It is exceedingly flexible, lightweight, and—unlike other insulators—totally sustainable. Because it’s made from trees, it can be commercially produced to fit any shape sculpted from wood. Like paper, thin slices of nanowood can be rolled up without breaking. Moreover, nanowood has super-thermal insulating capacity on par with Styrofoam, making it suitable for applications requiring tightly regulated heat transfer— like energy-efficient buildings, electrical devices, and aerospace applications.
</p>
<div class="ad"></div>
<p>
 To generate this new material, scientists at the University of Maryland used techniques very similar to the techniques used in developing this innovative
 <a href="https://www.youtube.com/watch?v=x4KS6NyFo3Y">
  spinach leaf heart
 </a>
 . Sheets of wood were cut along a tree’s plane of growth and treated with reagents that removed cells and stripped away most wood components. The scientists were left with many parallel sheets of plant-sugar scaffold (or nanocellulose), which not only reflect heat, but also dissipate any trace amounts of heat that do manage to penetrate. By combining reflection with deflection, nanowood has low emissivity, which gives it an advantage over Styrofoam in blocking the sun’s thermal radiation. This makes the earth-origin material suitable for applications in outer space.
</p>
<p>
 Since carbon dioxide is the main ingredient of plant sugar, and by extension, decellularized nanowood, this non-toxic, flexible, super-insulating material may seem too good to be true—and with estimated costs of manufacturing at $2.27 per square foot, this dream-product could be market-ready in no time.
</p>
<div class="newsletter"></div>
<p>
 Photo credit: Hua Xie / University of Maryland
</p>

<p>What we need is an insulation alternative that is effective, non-hazardous, and environmentally friendly, and this new material seems to fit the bill.</p>

Behind nearly every piece of science news you read is a magic number: the p-value.

Rethinking Science’s Magic Number

<div class="related"></div>
<p>
</p>
<p>
 When a new scientific finding arrives in your newsfeed, it is often too early to recognize whether it’s undercooked. Consider “power posing,” the focus of
 <a href="http://datacolada.org/wp-content/uploads/2015/05/Carney-Cuddy-Yap-2010.pdf">
  a Harvard Business School study
 </a>
 . By standing like a superhero one minute a day, its authors found that it boosted testosterone, reduced stress hormones, and generally made people feel empowered.
</p>
<p>
 Not long after, a “power pose”
 <a href="https://www.ted.com/talks/amy_cuddy_your_body_language_shapes_who_you_are">
  TED talk
 </a>
 went viral, leaving a trail of anecdotal success stories and garnering millions of views.
 <a href="http://www.businessinsider.com/power-poses-interview-body-language-2014-3">
  Business
 </a>
<a href="https://www.inc.com/magazine/201205/leigh-buchanan/strike-a-pose.html">
  websites
 </a>
 were quick to promote the “power pose” life hack—it was backed by science, after all.
</p>
<p>
 And then it all came crumbling down. A nearly identical experiment, using more thorough methods and four times as many study participants,
 <a href="http://datacolada.org/wp-content/uploads/2015/05/5110-Ranehill-Dreber-Johannesson-Leiberg-Sul-Weber-PS-2015-Assessing-the-robustness-of-power-posing-no-effect-on-hormones-and-risk-rolerance-in-a-large-sample-of-men-and-women.pdf">
  did not find evidence
 </a>
 that power posing did any of those things.
 <a href="http://journals.sagepub.com/doi/abs/10.1177/1948550616652209">
  An offshoot study
 </a>
 further failed to reproduce the magic of the “power pose.”
</p>
<p>
 It’s easy to feel hoodwinked, but at the time of its release, author
 <a href="https://www.nytimes.com/2017/10/18/magazine/when-the-revolution-came-for-amy-cuddy.html">
  Amy Cuddy
 </a>
 genuinely believed her research findings. In fact, her statistics seemed to back her up, and peer reviewers had given her the thumbs up. And she is not alone—for decades, scientists and the media alike have consumed claims from single-publications as fact. As meticulous as scientists are, inaccurate, false, or even misleading science claims are surprisingly common.
 <a href="http://science.sciencemag.org/content/349/6251/aac4716">
  A 2015 experiment
 </a>
 reported over one-third of psychology experiments are faulty, while a similar test for
 <a href="https://elifesciences.org/articles/23693">
  cancer research
 </a>
 indicates similar trends, though that one is still in progress. The problem isn’t limited to those two fields, either.
</p>
<h2>
 Magic Number
</h2>
<p>
 Behind nearly every piece of science news you read is a magic number: the p-value. To generate a p-value, scientists run statistical tests comparing data sets. Their starting assumption is that there’s no difference between the things they’re comparing—new drug vs. sugar pill, power pose vs. nothing—and when they find a result that deviates significantly enough from “no difference,” they report it as a positive finding.
</p>
<p>
 “If you assume that nothing is going on…and get a surprising set of data, it might make you wonder whether there is something going on.” said Daniel Lakens, an assistant professor of applied cognitive psychology at Einthoven University of Technology in the Netherlands.
</p>
<p>
 You can think of the p-value as a “warmer-colder” game for scientific knowledge: a low p-value shouts “warmer,” while a high p-value reads “colder, not much to see here.” It’s a helpful tool, but it’s not perfect. And the threshold that most scientists use is less perfect than some would like.
</p>
<p>
 The p-value was originally devised by French scholar Pierre-Simon Laplace, a contemporary of Napoleon Bonaparte. Sir Ronald A. Fisher later popularized it after a famous experiment involving a lady who could taste differences in virtually identical cups of tea. Fisher proposed a p-value threshold of 0.05, and his ideas laid the groundwork for modern experimentation as it exists today.
</p>
<div class="pullquote">
<p>
 Over time, p = 0.05 has morphed into a Caesar’s thumb for research.
</p>
</div>
<p>
 “For decades, the conventional p-value threshold has been 0.05,” says Dr. Paul Wakim, chief of the biostatistics and clinical epidemiology service at the National Institutes of Health Clinical Center, “but it is extremely important to understand that this 0.05, there’s nothing rigorous about it. It wasn’t derived from statisticians who got together, calculated the best threshold, and then found that it is 0.05. No, it’s Ronald Fisher, who basically said, ‘Let’s use 0.05,’ and he admitted that it was arbitrary.”
</p>
<p>
 But over time, p = 0.05 has morphed into a Caesar’s thumb for research, making it hard to publish anything that doesn’t fall under that threshold. What’s worse, the pressure to report a passing p-value leads some scientists to resort to
 <a href="https://www.nature.com/news/collateral-damage-how-one-misconduct-case-brought-a-biology-institute-to-its-knees-1.17427">
  misconduct
 </a>
 .
</p>
<p>
 One form of misconduct, called “p-hacking,” occurs when scientists selectively embellish a few, randomly extreme data points, and conceal many insignificant ones. Their work passes the p-value threshold and makes the effects of their experiment seem more convincing than what occurs in reality.
</p>
<p>
 The NIH Clinical Center has reined in p-hacking in federally funded projects through improved the surveillance. By registering on websites like
 <a href="https://www.clinicaltrials.gov/">
  www.clinicaltrials.gov
 </a>
 , researchers are required to outline their experiments upfront. Scientists must then report their results in that same registry. Since the information is tracked, non-significant results get regularly communicated, making it more difficult to p-hack.
</p>
<p>
 Most fields don’t have a registry and instead rely heavily on publications for peer-review of research, which has its shortcomings as well. Peer-review can work well, but it only analyzes experiments after p-values have been produced, for example.
</p>
<div class="ad"></div>
<h2>
 A Warning
</h2>
<p>
 Misconduct or not, many statisticians and scientists are concerned about the improper use of p-values. The American Statistical Association recently issued
 <a href="https://amstat.tandfonline.com/doi/abs/10.1080/00031305.2016.1154108#.WpB1D62ZP-a">
  a stern warning
 </a>
 against misuse of p-values, a first in its 179-year existence. “The p-value was never intended to be a substitute for scientific reasoning,” its director wrote.
</p>
<p>
 That said, there may be one easy fix—lowering the p-value for significance. When examining the p-values of psychology claims that successfully passed the
 <a href="http://science.sciencemag.org/content/349/6251/aac4716.full?ijkey=1xgFoCnpLswpk&amp;keytype=ref&amp;siteid=sci">
  reproducibility test
 </a>
 , Open Science Collaboration authors noted that studies with the lowest p-values were more likely to pass, suggesting that low p-values could help weed out intentionally and unintentionally spurious claims.
</p>
<p>
</p>
<div class="pullquote">
<p>
 “The p-value was never intended to be a substitute for scientific reasoning.”
</p>
</div>
<p>
 Some research fields have voluntarily opted for more stringent p-values, because p = 0.05 was not strict enough. Particle physics, for example, has replaced the conventional p = 0.05 with “five sigma” or, p = 0.0000003, allowing them to announce discoveries, like the
 <a href="https://home.cern/topics/higgs-boson">
  Higgs boson
 </a>
 , with extraordinary confidence. Genetics applies similar thresholds to help scientists focus on the most relevant genes among the thousands residing in our genomes.
</p>
<p>
 Another branch of statistics also supports lowering the p-value threshold. Valen Johnson of Texas A&amp;M University analyzed scientific claims made with p-values of 0.05 using what’s known as a Bayesian approach. Most results were inconclusive. It wasn’t until the p-value neared
 <a href="http://www.pnas.org/content/110/48/19313">
  0.005
 </a>
 that the research claims satisfied both varieties of stats—and in Johnson’s eyes, were scientifically believable.
</p>
<p>
 Last summer, a similar, albeit less-technical
 <a href="https://www.nature.com/articles/s41562-017-0189-z">
  publication
 </a>
 emerged in the renowned journal
 <i>
  Nature
 </i>
 , where 72 prominent scientists reiterated Johnson’s sweeping proposal. Unsurprisingly, Johnson was among the lead authors.
</p>
<p>
 Think of the p-value as a filter: with a high p-value, the holes in the filter are wider, and some plausible-sounding data that was actually just randomness will make it through the peer-review process. However, if you reduce the p-value to 0.005, the filter becomes much more stringent.
</p>
<p>
 That sounds great, but Dr. Wakim argues that the proposed solution creates a new problem for clinical research: filtering out real discoveries that could translate into treatments—and potentially lives saved. He’d rather see a false-positive discovery be neatly corrected later, rather than miss it altogether.
</p>
<h2>
 Alternative Approaches
</h2>
<p>
 The breadth of the p = 0.005 proposal stepped on many researchers’ toes, and it wasn’t long before the internet blew up in response. Lakens seized the wave of attention to organize
 <a href="https://docs.google.com/document/d/12CNMlmxhD2q6mymyDl_Bp1mTIAAiWCblZneDDQc2nQY/edit">
  a Google Doc
 </a>
 , hosting a massive (and surprisingly organized) open-dialogue on the subject. Scientists with diverse backgrounds and experience levels contemplated how a standard of p = 0.005 would be implemented.
</p>
<p>
 Ultimately they did not agree with the 72-author publication—there was too much uncertainty on how potential benefits could justify costs.
</p>
<div class="pullquote">
<p>
 If a specific experiment is repeated, and the outcomes are the same as before, we can feel more confident in the results.
</p>
</div>
<p>
 Instead, the Google Doc’s authors entertained less rigid alternatives. One proposal was that each field or community might set its own threshold of significance. This is because the costs and benefits of changing a p-value vary depending on the discipline.
</p>
<p>
 Lakens likens the conflict to a debate over speed limits. “It’s like saying we have a lot of accidents when we drive, and we could fix it if we set the driving speed to 20 miles per hour because then we’re never going to have lethal accidents,” he says. “But then of course you don’t get anywhere as quickly anymore. And maybe you cannot make some trips because they are too far. It’s sort-of the same for science, so you can make it very slow, and careful…but whether it’s actually optimal in the complete sense of weighing all the costs and benefits, it’s not very clear.”
</p>
<div class="newsletter"></div>
<p>
 Lakens would rather see more studies that attempt to reproduce previous experiments. After all, if a specific experiment is repeated, and the outcomes are the same as before, we can feel more confident in the results. And with scientists knowing their work could be challenged with replications, they might feel less inclined to p-hack.
</p>
<p>
 But repeating an experiment is currently less dazzling than publishing a new finding, and it’s expensive. A few grants exist to address the latter issue, but the former is more intractable. New results are almost always better received than confirmations of what we already thought we knew.
</p>
<p>
 Dr. Wakim would like to see scientific journals accept results no matter the p-value. “Scientific discoveries take a long time, and many of them happen by elimination of unlikely explanations,” he says. Negative results, while not as exciting, have real value.
</p>
<p>
 Similar to Thomas Edison’s discovery of 999 ways not to make a lightbulb, negative results inform future research efforts, making science more efficient. What’s more, if scientists are transparent about their p-values, organizations like the
 <a href="http://www.cochrane.org/">
  Cochrane Collaboration
 </a>
 readily use this data for meta-analyses. Detecting a cohort of similar claims, all with p-values slightly above 0.05, results in a form of significance all its own that stand-alone studies could not necessarily substantiate.
</p>
<p>
 “People say, ‘Ugh, it’s above 0.05, I wasted my time.’ No, you didn’t waste your time.” says Dr. Wakim. “If the research question is important, the result is important. Whatever it is.”
</p>
<p>
 Photo credit:
 <a href="https://www.flickr.com/photos/dvanzuijlekom/9223220544/">
  Dennis van Zuijlekom / Flickr (CC BY-SA 2.0)
 </a>
</p>

<p>For generations, scientists have relied on one number to determine the value of their research. Now, some are considering alternatives.</p>

Tiffany Dill

Articles