John Arnold’s tweet, “A new study shows …” are “the four most dangerous words,” is a perfect crystallization of the philanthropist’s skeptical attitude toward a lot of scientific research. BRENT HUMPHREYS
BRIAN NOSEK HAD pretty much given up on finding a funder. For two years he had sent out grant proposals for his software project. And for two years they had been rejected again and again—which was, by 2011, discouraging but not all that surprising to the 38-year-old scientist. An associate professor at the University of Virginia, Nosek had made a name for himself in a hot subfield of social psychology, studying people’s unconscious biases. But that’s not what this project was about. At least, not exactly.
Like a number of up-and-coming researchers in his generation, Nosek was troubled by mounting evidence that science itself—through its systems of publication, funding, and advancement—had become biased toward generating a certain kind of finding: novel, attention grabbing, but ultimately unreliable. The incentives to produce positive results were so great, Nosek and others worried, that some scientists were simply locking their inconvenient data away.
The problem even had a name: the file drawer effect. And Nosek’s project was an attempt to head it off at the pass. He and a graduate student were developing an online system that would allow researchers to keep a public log of the experiments they were running, where they could register their hypotheses, methods, workflows, and data as they worked. That way, it would be harder for them to go back and cherry-pick their sexiest data after the fact—and easier for other researchers to come in and replicate the experiment later.
Nosek was so taken with the importance of redoing old experiments that he had also rallied more than 50 like-minded researchers across the country to participate in something he called the Reproducibility Project. The aim was to redo about 50 studies from three prominent psychology journals, to establish an estimate of how often modern psychology turns up false positive results.
It was little wonder, then, that funders didn’t come running to support Nosek: He wasn’t promising novel findings, he was promising to question them. So he ran his projects on a shoestring budget, self-financing them with his own earnings from corporate speaking engagements on his research about bias.
But in July 2012, Nosek received an email from an institution whose name he didn’t recognize: the Laura and John Arnold Foundation. A Google search told him that the Arnolds were a young billionaire couple in Houston. John, Nosek learned, had made his first millions as a wunderkind natural gas trader at Enron, the infamous energy company, and he’d managed to walk away from Enron’s 2001 collapse with a seven-figure bonus and no accusations of wrongdoing attached to his name. After that Arnold started his own hedge fund, Centaurus Energy, where he became, in the words of one hedge fund competitor, “the best trader that ever lived, full stop.” Then Arnold had abruptly retired at the ripe age of 38 to focus full time on philanthropy.
As Nosek tells it, John Arnold had read about the Reproducibility Project in The Chronicle of Higher Education and wanted to talk. By the following year, Nosek was cofounding an institution called the Center for Open Science with an initial $5.25 million grant from the Arnold Foundation. More than $10 million more in Arnold Foundation grants have come since. “It completely transformed what we could imagine doing,” Nosek says. Projects that Nosek had once envisioned as modest efforts carried out in his lab were now being conducted on an entirely different scale at the center’s startup-like offices in downtown Charlottesville, with some 70 employees and interns churning out code and poring over research. The skeletal software behind the data-sharing project became a slick cloud-based platform, which has now been used by more than 30,000 researchers.
The Reproducibility Project, meanwhile, swelled to include more than 270 researchers working to reproduce 100 psychology experiments—and in August 2015, Nosek revealed its results. Ultimately his army of volunteers could verify the findings of only about 40 percent of the studies. Media reports declared the field of psychology, if not all of science, to be in a state of crisis. It became one of the biggest science stories of the year.
But as it happens, Nosek is just one of many researchers who have received unsolicited emails from the Arnold Foundation in the past few years—researchers involved in similar rounds of soul-searching and critique in their own fields, who have loosely amounted to a movement to fix science.
John Ioannidis was put in touch with the Arnolds in 2013. A childhood math prodigy turned medical researcher, Ioannidis became a kind of godfather to the science reform crowd in 2005, when he published two devastating papers—one of them titled simply “Why Most Published Research Findings Are False.” Now, with a $6 million initial grant from the Arnold Foundation, Ioannidis and his colleague Steven Goodman are setting out to turn the study of scientific practice—known as meta-research—into a full-fledged field in its own right, with a new research center at Stanford.
British doctor Ben Goldacre also got an email from the Arnold Foundation in 2013. Famous in England as a sharp-witted scourge of “bad science,” Goldacre spent years building up a case that pharmaceutical companies, by refusing to reveal all their data, have essentially deceived the public into paying for worthless therapies. Now, with multiple grants from the Arnolds, he is leading an effort to build an open, searchable database that will link all publicly available information on every clinical trial in the world.
A number of the Arnolds’ reform efforts have focused on fixing nutrition science. In 2011 the science journalist Gary Taubes received an email from Arnold himself. Having spent more than a decade picking apart nutrition science, Taubes soon found himself cofounding an organization with a substantial grant from the Arnold Foundation, to rebuild the study of obesity from the ground up. And in 2015 the Arnold Foundation paid journalist Nina Teicholz to investigate the scientific review process that informs the US Dietary Guidelines. Just weeks before the federal guidelines were due for an update, Teicholz’s blistering report appeared in the prominent medical journal The BMJ, charging that the government’s panel of scientists had failed to consider evidence that would have done away with long-held worries about eating saturated fat.
And those are just a few of the people who are calling out iffy science with Arnold funding. Laura and John Arnold didn’t start the movement to reform science, but they have done more than anyone else to amplify its capabilities—typically by approaching researchers out of the blue and asking whether they might be able to do more with more money. “The Arnold Foundation has been the Medici of meta-research,” Ioannidis says. All told, the foundation’s Research Integrity initiative has given more than $80 million to science critics and reformers in the past five years alone.
Not surprisingly, researchers who don’t see a crisis in science have started to fight back. In a 2014 tweet, Harvard psychologist Daniel Gilbert referred to researchers who had tried and failed to replicate the findings of a senior lecturer at the University of Cambridge as “shameless little bullies.” After Nosek published the results of his reproducibility initiative, four social scientists, including Gilbert, published a critique of the project, claiming, among other things, that it had failed to accurately replicate many of the original studies. The BMJ investigation, in turn, met with angry denunciations from nutrition experts who had worked on the US Dietary Guidelines; a petition asking the journal to retract Teicholz’s work was signed by more than 180 credentialed professionals. (After an external and internal review, The BMJ published a correction but chose not to retract the investigation.)
The backlash against Teicholz also furnished one of the few occasions when anyone has raised an eyebrow at the Arnolds’ funding of science critics. On the morning of October 7, 2015, the US House Agriculture Committee convened a hearing on the controversy surrounding the dietary guidelines, fueled by the BMJ article. For two and a half hours, a roomful of testy representatives asked why certain nutrition studies had been privileged over others. But about an hour in, Massachusetts representative Jim McGovern leaned into his microphone. Aiming to defend the science behind the guidelines, McGovern suggested that the doubts that had been cast over America’s nutrition science were being driven by a “former Enron executive.” “I don’t know what Enron knows about dietary guidelines,” McGovern said. But “powerful special interests” are “trying to question science.”
McGovern’s quip about Enron, a company that hasn’t existed in 15 years, was a bit of a potshot. But given the long history of deep-pocketed business interests sowing doubt in research, his underlying question was a fair one: Who is John Arnold, and why is he spending so much money to raise questions about science?
FORTUNE magazine once dubbed Arnold “one of the least-known billionaires in the US.” His profile in the public consciousness is almost nonexistent, and he rarely gives interviews. But among hedge funders and energy traders, Arnold is a legend. John D’Agostino, former head of strategy of the New York Mercantile Exchange, says that in Arnold’s heyday, people in the industry would discuss him in “hushed and reverent tones.” In 2006, Centaurus reportedly saw returns of over 300 percent; the next year Arnold became the youngest billionaire in the country. “If Arnold decided he wanted to beat hunger,” D’Agostino says, “I wouldn’t want to bet on hunger.”
For all the swagger of that description, Arnold himself has virtually none. He is universally described as quiet and introspective. At Enron, a company famous for its brash, testosterone-laced cowboy culture, the perennially boyish-looking trader was reportedly so soft-spoken that his colleagues had to gather in close to hear him at restaurants. “People would read into it, and they would say he’s just being cagey,” D’Agostino says. “And then, after a couple of years, people were like, oh, no, he’s actually like that.”
Arnold is still quiet. “Usually the division of labor in most of our work is that I talk,” Laura Arnold says in a phone interview. By all accounts, Laura, who attended Harvard College and Yale Law School and worked as an oil executive, has been equally influential in setting the direction for the foundation. But when I visit the Arnold Foundation’s Houston headquarters in June, Laura has been called away on a family emergency, leaving John to do the talking. Arnold is 5’10”, trim, and blandly handsome, his unusually youthful appearance now somewhat concealed by a salt-and-pepper beard.
Arnold grew up in Dallas. His mother was an accountant (she would later help manage the books at his hedge fund). His father, who died when Arnold was 18, was a lawyer. By kindergarten, Arnold’s talent for math was apparent. “I think I was just born with a natural gift for seeing numbers in a special way,” he says. Gregg Fleisher, who taught him calculus in high school, recalls an occasion when Arnold instantly solved a math puzzle that had been known to stump PhDs. But he also stood out for his skepticism. “He questioned everything,” Fleisher says.
By the time he was 14, Arnold was running his first company, selling collectible sports cards across state lines. Those were the early days of the internet, and he managed to gain access to an online bulletin board intended only for card dealers. The listings let him see that the same cards were sold at different prices in different parts of the country—which presented an opportunity for arbitrage. “Hockey cards didn’t have much of a market in Texas,” he tells me. “I would buy up all the premium hockey cards and send them to Canada or upstate New York.” He called the company Blue Chip Cards. Arnold estimates that he made $50,000 before he finished high school.
Arnold graduated from Vanderbilt University in 1995, taking only three years to finish his degree. He started working at Enron four days later. A year after that, at age 22, he was overseeing Enron’s Texas natural gas trading desk, one of the company’s core businesses.
Arnold’s work at Enron—seeking to capitalize on seasonal price differences in natural gas—wasn’t all that different from what he’d done as a teenager selling sports cards. In Hedge Hogs, a 2013 book about hedge fund traders, Jeff Shankman, another star trader at Enron, is quoted describing Arnold as “the most thoughtful, deliberate, and inquisitive person” he worked with on the gas floor. But Shankman recognized that he and Arnold were different in one key respect: Arnold had a greater appetite for risk, a quality that seemed at odds with his quiet demeanor. On some days at Enron, Arnold would trade more than a billion dollars’ worth of gas contracts. In 2001, even as Enron was collapsing amid an accounting scandal that covered up billions in debt, he was reported to have earned $750 million for the company. A former executive at Salomon Brothers later told The New York Times that there were very few incidents in the history of Wall Street comparable to Arnold’s success that year.
As Enron neared bankruptcy, executives scrambled to hold its operation together, offering bonuses to keep traders on board. Arnold was given $8 million, the biggest payout of all, just days before Enron filed for bankruptcy. He started Centaurus the next year, bringing along a small group of former Enron traders, who worked out of a single large room.
Arnold says he wasn’t sure if he could match the success he’d enjoyed as a futures trader at Enron. As a pipeline company, Enron had a direct view onto many of the factors that influence gas prices. Now he’d have to rely purely on his prowess with data. By law, natural gas pipelines had to make much of their information public, and around the time Centaurus was forming, more of that information began to appear online. “A lot of people didn’t know it was out there,” Arnold says. “People who did, didn’t know how to clean it up and analyze it as well as we did.”
It wasn’t long before Arnold had the answer to his doubts. In 2006, Centaurus reportedly generated a 317 percent return overall, after taking the opposite side of a risky bet that another hedge fund, Amaranth, had made on fluctuations in natural gas prices. Amaranth, which was gambling with money from large pension funds, suffered a $6 billion loss and collapsed. By 2009, Centaurus was managing over $5 billion and had more than 70 employees. In its first seven years, according to Fortune, the fund never returned less than 50 percent.
But Arnold had to come down to earth eventually. In 2010, Centaurus experienced its first annual loss. And though the fund bounced back the next year, tighter regulations on trading and a far less volatile market—thanks to a growing supply of natural gas from shale rock—made it unlikely that Arnold would again see the astonishing returns of only a few years earlier. And so, at age 38, Arnold walked away from it all. He announced that he was closing Centaurus in a letter to investors: “After 17 years as an energy trader, I feel that it is time to pursue other interests.”
Arnold tells me that he had lost some of his passion for trading. At the time, his net worth was estimated to be around $3 billion. In 2010 the Arnolds had signed the Giving Pledge, promising to give away at least half their wealth—and he wanted to be as strategic about that goal as he had once been about trading. Arnold has said that the first phase of his life was “100 percent trying to make money” and that it’s now “100 percent trying to do good.” As The Wall Street Journal noted, in “US history, there may have never been a self-made individual with so much money who devoted himself to philanthropy at such a young age.”
John Arnold’s brief but legendary career in finance
THE ARNOLDS had been dabbling in philanthropy for years, supporting a few handpicked programs in education, criminal justice reform, and other areas that were important to them. But now, with their stepped-up ambitions, the couple entered a new realm. Arnold had always been ready to make huge bets, but it was ultimately his hunger for reliable data that made him a brilliant trader. That same hunger would make large-scale philanthropy far more challenging than he had anticipated.
In a glass conference room at the Arnold Foundation’s offices—which occupy the same space as the old Centaurus trading floor, a 15-minute drive from the glass tower whose entrance was once adorned with Enron’s famous E—Arnold explains that his and Laura’s initial plan had been to simply locate the most effective organizations and write them checks. But figuring out which organizations were most effective turned out to be vexing. Nonprofits are very good at reporting their success rates and citing the science behind their interventions, but dig into their claims—as the Arnolds would try to do—and you find that they often omit relevant context or confuse correlation with causation. “The more you read the research, the less you know,” Arnold says. “It became extraordinarily frustrating.”
Then, one day in November 2011, he was listening to the podcast EconTalk, hosted by libertarian economist Russ Roberts. The guest that day was science journalist Gary Taubes, and he was talking about how the prevailing dietary wisdom of the past 40 years—that eating too much fat leads to obesity and heart disease—arose from the flimsiest of scientific evidence. The foundational studies, Taubes said, looked at the diets and disease rates in various countries, then essentially guessed at which items in the diet were responsible for the country’s good or bad health statistics. Worse yet, whenever evidence came along that contradicted the consensus about the dangers of eating fat—often evidence that was much stronger than the evidence for the dangers—it was ignored or not even published. Hardly anyone in the world of nutrition science seemed willing to question the science behind the low-fat diet, even after Americans grew fat and diabetic in record numbers.
The picture Taubes painted wasn’t of a flawed study here or there but of a fundamentally broken scientific culture. During the podcast, he mentioned that he was raising money in the hope of funding experiments that might deepen our understanding of the root causes of obesity. Not long after the podcast went online, he received a five-line email from Arnold. “From the little I know about the science of nutrition, your study makes a lot of sense,” Arnold wrote. Like Nosek, Taubes had to Google Arnold to learn who he was. Six months later the Arnold Foundation made a $4.7 million seed grant to the Nutrition Science Initiative (NuSI), the nonprofit Taubes cofounded to support fundamental research on diet and health. The next year the Arnolds promised $35.5 million more. (WIRED wrote about NuSI in issue 22.09).
Arnold is careful not to lump all researchers together when he talks about the problems in science. But he tells me that listening to Taubes and reading his book, Good Calories, Bad Calories, had been an “aha moment” for him. “Science is built like a building,” Arnold says. “One floor on top of the next.” In nutrition, “the whole foundation of the research had been flawed. All these things that we thought we knew—when we step back and look at the evidence base—it’s just not there.”
Arnold says that now, unless he trusts a researcher’s work, he no longer believes the findings of any scientific study until he or someone on the staff carefully vets the paper. “A new study shows …” are “the four most dangerous words,” Arnold wrote on Twitter.
Together with Taubes’ work, Arnold was also reading Ioannidis’ and Goldacre’s equally devastating analyses. These critiques of science amounted to a deep philosophical quandary for the Arnolds, philanthropists who had dedicated their lives to a data-based approach to giving. “In everything they do, they want to be evidence-driven,” says Stuart Buck, vice president of research integrity at the Arnold Foundation. But if you look at the studies that can’t be reproduced and other issues facing science, “you start to think: What is evidence? What do we actually know?”
The Arnolds had already decided that, with decades of life ahead of them and almost unlimited resources, they had the time and money to evaluate charitable programs properly, even when that meant paying for expensive randomized controlled trials that could take years to complete. But now they were widening their scope. If they wanted to embark on truly “transformational change,” as their foundation literature states, it wouldn’t be enough to properly evaluate this or that education or criminal justice program. They would also have to take on a far more ambitious project: The Arnolds would have to try and fix science itself.
IN THEIR philanthropy, the Arnolds like to say, they follow data where it leads rather than let themselves be guided by ideology. And it’s true that, when it comes to political leanings, they are somewhat hard to pin down. The Arnolds identify as Democrats and were major financial supporters of President Barack Obama. In 2013 they donated $10 million to keep Head Start, the early-childhood education program for low-income kids, running through the federal government shutdown, and many of the issues they’ve taken on, from criminal justice reform to making prescription drugs more affordable, are decidedly progressive. Yet the foundation is also focused on reforming what the Arnolds see as a broken public pension system—a project that, in practice, usually means cutting payments to retirees, raising retirement ages, and switching new workers to 401(k)-style plans. That focus led Rolling Stone to call Arnold a “young right-wing kingmaker with clear designs on becoming the next generation’s Koch brothers.” (In 2015, Bloombergsuggested that Arnold may have somehow managed to become less popular as a philanthropist than he was as a billionaire trader.)
If John Arnold does have an identifiable ideology, it is that of a lifelong trader and quant: unsentimental, metrics-focused, interventionist. He is unapologetic about having worked at Enron, and he can be defensive about the moral standing of Wall Street in the public mind. In 2015, after a cancer researcher was found to have falsified research data and defrauded the government out of millions of dollars, Arnold complained on Twitter that the penalty, a five-year funding restriction, was too light. Had something similar happened on Wall Street, he tweeted, the perpetrator would have been sentenced to 10 years in jail and the bank would have been fined a billion dollars. “Is there something special about frauds in the securities biz that they should be penalized infinitely more harshly than other business frauds?” he went on. “Or is Wall Street just an easy target while cancer researchers and universities are not?”
So it’s no surprise that, in practice, the Arnolds’ approach to giving has a lot in common with John Arnold’s approach to investing. Laura tells me she sees her husband’s appetite for risk—an appetite she says she shares—as the most obvious link between his approach to trading and philanthropy. Once the foundation has identified areas where they believe they can make the biggest difference, they go all in. “We’re not looking to create an organization of safe success,” she says. “We’re looking to create an organization of thoughtful failure and fantastic success.”
Arnold is, in at least one respect, trying to make science a little more like finance. In recent decades, math and science whizzes like Arnold have invaded Wall Street, bringing a level of scientific precision to trading and often making fortunes in the process. And good traders, as Arnold sees it, naturally come to appreciate something that researchers too often miss: It’s very easy to be fooled by your own data. They internalize the risk of mistaking correlation for causation—not because they’re smarter than scientists but because they have money riding on the outcome. “As a general rule, the incentives related to quantitative research are very different in the social sciences and in financial practice,” says James Owen Weatherall, author of The Physics of Wall Street. “In the sciences, one is mostly incentivized to publish journal articles, and especially to publish the sorts of attention-grabbing and controversial articles that get widely cited and picked up by the popular media. The articles have to appear methodologically sound, but this is generally a lower standard than being completely convincing. In finance, meanwhile, at least when one is trading with one’s own money, there are strong incentives to work to that stronger standard. One is literally betting on one’s research.”
In my conversations with Arnold and his grantees, the word incentives seems to come up more than any other. The problem, they claim, isn’t that scientists don’t want to do the right thing. On the contrary, Arnold says he believes that most researchers go into their work with the best of intentions, only to be led astray by a system that rewards the wrong behaviors. Says Goodman, “Scientists really do want to discover things that make a difference in people’s lives. In a sense, that’s the strongest weapon that we have. We can feed off that.” Figuring out exactly which rewards work best and how to simultaneously change the incentives for researchers, institutions, journals, and funders is now a key area of interest for Goodman and Ioannidis.
At the Center for Open Science, Nosek has already begun to experiment with new incentives for scientists. Because investigating and replicating research begins with having the data and materials necessary to do so, he is particularly focused on making science more transparent. In 2014 he partnered with the journal Psychological Science to offer colorful “Open Data” and “Open Materials” badges for papers that met specific criteria for sharing. A 2016 study to determine the effectiveness of the badges showed that the number of articles that reported publicly available data had increased tenfold. “It’s a stupid little badge,” Nosek says, but it works.
Nosek is also still campaigning to convince researchers to preregister what they plan to analyze and report in a study, so that they can’t adjust their experiment on the fly or hide less-than-dazzling results—a problem that Goldacre is also tackling. To promote preregistration, the Center for Open Science offered the first 1,000 scientists who preregister their studies with the organization $1,000 each. Nosek says that the cash offers were Arnold’s idea.
Denis Calabrese, the Arnold Foundation’s president, says they don’t expect immediate results. The Arnolds have a “multiple-decade timeline to work on problems.” Yet the most remarkable thing about the Arnold Foundation’s research integrity projects is that they already appear to be paying off. For one thing, the problems plaguing scientific research are now increasingly well known. Of 1,576 researchers who responded to a recent online survey from Nature, more than half agreed there is “a significant crisis” of reproducibility. The comedian John Oliver spent 20 prime-time minutes on HBO last May mocking the reign of terrible science on TV news shows and in public debate: “After a certain point, all that ridiculous information can make you wonder: Is science bullshit? To which the answer is clearly no, but there’s a lot of bullshit masquerading as science.” (Some of the background footage in the segment came from the Arnold Foundation.)
Ioannidis, whose name is almost synonymous with scientific skepticism, says he has seen immense progress in recent years. The journals Science and Nature have started bringing in statisticians to review their papers. The National Institutes of Health is moving forward with new requirements for data sharing; starting as early as this year, all NIH-funded training programs must include plans for teaching researchers the principles of reproducibility. “Now everybody says we need replication; we need reproducibility,” Ioannidis tells me. “Otherwise our field is built on thin air.”
The Center for Open Science’s next big undertaking is another reproducibility project—this one for cancer studies—which recently revealed its initial findings (two of five studies yielded the same results the second time around). In 2012 the former head of cancer research at the biotech firm Amgen revealed the results of the company’s effort to replicate 53 “landmark” papers in hematology and oncology; only six studies’ findings could be confirmed. So there is already widespread concern about reproducibility in the field. The center’s replication efforts, in turn, have inspired economists and even tropical ecologists to plan reproducibility projects of their own.
Whether all this momentum will lead to transformational change decades from now is impossible to know. Arnold figures that some of his specific grants might not work out as planned. (The foundation’s funding of the Nutrition Science Initiative is now scheduled to end in November.) More generally, it may not be possible to truly reform a system where the incentives are already so deeply embedded. “It’s probably too big a lift for us to expect we’re going to change researchers who have been around for decades,” he says. Plus, systems of prestige and advancement die hard. “You don’t shift a culture overnight,” Nosek says. But as many Wall Street veterans can testify, betting against John Arnold is usually a bad idea.
The Four Most Dangerous Words? A New Study Shows | Laura Arnold | TEDxPennsylvaniaAvenue
Laura Arnold, co-chair of the Laura and John Arnold Foundation, describes how junk science is harming all of us, personally and as a society, and offers ideas for how philanthropists and policymakers can incorporate rigorous research into efforts to solve pressing problems.