How Poorly Designed Studies Hurt ALS Patients

The following is an excerpt from Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions, by Richard Harris.

One of the most heartbreaking stories about abysmal experimental design involves amyotrophic lateral sclerosis (ALS), better known as Lou Gehrig’s disease. The search for a treatment for this deadly degenerative disease is rife with studies so poorly designed that they offered nothing more than false hope for people essentially handed a death sentence along with their diagnosis. Tom Murphy was one of them.

Once an imposing figure, Murphy had played football and rugby in college. His six-foot-three frame and barrel chest gave him a solid presence. But his handshake wasn’t the crushing grip you might expect. The first time we met, it was a gentle squeeze. When we met again a year later, we didn’t shake hands at all. Murphy had lost his formerly impressive strength due to ALS.

People around the world donated more than $100 million to fight this deadly ailment during the Ice Bucket Challenge of 2014, but for most people its real-life consequences are an abstraction: something about the degeneration of nerves. For Murphy, a fifty-six-year-old father of three, ALS was a very concrete, slow march toward the day when his nerves could no longer direct his diaphragm to draw air into his lungs. (Physicist Stephen Hawking is the rare exception who has managed to survive for many years despite the diagnosis.)

[Listen to an interview with this book’s author.]

Murphy, remarkably, was not bitter about this turn of events when he told his story. Nor was he resigned to fading away when he first noticed some unusual muscle twitches in the winter of 2010. He went to his doctor, who, after a brief examination, sent him to a neurologist. Murphy actually ended up seeing three different neurologists before he finally got the diagnosis.

“When the guy said, ‘Sorry to tell you, but you have two to four years. Get your stuff together,’ I thought, ‘Really?’ It was a real curveball. I would never have thought that in a million years.” To prepare for what was likely to come, Murphy and his wife, Keri, sold the family home and bought a modern ranch-style house in Gainesville, Virginia, which Murphy could navigate without having to contend with stairs. He would eventually be getting around on wheels, once the muscle tone in his legs had faded. A giant TV graced the open and airy living room, where Murphy watched sports that he could no longer play himself.

But Murphy’s doctors also offered at least a sliver of hope. “The first thing they told me is we have a drug trial; would you like to be in it? And of course I thought it sounded pretty good,” Murphy said. People with ALS find their strength declines within a few years, and trials of potential drugs are only available to reasonably strong patients. So most only get one shot at an experimental treatment. In May 2011 he settled on the test of a drug called dexpramixole (or simply “dex”), becoming one of about nine hundred patients enrolled in a multi-million-dollar study. But when the drug company analyzed the data collected, the news was disappointing. Dex was not slowing the progression of symptoms in this group of patients. The trial was a bust.

Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

Buy

Murphy was philosophical. There’s no question the disease is a tough one to counteract. Almost everything scientists have tried for ALS has failed (other than one drug with very marginal benefit). So all scientists in the field have gone in knowing the likelihood of failure is high, but they didn’t know exactly why until a nonprofit research center called the ALS Therapy Development Institute (ALS TDI) in Cambridge, Massachusetts, began investigating that question. Researchers there decided to look at the original studies to see what they could learn. They discovered that the original animal studies to test these drugs were deeply flawed. They all used far too few mice, and as a result they all came up with spurious results. Some experiments used as few as four mice in a test group. Sean Scott, then head of the institute, decided to rerun those tests, this time with a valid experimental design involving an adequate number of mice that were handled more appropriately. He discovered that none of those drugs showed any signs of promise in mice. Not one. His 2008 study shocked the field but also opened a path forward. ALS TDI would devote its efforts to doing this basic biology right.

Scott died of ALS in 2009 at the age of thirty-nine—the disease runs in his family. His successor, Steve Perrin, has carried on as Scott would have, insisting on rigorous animal studies as the institute’s scientists search for anything to help people like Tom Murphy. And they’re not simply taking the basic—and what should have been obvious—step of starting with enough mice in each experiment. Male and female mice develop the disease at somewhat different rates, so if scientists aren’t careful about balancing the sexes in their experiments, they can get spurious results. Another problem is that the ALS trait in these genetically modified mice can change from one generation to the next. The scientists at ALS TDI look at the genetics of every single animal they use in an experiment to make sure that all are identical. “These variables are incredibly important,” Perrin said. Other scientists had often overlooked those pitfalls.

To get robust results, Perrin’s group uses thirty-two animals—and compares them to an untreated group of thirty-two more mice. Academic labs don’t use large numbers of mice in their experiments in part because they cost a lot of money. Perrin said each one of these tests costs $112,000, and it takes nine months to get a result. If you’re testing three dosages of a medication, each requires its own test. Perrin’s institute has shown clearly that cutting corners here can lead to pointless and wasteful experiments. Even so, “we still get some pushback from the academic community that we can’t afford to do an experiment like that,” he said. It’s so expensive that they choose to do the experiments poorly. (Courtney: big cut right here)

Given their expertise with the mouse model of ALS, they offer to reproduce the results from other laboratories, to validate—or most often to deflate—findings from academic and pharmaceutical labs. They have an aggressive program to develop their own drugs, based on tests carried out in their labs. The institute resides on the fourth floor of a modern lab and office building right across the street from the Massachusetts Institute of Technology (MIT). Next door are two other world-class institutions: the Broad Institute (which sequences genomes for ALS TDI) and MIT’s Whitehead Institute. Perrin doesn’t hesitate to farm out work that the many capable firms around Cambridge and the rest of the world can do more efficiently. Each Monday, the institute takes delivery of one hundred young genetically engineered mice, which are housed in a windowless expanse behind the light-filled and cheerful labs where nearly forty scientists work on artfully curved lab benches. Some spend their days doing experiments with the mice; others explore chemicals and biological compounds that are potential new drugs. This is the face of rigorous work, but rigor takes time and money—two commodities in perpetually short supply.

Excerpted from Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions by Richard Harris. 2017. Available from Basic Books, an imprint of Perseus Books, a division of PBG Publishing, LLC, a subsidiary of Hachette Book Group, Inc.