The Nesting-Egg Problem: Why Comparative Effectiveness Research Is Trickier Than It Looks

A sixty-five-year-old woman goes to her doctor with excruciating lower back pain. When she first felt it, she assumed that she had merely pulled a muscle and that the pain would go away, but a month has passed with no relief. The diagnosis is straightforward: an MRI shows a hairline fracture in her spine due to osteoporosis. But choosing the right treatment will be anything but straightforward.

One option is to keep doing what she has been doing: using over-the-counter pain relievers as needed until, ideally, the fracture heals. Another option is vertebroplasty, an injection of acrylic medical cement into the fracture. Until recently, vertebroplasty was the obvious choice. A few small trials had found it superior to conservative treatment. A consensus statement signed by several medical groups endorsed the procedure, and Medicare and private health insurers paid for it. But last summer, that certainty evaporated. Two trials involving about two hundred patients, published in the August 6th issue of the New England Journal of Medicine, found that vertebroplasty—which costs $2,500 or more, plus $1,000 to $2,000 for an MRI scan—was no better than a sham injection for relieving back pain from spinal fractures.

This is the sort of strong evidence that the federal government was aiming for last February when it allotted $1.1 billion for comparative effectiveness research as part of the American Recovery and Reinvestment Act of 2009, better known as the stimulus bill. In the most systematic and best-funded effort of this kind in the United States, the bill supports research comparing drugs, surgery, and other means of preventing, diagnosing, and treating particular health conditions. A panel of federal employees will oversee the research and advise the president and Congress on how to spend the money. Health care reform legislation being debated in Congress at press time also calls for comparative effectiveness research.

Fewer than half of the treatments and tests that doctors recommend are supported by scientific evidence, according to a recent report from the Institute of Medicine. "This uncertainty contributes to great variability in managing clinical problems, with costs and outcomes differing markedly across the country," the report concludes. The great hope of comparative effectiveness research is that it will supply some of the missing science. It will put doctors' decisions on firmer ground, help patients get the best possible care by sparing them the inconvenience, potential harm, and expense of unnecessary procedures, and ultimately save the health care system money. President Obama put it this way: "There's going to be some disagreement, but if there's broad agreement that in this situation the blue pill works better than the red pill, and it turns out the blue pills are half as expensive as the red pill, then we want to make sure that doctors and patients have that information available to them."

If only it were that easy. While comparative effectiveness research has had some game-changing successes—the association between Vioxx and increased heart attack risk came out in a comparative effectiveness study of pain relievers—it often fails to show which treatment is best, and for whom. It suffers from what might be called the nesting-egg problem. Instead of giving definitive answers, it opens the door to new questions, like little eggs revealed within larger ones. The blue pill may be more effective for most patients, but maybe the red pill could benefit a particular group. And what is meant by "effective"—shrinking a tumor by 10 percent? Extending life by a year? What outcomes really matter?

The New England Journal of Medicine studies provide a prime example of the nesting-egg problem. In both studies, the group of people who got the vertebroplasty and the group that got the sham injection improved, but the difference between the groups was negligible. Does that mean that the treatment is worthless? Yes, concluded one of the studies, which said, "These findings call into question the use of vertebroplasty." No, concluded the other study; it saw a nesting egg—the possibility that...