Making Sense of Evidence

On January 30th of this year President Obama announced a $215 million precision medicine initiative to improve health and disease. This initiative is part of a broad trend towards disease prevention and treatment that takes into account individual differences in people’s genes, environments, and lifestyles. Fulfilling the promise of precision medicine will depend on high quality evidence linking patient measurements and health. But what makes for high quality evidence? The different kinds of evidence used to justify healthcare intervention are well understood, and fit into a hierarchy of evidence.

To better understand the different kinds of evidence, let’s consider testing a specific hypothesis: Drinking a lot of Gatorade causes individuals to be more athletic. 

If we did a randomized control trial (RCT), we would choose a study group (i.e. males between the ages of 18-35), and then randomly assign individuals in our study group to the intervention (Gatorade) or control (maybe water) arm of the trial. After the study period, we would evaluate the athleticism of our intervention and our control arms to see if the individuals receiving Gatorade were in better physical shape (i.e. had faster 1 mile times). We probably wouldn’t see much signal there and could largely reject our hypothesis.

If we did an observational study, we would take a random sample of 1000 individuals (i.e. males between the ages of 18-35), and simply ask if Gatorade consumption correlates with athleticism. But there’s a problem here. It’s quite possible we would observe that individuals who drank more Gatorade would be more athletic—the opposite of what we found in our RCT! Why? In this case, it’s quite simple: Athletes probably drink more Gatorade than the average person because they exercise and sweat a lot. It’s not because drinking Gatorade makes you an athlete. In the real world, the reason for associations in observational studies is less clear. For example, recent reports suggest that moderate coffee consumption is associated with decreased cardiovascular risk. That could be because caffeine really is good for your heart, or because a person who drinks a lot of coffee is up and about a lot.

The key distinction here is that the observational study is able to robustly establish association between Gatorade and athleticism but has a hard time establishing causation

Does this mean that observational studies are useless? No. While a doctor shouldn’t tell you to drink Gatorade to increase muscle mass, we can still use Gatorade consumption to predict athleticism. If all we knew about someone was that they drank a lot of Gatorade, we would be correct in concluding that they were more likely to be an athlete. This is why the FDA mandates that pharmaceuticals be tested via RCTs before being marketed in the US.

While robust evidence of causation likely requires an RCT, RCTs are often very expensive and time consuming. Thus, in addition to enabling prediction, a good observational study can generate interesting hypotheses using subsequent RCTs. In any case, it’s imperative for the person interpreting the evidence (the doctor, nurse, scientist or patient) to understand what the data can tell you, and what it cannot.

Gatorade is a registered trademark of PepsiCo. The use of Gatorade in this example is solely to provide a relatable reference to the reader and is not intended to convey an opinion or endorsement on the product and or connection with athletic ability.

By Gaurav Bhatia, Data Scientist at Koneksa Health