Breaking Down a Gold Standard

One of the common challenges we see at Koneksa Health is the tendency of the research community to gravitate toward evaluating novel measures or endpoints for concurrence with older and very limited ‘gold standards.’ Previously I discussed the lack of an underlying conceptual framework for activity measures that rely on patient reported outcomes (PROs).  However similar challenges also emerge with more objective performance measures.

From Physical Fitness Measure to Multiple Sclerosis Outcome

The six minute walk test (6MWT) and its cousins (2MWT, 12MWT, etc.) are among my favorite examples. This test is based on physical fitness assessments developed in the mid-1960s; it usually involves setting up a standard length course in a doctor’s office and measuring how far an individual can walk in a set number of minutes. In the 1980s the assessment began to be utilized on patients with cardio-pulmonary conditions. Since then, significant evidence of its ability to classify patients by severity and predict disease progression has accumulated. 

The 6MWT has evolved into a tool for measuring the effectiveness of new interventions across a wide range of conditions that impact activities of daily living (e.g. cystic fibrosis, muscular dystrophy and Parkinson’s disease). This is where things get problematic: patients are rarely asked if they care about how far they can walk in six minutes – much less six minutes inside a doctor’s office! 

Understanding Ecological Validity

Jan-Patrick Stellmann and his colleagues recently approached this challenge with what seems at first to be a straightforward validation exercise - however one in which they interrogated important, patient centric questions to assess the ecological validity of the test. The study involved instrumenting Multiple Sclerosis patients with accelerometers, devices that track acceleration and movement, and compared them to the patients’ results on the 6MWT and 2MWT. 

actibelt® devices were used in the study. actibelt® is a registered trademark of Trium Analysis Online GmbH.  http://www.actibelt.com


actibelt® devices were used in the study. actibelt® is a registered trademark of Trium Analysis Online GmbH. 
http://www.actibelt.com

Stellmann and colleagues approached their study with the understanding that mobility matters a great deal to MS patients, but did not take for granted that the 6MWT was an appropriate proxy for real life behavior. They interrogated the week-long actigraphy data and demonstrated that six minute sequences of walking were only occasional in these patients (less than once a day) and even two minutes sequences not that frequent (less than 3x a day). Moreover, results on the 6MWT test were poorly correlated with real world measures of mobility such as the average number of steps per day. All together, this suggests that the 6MWT is not giving clinicians and researchers a clear picture of mobility for MS patients.

Sequences of 15s / 30s / 2 min / 6 min uninterrupted walking in real life. Mean number of uninterrupted walking sequences per day for each patient during a 7-day assessment of real life mobility with a mobile accelerometer. Data presented as boxplots with median, 25%-/75%-quantiles and whiskers representing 5%-/95%-quantiles. doi:10.1371/journal.pone.0123822.g001


Sequences of 15s / 30s / 2 min / 6 min uninterrupted walking in real life.

Mean number of uninterrupted walking sequences per day for each patient during a 7-day assessment of real life mobility with a mobile accelerometer. Data presented as boxplots with median, 25%-/75%-quantiles and whiskers representing 5%-/95%-quantiles.

doi:10.1371/journal.pone.0123822.g001

In theory, the large number of teams researching the 6MWT should give us confidence that the metric is not heavily biased.  However in the case of 6MWT, research often tests the concurrent validity of the 6MWT against other metrics that suffer from the same flaws. Much less attention is paid to fundamentally questioning the content and ecological validity – for more on this with the 6MWT see here.

New Frontiers, New Challenges

Abandoning a well-established standard isn’t easy. The absence of baseline data across different conditions and populations makes it quite difficult to estimate effect size and design good studies to test interventions.  Accordingly, it’s easy to see why teams who’ve committed years to developing new therapies gravitate to the ‘devil they know’ gold standards rather than charting new territory.  Although there are exciting initiatives and sources of funding on the horizon to advance our understanding of how to better measure patients (Precision Medicine Initiative, Clinical Trials Transformation Initiative, and PCORI to name a few), there are pragmatic ways for life sciences companies to do more in this area. Investing early and in translational research as well as continuing to study the natural history of diseases is critical. 

 

 1. J. P. Stellmann et al., “Ecological Validity of Walking Capacity Tests in Multiple Sclerosis,” PLOS ONE 10, no. 4 (April 2015): e0123822, accessed November 23, 2015,http://dx.doi.org/10.1371/journal.pone.0123822

 2. Ioannidis, John P. A. “Why Most Published Research Findings Are False.” PLoS Medicine 2.8 (2005): e124. PMC. Web. 24 Nov. 2015., accessed November 23, 2015, http://dx.doi.org/10.1371%2Fjournal.pmed.0020124 


By Chris Benko, CEO of Koneksa Health