Open Access Highly Accessed Open Badges Research

Exploiting the noise: improving biomarkers with ensembles of data analysis methodologies

Maud HW Starmans12*, Melania Pintilie3, Thomas John4, Sandy D Der3, Frances A Shepherd5, Igor Jurisica3678, Philippe Lambin2, Ming-Sound Tsao3 and Paul C Boutros16*

Author Affiliations

1 Informatics and Biocomputing Platform, Ontario Institute for Cancer Research, Toronto, ON, M5G 0A3, Canada

2 Department of Radiation Oncology (Maastro), GROW-School for Oncology and Developmental Biology, Maastricht University Medical Center, Maastricht, The Netherlands

3 Ontario Cancer Institute and the Campbell Family Institute for Cancer Research, University Health Network, Toronto, ON, M5G 2M9, Canada

4 Ludwig Institute for Cancer Research, Austin Health, Melbourne, Australia

5 Department of Medical Oncology and Hematology, Princess Margaret Hospital, University Health Network, Toronto, ON, M5G 2M9, Canada

6 Department of Medical Biophysics, University of Toronto, Toronto, ON, M5G 2M9, Canada

7 Techna Institute, University Health Network, Toronto, ON, M5G 2M9, Canada

8 Department of Computer Science, University of Toronto, Toronto, ON, M5G 2M9, Canada

For all author emails, please log on.

Genome Medicine 2012, 4:84  doi:10.1186/gm385

Published: 12 November 2012



The advent of personalized medicine requires robust, reproducible biomarkers that indicate which treatment will maximize therapeutic benefit while minimizing side effects and costs. Numerous molecular signatures have been developed over the past decade to fill this need, but their validation and up-take into clinical settings has been poor. Here, we investigate the technical reasons underlying reported failures in biomarker validation for non-small cell lung cancer (NSCLC).


We evaluated two published prognostic multi-gene biomarkers for NSCLC in an independent 442-patient dataset. We then systematically assessed how technical factors influenced validation success.


Both biomarkers validated successfully (biomarker #1: hazard ratio (HR) 1.63, 95% confidence interval (CI) 1.21 to 2.19, P = 0.001; biomarker #2: HR 1.42, 95% CI 1.03 to 1.96, P = 0.030). Further, despite being underpowered for stage-specific analyses, both biomarkers successfully stratified stage II patients and biomarker #1 also stratified stage IB patients. We then systematically evaluated reasons for reported validation failures and find they can be directly attributed to technical challenges in data analysis. By examining 24 separate pre-processing techniques we show that minor alterations in pre-processing can change a successful prognostic biomarker (HR 1.85, 95% CI 1.37 to 2.50, P < 0.001) into one indistinguishable from random chance (HR 1.15, 95% CI 0.86 to 1.54, P = 0.348). Finally, we develop a new method, based on ensembles of analysis methodologies, to exploit this technical variability to improve biomarker robustness and to provide an independent confidence metric.


Biomarkers comprise a fundamental component of personalized medicine. We first validated two NSCLC prognostic biomarkers in an independent patient cohort. Power analyses demonstrate that even this large, 442-patient cohort is under-powered for stage-specific analyses. We then use these results to discover an unexpected sensitivity of validation to subtle data analysis decisions. Finally, we develop a novel algorithmic approach to exploit this sensitivity to improve biomarker robustness.