Wisdom of Crowds for Diagnostics Signature Challenge

Modern society demands greater scrutiny of the potential health risks and benefits of long-term, and sometimes lifelong, exposure to drugs, chemicals, and substances found in consumer products and the environment. Organizations such as companies and academic consortia conduct large multi-year scientific studies that entail the collection and analysis of thousands of data points. The individual experiments are often conducted over many physical sites and with internal and outsourced components. To extract maximum value, the interested parties need to verify the accuracy and reproducibility of automated collection and analysis workflows in systems biology before the initiation of large multi-year studies. Traditional verification using the peer-review process has shortcomings, such as lack of scalability, which renders it insufficient for the assessment of high throughout research. A team of researchers at PMI and IBM, whose aim is to improve the effectiveness of scientific studies and verification of scientific findings propose a scheme called improver, for industrial methodology for process verification of research. This methodology evaluates a research program by dividing its workflow into smaller building blocks, whereby the verification of each building block can be done internally or externally via challenge-based `crowdsourcing` to a research community. Scientific challenges will be broadcast to potential stakeholders in the form of an open call for participation with the intention of providing the community with the opportunity to test their computational methods on new data as well as to partake in a collaborative effort whose ultimate goal could contribute to solving a grand scientific problem. Considering cancer as the leading cause of death worldwide, we formulate the Diagnostics Signature Challenge to evaluate novel approaches for the identification of robust and predictive signatures for this disease. The goal of a Diagnostics Signature Challenge is to verify that transcriptomics data contains enough information for the determination and prognosis of certain human disease states that could profit from better diagnostics signatures. Here we will describe the approach, the necessary operational steps, and how we intend to engage the wider scientific community to assess the applicability of the improver approach to molecular diagnostics (i.e., genomic signatures).