Peer-Reviewed Publications

      Crowdsourced benchmarking of taxonomic metagenome profilers: lessons learned from the sbv IMPROVER Microbiomics challenge

      Poussin, C.; Khachatryan, L.; Sierro, N.; Narsapuram, V. K.; Meyer, F.; Kaikala, V.; Chawla, V.; Muppirala, U.; Kumar, S.; Belcastro, V.; Battey, J. N. D.; Scotti, E.; Boué, S.; McHardy, A. C.; Peitsch, M. C.; Ivanov, N. V.; Hoeng, J.

      Published
      Aug 30, 2022
      DOI
      10.1186/s12864-022-08803-2
      Link to publication
      Topic
      Summary

      Background

      Selection of optimal computational strategies for analyzing metagenomics data is a decisive step in determining the microbial composition of a sample, and this procedure is complex because of the numerous tools currently available. The aim of this research was to summarize the results of crowdsourced sbv IMPROVER Microbiomics Challenge designed to evaluate the performance of off-the-shelf metagenomics software as well as to investigate the robustness of these results by the extended post-challenge analysis. In total 21 off-the-shelf taxonomic metagenome profiling pipelines were benchmarked for their capacity to identify the microbiome composition at various taxon levels across 104 shotgun metagenomics datasets of bacterial genomes (representative of various microbiome samples) from public databases. Performance was determined by comparing predicted taxonomy profiles with the gold standard.

      Results

      Most taxonomic profilers performed homogeneously well at the phylum level but generated intermediate and heterogeneous scores at the genus and species levels, respectively. kmer-based pipelines using Kraken with and without Bracken or using CLARK-S performed best overall, but they exhibited lower precision than the two marker-gene-based methods MetaPhlAn and mOTU. Filtering out the 1% least abundance species—which were not reliably predicted—helped increase the performance of most profilers by increasing precision but at the cost of recall. However, the use of adaptive filtering thresholds determined from the sample’s Shannon index increased the performance of most kmer-based profilers while mitigating the tradeoff between precision and recall.

      Conclusions 

      kmer-based metagenomic pipelines using Kraken/Bracken or CLARK-S performed most robustly across a large variety of microbiome datasets. Removing non-reliably predicted low-abundance species by using diversity-dependent adaptive filtering thresholds further enhanced the performance of these tools. This work demonstrates the applicability of computational pipelines for accurately determining taxonomic profiles in clinical and environmental contexts and exemplifies the power of crowdsourcing for unbiased evaluation.

      PMIScience.com is operated by Philip Morris International for the purpose of publishing and disseminating scientific information about Philip Morris International’s efforts in support of its smoke-free product portfolio. This site is a global site for use by scientists, the public health and regulatory communities, and other stakeholders with an interest in tobacco policy. The purpose of this site is not advertising or marketing, nor is it directed at any specific market. It is not intended for use by consumers. New tobacco products sold in the United States are subject to FDA regulation; therefore the content of this site is not intended to make, and nor should it be construed as making, any product related claims in the United States without proper FDA authorization. ​
      ​
      Reduced Risk Products ("RRPs”) is the term we use to refer to products that present, are likely to present, or have the potential to present less risk of harm to smokers who switch to these products versus continuing smoking. PMI has a range of RRPs in various stages of development, scientific assessment and commercialization. All of our RRPs are smoke-free products that deliver nicotine with far lower quantities of harmful and potentially harmful constituents than found in cigarette smoke.