Difference between revisions of "Journal:Defending our public biological databases as a global critical infrastructure"

From LIMSWiki
Jump to: navigation, search
(Saving and adding more.)
(Saving and adding more.)
Line 45: Line 45:
  
 
In this section we discussed the issue of errors in genomic and proteomic databases and their impacts for research and application. Sources of these errors may include, among others, entry errors derived from data transfer, original errors derived from source data, and metadata errors (typically provenance-related) derived from the analysis pipeline. Original errors can arise from sequencing and sample preparation instrumentation chemistry, hardware, and software. Metadata errors can arise from bioinformatics software and faulty human interpretation. Each of these errors may be considered noise or the result of some other unintentional cause, but the key problem to note is that each element of the analytical process introduces some level of artifact when creating the analytical product, i.e., what is defined as a peak or a spot, what is the gene scaffold, what is the closed genome, etc. Any difference in process would therefore by its nature have some impact on the final genome. Our goal here is to start drawing connections between these process elements and genome anomalies.
 
In this section we discussed the issue of errors in genomic and proteomic databases and their impacts for research and application. Sources of these errors may include, among others, entry errors derived from data transfer, original errors derived from source data, and metadata errors (typically provenance-related) derived from the analysis pipeline. Original errors can arise from sequencing and sample preparation instrumentation chemistry, hardware, and software. Metadata errors can arise from bioinformatics software and faulty human interpretation. Each of these errors may be considered noise or the result of some other unintentional cause, but the key problem to note is that each element of the analytical process introduces some level of artifact when creating the analytical product, i.e., what is defined as a peak or a spot, what is the gene scaffold, what is the closed genome, etc. Any difference in process would therefore by its nature have some impact on the final genome. Our goal here is to start drawing connections between these process elements and genome anomalies.
 +
 +
===Vulnerabilities and intentional tampering===
 +
In contrast to the data integrity issues discussed in the prior section, errors may also be intentionally introduced into a biological database. For example, consider the hypothetical scenario discussed by Peccoud ''et al.''<ref name="PeccoudCyber18">{{cite journal |title=Cyberbiosecurity: From Naive Trust to Risk Awareness |journal=Trends in Biotechnology |author=Peccoud, J.; Gallegos, J.E.; Murch, R. et al. |volume=36 |issue=1 |pages=4–7 |year=2018 |doi=10.1016/j.tibtech.2017.10.012 |pmid=29224719}}</ref> whereby a graduate student reads an article and subsequently requests the plasmids described, but receives a faulty sample. It may be that the published sequences were fabricated, or that the source [[laboratory]] unwittingly sent faulty plasmids. One could also imagine a scenario where an intentionally mislabeled or harmful sequence is submitted to an open database that could later be unknowingly synthesized in a research setting or, more seriously, in a production capacity. Furthermore, depending on how sequences could be submitted to the database, the adversary may be able to keep the pathogenic sequence from being detected by certain anomaly detection heuristics.
 +
 +
Individuals may also exploit the vulnerabilities inherent in the database as a cybersystem, leading to errors introduced after publication of data despite manipulation and deletion controls. As with any database, biological databases can be compromised, enabling data integrity issues related to insertion, manipulation, exfiltration, and deletion of data, as well as providing a platform for privilege escalation, unauthorized surveillance, or distribution of malware. Ultimately, the effects of the operating environment and the tools used to deliver databases will inform the most appropriate threat model.
 +
 +
==Approaches for improving biological databases==
 +
In 2000, a workshop titled ''Bioinformatics: Converting Data to Knowledge''<ref name="NRCBio00">{{cite web |url=https://www.nap.edu/catalog/9990/bioinformatics-converting-data-to-knowledge-workshop-summary |title=Bioinformatics: Converting Data to Knowledge |author=National Research Council |publisher=National Academies Press |year=2000 |doi=10.17226/9990}}</ref> tackled the question of biological database integrity as one of its focus areas. At that time, suggested solutions included building organism-type (e.g., eukaryote) specific grammar-based tools, enabling database self-validation through specialized ontologies, advocating for quality control in laboratories to minimize likelihood of errors, and authorizing only trained curators and annotators to enter data. They also recommended that data provenance be maintained so that the data history and evolution can be understood over time. These approaches broadly fall into two categories: ensuring integrity before or during data entry and [[Data analysis|analyzing data]] already in a database. Nearly 20 years later, we still emphasize the importance of [[quality control]] in laboratories and standardized data entry procedures, but it is clear that errors continue to make their way into databases for a variety of reasons. In this section, we highlight several categories of existing methods to detect data integrity issues in biological databases and outline the strengths and weaknesses of each. We also provide recommendations for improving biological database security.
 +
 +
===Automated approaches for detecting anomalies===
 +
Some biological databases take the manual curation approach, such as the SwissProt subset of the UniProt (Universal Protein Resource Database). This effort requires significant resources to maintain, consisting of three principal investigators, a large staff, and external advisory board.<ref name="PundirUniProt17">{{cite journal |title=UniProt Protein Knowledgebase |journal=Methods in Molecular Biology |author=Pundir, S;. Martin, M.J.; O'Donovan, C. |volume=1558 |pages=41–55 |year=2017 |doi=10.1007/978-1-4939-6783-4_2 |pmid=28150232 |pmc=PMC5565770}}</ref> Given the complexity and exponential growth of biological data, automatic methods are needed.
 +
 +
Some tools have been developed to assess the technical quality of genome assemblies (e.g., QUAST<ref name="GurevichQUAST13">{{cite journal |title=QUAST: Quality assessment tool for genome assemblies |journal=Bioinformatics |author=Gurevich, A.; Saveliev, V.; Vyahhi, N.; Tesler, G. |volume=29 |issue=8 |pages=1072–5 |year=2013 |doi=10.1093/bioinformatics/btt086 |pmid=23422339 |pmc=PMC3624806}}</ref>), their completeness in terms of gene content (e.g., BUSCO<ref name="SimãoBUSCO15">{{cite journal |title=BUSCO: Assessing genome assembly and annotation completeness with single-copy orthologs |journal=Bioinformatics |author=Simão, F.A.; Waterhouse, R.M.; Ioannidis, P. et al. |volume=31 |issue=19 |pages=3210–2 |year=2015 |doi=10.1093/bioinformatics/btv351 |pmid=26059717}}</ref>, ProDeGe<ref name="TennessenProDeGe16">{{cite journal |title=ProDeGe: A computational protocol for fully automated decontamination of genomes |journal=The ISME Journal |author=Tennessen, K.; Andersen, E.; Clingenpeel, S. et al. |volume=10 |issue=1 |pages=269–72 |year=2016 |doi=10.1038/ismej.2015.100 |pmid=26057843 |pmc=PMC4681846}}</ref>) and even their contamination level (e.g., acdc<ref name="LuxACDC16">{{cite journal |title=acdc - Automated Contamination Detection and Confidence estimation for single-cell genome data |journal=BMC Bioinformatics |author=Lux, M.; Krüger, J.; Rinke, C.  et al. |volume=17 |issue=1 |pages=543 |year=2016 |doi=10.1186/s12859-016-1397-7 |pmid=27998267 |pmc=PMC5168860}}</ref>, CheckM<ref name="ParksCheckM15">{{cite journal |title=CheckM: Assessing the quality of microbial genomes recovered from isolates, single cells, and metagenomes |journal=Genome Research |author=Parks, D.H.; Imelfort, M.; Skennerton, C.T. et al. |volume=25 |issue=7 |pages=1043–55 |year=2015 |doi=10.1101/gr.186072.114 |pmid=25977477 |pmc=PMC4484387}}</ref>). Currently there are several analysis pipelines and search methodologies to detect potentially contaminated sequences in the published and assembled genome, such as Taxoblast<ref name="DittamiDetect17">{{cite journal |title=Detection of bacterial contaminants and hybrid sequences in the genome of the kelp ''Saccharina japonica'' using Taxoblast |journal=PeerJ |author=Dittami, S.M.; Corre, E. |volume=5 |pages=e4073 |year=2017 |doi=10.7717/peerj.4073 |pmid=29158994 |pmc=PMC5695246}}</ref>, GenomePeek<ref name="McNairGenome15">{{cite journal |title=GenomePeek: An online tool for prokaryotic genome and metagenome analysis |journal=PeerJ |author=McNair, K.; Edwards, R.A. |volume=3 |pages=e1025 |year=2015 |doi=10.7717/peerj.1025 |pmid=26157610 |pmc=PMC4476108}}</ref>, homology search<ref name="RyukovHuman16">{{cite journal |title=Human Contamination in Public Genome Assemblies |journal=PLoS One |author=Kryukov, K.; Imanishi, T. |volume=11 |issue=9 |pages=e0162424 |year=2016 |doi=10.1371/journal.pone.0162424 |pmid=27611326 |pmc=PMC5017631}}</ref>, and a multi-step cleaning process followed by a consensus of rankings.<ref name="CornetConcensus18">{{cite journal |title=Consensus assessment of the contamination level of publicly available cyanobacterial genomes |journal=PLoS One |author=Cornet, L.; Meunier, L.; Van Vlierberghe, M. et al. |volume=13 |issue=7 |pages=e0200323 |year=2018 |doi=10.1371/journal.pone.0200323 |pmid=30044797 |pmc=PMC6059444}}</ref><ref name="LuRemoving18">{{cite journal |title=Removing contaminants from databases of draft genomes |journal=PLoS Computational Biology |author=Lu, J.; Salzberg, S.L. |volume=14 |issue=6 |pages=e1006277 |year=2018 |doi=10.1371/journal.pcbi.1006277 |pmid=29939994 |pmc=PMC6034898}}</ref> All these tools and methods require human review or use of additional tools to distinguish true positive from true negative and are therefore not feasible at scale.
  
 
==References==
 
==References==

Revision as of 22:00, 18 June 2019

Full article title Defending out public biological databases as a global critical infrastructure
Journal Frontiers in Bioengineering and Biotechnology
Author(s) Caswell, Jacob; Gans, Jason D.; Generous, Nicolas; Hudson, Corey M.; Merkley, Eric; Johnson, Curtis; Oehmen, Christopher;
Omberg, Kristin; Purvine, Emilie; Taylor, Karen; Ting, Christina L.; Wolinsky, Murray; Xie, Gary
Author affiliation(s) Sandia National Laboratories, Los Alamos National Laboratory, Pacific Northwest National Laboratory
Primary contact Email: karen at pnnl dot gov
Editors Murch, Randall S.
Year published 2019
Volume and issue 7
Page(s) 58
DOI 10.3389/fbioe.2019.00058
ISSN 2296-4185
Distribution license Creative Commons Attribution 4.0 International
Website https://www.frontiersin.org/articles/10.3389/fbioe.2019.00058/full
Download https://www.frontiersin.org/articles/10.3389/fbioe.2019.00058/pdf (PDF)

Abstract

Progress in modern biology is being driven, in part, by the large amounts of freely available data in public resources such as the International Nucleotide Sequence Database Collaboration (INSDC), the world's primary database of biological sequence (and related) information. INSDC and similar databases have dramatically increased the pace of fundamental biological discovery and enabled a host of innovative therapeutic, diagnostic, and forensic applications. However, as high-value, openly shared resources with a high degree of assumed trust, these repositories share compelling similarities to the early days of the internet. Consequently, as public biological databases continue to increase in size and importance, we expect that they will face the same threats as undefended cyberspace. There is a unique opportunity, before a significant breach and loss of trust occurs, to ensure they evolve with quality and security as a design philosophy rather than costly “retrofitted” mitigations. This perspective article surveys some potential quality assurance and security weaknesses in existing open genomic and proteomic repositories, describes methods to mitigate the likelihood of both intentional and unintentional errors, and offers recommendations for risk mitigation based on lessons learned from cybersecurity.

Keywords: cyberbiosecurity, biosecurity, cybersecurity, biological databases, machine learning, bioeconomy

Introduction

Although an openly shared interaction platform confers great value to the biological research community, it may also introduce quality and security risks. Without a system for trusted correction and revision, these shared resources may facilitate widespread dissemination and use of low-quality content, for instance, taxonomically misclassified or erroneous sequences. Furthermore, as these public databases increase in size and importance, they may fall victim to the same security issues and abuses that plague cyberspace to this day. If we act now by developing the databases with quality and security as a design philosophy, we can protect these databases at a much lower cost and with fewer challenges than we currently face with the internet.

In this perspective article, the authors aim to outline some potential quality assurance and security weaknesses in existing public biological repositories. In the background section we provide a discussion of errors present in public biological databases and discuss possible security vulnerabilities inherent in their access, publication, and distribution models and systems. Both unintentional and intentional errors are discussed, the latter of which has not been given significant consideration in literature.[1] Afterwards, we attempt to introduce greater trust in the data and analyses by providing recommendations to mitigate or account for these errors and vulnerabilities and point to approaches used by other internet databases. Finally, we summarize our recommendations in the conclusions section.

This article focuses on databases which contain public and freely available data. We recognize that other biological databases exist which contain private, sensitive, or otherwise valuable data (e.g., human genomes). While unauthorized disclosure is not a formal concern in public, non-human databases, safeguarding against intentional or unintentional erroneous content is. Some approaches have been proposed to protect unauthorized disclosure[2][3][4] and, while we don't survey these approaches in this perspective, we note that the public database community may benefit from these ideas as well.

Background: Problems with public biological databases

Data integrity

An important goal for bioinformatics is the continuous improvement of biological databases. Given the rapid nature of this improvement and the rate of data production though, the content of these repositories is not without error. For example, the problem of contaminated sequences has been recognized for nearly two decades, with evidence stating that bacteria and human error are the two most common sources of contamination.[5][6] Ancient DNA is also particularly affected by human contamination.[7] These contaminants are frequently introduced during experiments[5][8] from natural associations and insufficient purification.[9] In the past few years, additional reports have highlighted cases of DNA contamination in published genome data[10][11], suggesting that DNA contamination may be more widespread than previously thought. We recognize that errors and omissions can occur in open databases both at the sequence and at the metadata levels, but for this article we mainly focus on sequence and taxonomic data concerns for the purposes of illustrating some of the many data integrity challenges possible.

In addition to contamination, two high-profile examples of sequence errors include the reassembly of a misassembled Francisella tularensis genome[12] and the identification of single nucleotide errors in a reference Tobacco mosaic virus (TMV) genome.[13] Without a way to flag or remove the erroneous entries, future researchers are left to continually rediscover them. The errors in the reference TMV sequence are particularly disturbing. The taxonomic assignment corresponds to a pathogenic strain, but due to two erroneous single nucleotide polymorphisms (SNPs), virions synthesized from the published reference sequence are atypically not infectious. Overlooked contamination in reference genomes can thereby lead to wrong or confusing results and may have major detrimental effects on biological conclusions.[14][15] While resequencing could be used to identify and correct sequence errors, it is only possible when the original source material is available. For the given example of single nucleotide errors in the TMV genome, the biological sample (sequenced in 1982) no longer exists. In addition to missing samples, samples of high consequence human and agricultural pathogens may not be available for resequencing.

Database integrity considerations for proteomics are generally similar to those for genomics because databases of protein sequences are derived from genome sequencing, via genome annotation and in silico translation. A sequence database error is unlikely to result in spurious detection of a protein that is present in the sample (false positive), but it could easily lead to a failure to detect a protein that is present (false negative). This is particularly concerning for discovery of accurate peptide signatures for use in targeted assays, a rapidly growing area of research.

In this section we discussed the issue of errors in genomic and proteomic databases and their impacts for research and application. Sources of these errors may include, among others, entry errors derived from data transfer, original errors derived from source data, and metadata errors (typically provenance-related) derived from the analysis pipeline. Original errors can arise from sequencing and sample preparation instrumentation chemistry, hardware, and software. Metadata errors can arise from bioinformatics software and faulty human interpretation. Each of these errors may be considered noise or the result of some other unintentional cause, but the key problem to note is that each element of the analytical process introduces some level of artifact when creating the analytical product, i.e., what is defined as a peak or a spot, what is the gene scaffold, what is the closed genome, etc. Any difference in process would therefore by its nature have some impact on the final genome. Our goal here is to start drawing connections between these process elements and genome anomalies.

Vulnerabilities and intentional tampering

In contrast to the data integrity issues discussed in the prior section, errors may also be intentionally introduced into a biological database. For example, consider the hypothetical scenario discussed by Peccoud et al.[16] whereby a graduate student reads an article and subsequently requests the plasmids described, but receives a faulty sample. It may be that the published sequences were fabricated, or that the source laboratory unwittingly sent faulty plasmids. One could also imagine a scenario where an intentionally mislabeled or harmful sequence is submitted to an open database that could later be unknowingly synthesized in a research setting or, more seriously, in a production capacity. Furthermore, depending on how sequences could be submitted to the database, the adversary may be able to keep the pathogenic sequence from being detected by certain anomaly detection heuristics.

Individuals may also exploit the vulnerabilities inherent in the database as a cybersystem, leading to errors introduced after publication of data despite manipulation and deletion controls. As with any database, biological databases can be compromised, enabling data integrity issues related to insertion, manipulation, exfiltration, and deletion of data, as well as providing a platform for privilege escalation, unauthorized surveillance, or distribution of malware. Ultimately, the effects of the operating environment and the tools used to deliver databases will inform the most appropriate threat model.

Approaches for improving biological databases

In 2000, a workshop titled Bioinformatics: Converting Data to Knowledge[17] tackled the question of biological database integrity as one of its focus areas. At that time, suggested solutions included building organism-type (e.g., eukaryote) specific grammar-based tools, enabling database self-validation through specialized ontologies, advocating for quality control in laboratories to minimize likelihood of errors, and authorizing only trained curators and annotators to enter data. They also recommended that data provenance be maintained so that the data history and evolution can be understood over time. These approaches broadly fall into two categories: ensuring integrity before or during data entry and analyzing data already in a database. Nearly 20 years later, we still emphasize the importance of quality control in laboratories and standardized data entry procedures, but it is clear that errors continue to make their way into databases for a variety of reasons. In this section, we highlight several categories of existing methods to detect data integrity issues in biological databases and outline the strengths and weaknesses of each. We also provide recommendations for improving biological database security.

Automated approaches for detecting anomalies

Some biological databases take the manual curation approach, such as the SwissProt subset of the UniProt (Universal Protein Resource Database). This effort requires significant resources to maintain, consisting of three principal investigators, a large staff, and external advisory board.[18] Given the complexity and exponential growth of biological data, automatic methods are needed.

Some tools have been developed to assess the technical quality of genome assemblies (e.g., QUAST[19]), their completeness in terms of gene content (e.g., BUSCO[20], ProDeGe[21]) and even their contamination level (e.g., acdc[22], CheckM[23]). Currently there are several analysis pipelines and search methodologies to detect potentially contaminated sequences in the published and assembled genome, such as Taxoblast[24], GenomePeek[25], homology search[26], and a multi-step cleaning process followed by a consensus of rankings.[27][28] All these tools and methods require human review or use of additional tools to distinguish true positive from true negative and are therefore not feasible at scale.

References

  1. Moussouni, F.; Berti‐Équille, L. (2014). "Cleaning, Integrating, and Warehousing Genomic Data From Biomedical Resources". In Elloumi, M.; Zomaya, A.Y.. Biological Knowledge Discovery Handbook. John Wiley & Sons. pp. 35–58. doi:10.1002/9781118617151.ch02. ISBN 9781118617151. 
  2. Kim, M.; Lauter, K. (2015). "Private genome analysis through homomorphic encryption". BMC Medical Informatics and Decision Making 15 (Suppl 5): S3. doi:10.1186/1472-6947-15-S5-S3. PMC PMC4699052. PMID 26733152. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC4699052. 
  3. Mandal, A.; Mitchell, J.C.; Montgomery, H.; Roy, A. (2018). "Data oblivious genome variants search on Intel SGX". In Garcia-Alfaro, J.; Herrera-Joancomartí, J.; Livraga, G.; Rios, R.. Data Privacy Management, Cryptocurrencies and Blockchain Technology. Lecture Notes in Computer Science. Springer International Publishing. pp. 21. doi:10.1007/978-3-030-00305-0_21. ISBN 9783030003050. 
  4. Ozercan, H.I.; Ileri, A.M.; Ayday, E.; Alkan, C. (2018). "Realizing the potential of blockchain technologies in genomics". Gemone Research 28 (9): 1255–63. doi:10.1101/gr.207464.116. PMC PMC6120626. PMID 30076130. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC6120626. 
  5. 5.0 5.1 Merchant, S.; Wood, D.E.; Salzberg, S.L. (2014). "Unexpected cross-species contamination in genome sequencing projects". PeerJ 2: e675. doi:10.7717/peerj.675. PMC PMC4243333. PMID 25426337. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC4243333. 
  6. Strong, M.J.; Xu, G.; Morici, L. et al. (2014). "Microbial contamination in next generation sequencing: implications for sequence-based analysis of clinical samples". PLoS Pathogens 10 (11): e1004437. doi:10.1371/journal.ppat.1004437. PMC PMC4239086. PMID 25412476. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC4239086. 
  7. Pilli, E.; Modi, A.; Serpico, C. et al. (2013). "Monitoring DNA contamination in handled vs. directly excavated ancient human skeletal remains". PLoS One 8 (1): e52524. doi:10.1371/journal.pone.0052524. PMC PMC3556025. PMID 23372650. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC3556025. 
  8. Ballenghien, M.; Faivre, N;. Galtier, N. (2017). "Patterns of cross-contamination in a multispecies population genomic project: Detection, quantification, impact, and solutions". BMC Biology 15 (1): 25. doi:10.1186/s12915-017-0366-6. PMC PMC5370491. PMID 28356154. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC5370491. 
  9. Simion, P.; Philippe, H.; Baurain, D. et al. (2017). "A Large and Consistent Phylogenomic Dataset Supports Sponges as the Sister Group to All Other Animals". Current Biology 27 (7): 958-967. doi:10.1016/j.cub.2017.02.031. PMID 28318975. 
  10. Witt, N.; Rodger, G.; Vandesompele, J. et al. (2009). "An assessment of air as a source of DNA contamination encountered when performing PCR". Journal of Biomolecular Techniques 20 (5): 236–40. PMC PMC2777341. PMID 19949694. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC2777341. 
  11. Longo, M.S.; O'Neill, M.J.; O'Neill, R.J. (2011). "Abundant human DNA contamination identified in non-primate genome databases". PLoS One 6 (2): e16410. doi:10.1371/journal.pone.0016410. PMC PMC3040168. PMID 21358816. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC3040168. 
  12. Puiu, D.; Salzberg, S.L. (2008). "Re-assembly of the genome of Francisella tularensis subsp. holarctica OSU18". PLoS One 3 (10): e3427. doi:10.1371/journal.pone.0003427. PMC PMC2561293. PMID 18927608. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC2561293. 
  13. Cooper, P. (2014). "Proof by synthesis of Tobacco mosaic virus". Genome Biology 15 (5): R67. doi:10.1186/gb-2014-15-5-r67. PMC PMC4072989. PMID 24887356. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC4072989. 
  14. Philippe, H;. Brinkmann, H.; Lavrov, D.V. et al. (2011). "Resolving difficult phylogenetic questions: Why more sequences are not enough". PLoS Biology 9 (3): e1000602. doi:10.1371/journal.pbio.1000602. PMC PMC3057953. PMID 21423652. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC3057953. 
  15. Laurin-Lemay, S.; Brinkmann, H.; Philippe, H. (2012). "Resolving difficult phylogenetic questions: Why more sequences are not enough". PLoS Biology 22 (15): R593–4. doi:10.1016/j.cub.2012.06.013. PMID 22877776. 
  16. Peccoud, J.; Gallegos, J.E.; Murch, R. et al. (2018). "Cyberbiosecurity: From Naive Trust to Risk Awareness". Trends in Biotechnology 36 (1): 4–7. doi:10.1016/j.tibtech.2017.10.012. PMID 29224719. 
  17. National Research Council (2000). "Bioinformatics: Converting Data to Knowledge". National Academies Press. doi:10.17226/9990. https://www.nap.edu/catalog/9990/bioinformatics-converting-data-to-knowledge-workshop-summary. 
  18. Pundir, S;. Martin, M.J.; O'Donovan, C. (2017). "UniProt Protein Knowledgebase". Methods in Molecular Biology 1558: 41–55. doi:10.1007/978-1-4939-6783-4_2. PMC PMC5565770. PMID 28150232. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC5565770. 
  19. Gurevich, A.; Saveliev, V.; Vyahhi, N.; Tesler, G. (2013). "QUAST: Quality assessment tool for genome assemblies". Bioinformatics 29 (8): 1072–5. doi:10.1093/bioinformatics/btt086. PMC PMC3624806. PMID 23422339. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC3624806. 
  20. Simão, F.A.; Waterhouse, R.M.; Ioannidis, P. et al. (2015). "BUSCO: Assessing genome assembly and annotation completeness with single-copy orthologs". Bioinformatics 31 (19): 3210–2. doi:10.1093/bioinformatics/btv351. PMID 26059717. 
  21. Tennessen, K.; Andersen, E.; Clingenpeel, S. et al. (2016). "ProDeGe: A computational protocol for fully automated decontamination of genomes". The ISME Journal 10 (1): 269–72. doi:10.1038/ismej.2015.100. PMC PMC4681846. PMID 26057843. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC4681846. 
  22. Lux, M.; Krüger, J.; Rinke, C. et al. (2016). "acdc - Automated Contamination Detection and Confidence estimation for single-cell genome data". BMC Bioinformatics 17 (1): 543. doi:10.1186/s12859-016-1397-7. PMC PMC5168860. PMID 27998267. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC5168860. 
  23. Parks, D.H.; Imelfort, M.; Skennerton, C.T. et al. (2015). "CheckM: Assessing the quality of microbial genomes recovered from isolates, single cells, and metagenomes". Genome Research 25 (7): 1043–55. doi:10.1101/gr.186072.114. PMC PMC4484387. PMID 25977477. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC4484387. 
  24. Dittami, S.M.; Corre, E. (2017). "Detection of bacterial contaminants and hybrid sequences in the genome of the kelp Saccharina japonica using Taxoblast". PeerJ 5: e4073. doi:10.7717/peerj.4073. PMC PMC5695246. PMID 29158994. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC5695246. 
  25. McNair, K.; Edwards, R.A. (2015). "GenomePeek: An online tool for prokaryotic genome and metagenome analysis". PeerJ 3: e1025. doi:10.7717/peerj.1025. PMC PMC4476108. PMID 26157610. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC4476108. 
  26. Kryukov, K.; Imanishi, T. (2016). "Human Contamination in Public Genome Assemblies". PLoS One 11 (9): e0162424. doi:10.1371/journal.pone.0162424. PMC PMC5017631. PMID 27611326. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC5017631. 
  27. Cornet, L.; Meunier, L.; Van Vlierberghe, M. et al. (2018). "Consensus assessment of the contamination level of publicly available cyanobacterial genomes". PLoS One 13 (7): e0200323. doi:10.1371/journal.pone.0200323. PMC PMC6059444. PMID 30044797. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC6059444. 
  28. Lu, J.; Salzberg, S.L. (2018). "Removing contaminants from databases of draft genomes". PLoS Computational Biology 14 (6): e1006277. doi:10.1371/journal.pcbi.1006277. PMC PMC6034898. PMID 29939994. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC6034898. 

Notes

This presentation is faithful to the original, with only a few minor changes to presentation, grammar, and punctuation. In some cases important information was missing from the references, and that information was added. The footnote in the original material were turned into an inline references for this version.