Journal:Ten simple rules to enable multi-site collaborations through data sharing

From LIMSWiki
Jump to navigationJump to search
Full article title Ten simple rules to enable multi-site collaborations through data sharing
Journal PLOS Computational Biology
Author(s) Boland, Mary Regina; Karczewski, Konrad J.; Tatonetti, Nicholas P.
Author affiliation(s) Columbia University (NY), Broad Institute of MIT and Harvard, Massachusetts General Hospital
Primary contact Email: mary dot boland @ columbia dot edu
Year published 2017
Volume and issue 13(1)
Page(s) e1005278
DOI 10.1371/journal.pcbi.1005278
ISSN 1553-7358
Distribution license Creative Commons Attribution 4.0 International
Website http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1005278
Download http://journals.plos.org/ploscompbiol/article/file?id=10.1371/journal.pcbi.1005278&type=printable (PDF)

Introduction

Open access, open data, and software are critical for advancing science and enabling collaboration across multiple institutions and throughout the world. Despite near universal recognition of its importance, major barriers still exist to sharing raw data, software, and research products throughout the scientific community. Many of these barriers vary by specialty[1], increasing the difficulties for interdisciplinary and/or translational researchers to engage in collaborative research. Multi-site collaborations are vital for increasing both the impact and the generalizability of research results. However, they often present unique data sharing challenges. We discuss enabling multi-site collaborations through enhanced data sharing in this set of Ten Simple Rules.

Collaboration is an essential component of research[2] that takes many forms, including internal (across departments within a single institution) and external collaborations (across institutions). However, multi-site collaborations with more than two institutions encounter more complex challenges because of institutional-specific restrictions and guidelines.[3] Vicens and Bourne focus on collaborators working together on a shared research grant.[4] They do not discuss the specific complexities of multi-site collaborations and the vital need for enhanced data sharing in the multi-site and large-scale collaboration context, in which participants may or may not have the same funding source and/or research grant.

While challenging, multi-site collaborations are equally rewarding and result in increased research productivity.[5][6] One highly successful multi-site and translational collaboration is the Electronic Medical Records and Genomics (eMERGE) network (URL: https://emerge.mc.vanderbilt.edu/) initiated in 2007.[7] The eMERGE network links biorepository data with clinical information from electronic health records (EHRs). They were able to find novel associations and replicate many known associations between genetic variants and clinical phenotypes that would have been more difficult without the collaboration.[8] eMERGE members also collaborated with other consortiums and networks, including the Alzheimer’s Disease Genetics Consortium[9] and the NINDS Stroke Genetics Network[10], to name a few. Other successful collaborations include OHDSI: Observational Health Data Sciences and Informatics (http://www.ohdsi.org/), which builds off of the methodology from the Observational Medical Outcomes Partnership (OMOP)[11], and CIRCLE: Clinical Informatics Research Collaborative (http://circleinformatics.org/). In genetics, there are many consortiums, including ExAC: The Exome Aggregation Consortium (http://exac.broadinstitute.org/), the 1000 Genomes Project Consortium (http://www.1000genomes.org/), the Australian BioGRID (https://www.biogrid.org.au/), The Cancer Genome Atlas (TCGA) (http://cancergenome.nih.gov/), Genotype-Tissue Expression Portal (GTEx: http://www.gtexportal.org/home/), and Encyclopedia of DNA Elements at UCSC (ENCODE: https://genome.ucsc.edu/ENCODE/) among others.

Based on our experiences as both users and participants in collaborations, we present 10 simple rules on how to enable multi-site collaborations within the scientific community through enhanced data sharing. The rules focus on understanding privacy constraints, utilizing proper platforms to facilitate data sharing, thinking in global terms, and encouraging researcher engagement through incentives. We present these 10 rules in the form of a pictograph of modern life (Fig. 1), and we provide a table of example sources and sites that can be referred to for each of the ten rules (Table 1). Please note that this table is not meant to be exhaustive, only to provide some sample resources of use to the research community.


Fig1 Boland PLOSCompBio2017 13-1.png

Figure 1. Modern life context for the ten simple rules: This figure provides a framework for understanding how the “Ten Simple Rules to Enable Multi-site Collaborations through Data Sharing” can be translated into easily understood modern life concepts. Rule 1 is Open-Source Software. The openness is signified by a window to a room filled with algorithms that are represented by gears. Rule 2 involves making the source data available whenever possible. Source data can be very useful for researchers. However, data are often housed in institutions and are not publicly accessible. These files are often stored externally; therefore, we depict this as a shed or storehouse of data, which, if possible, should be provided to research collaborators. Rule 3 is to “use multiple platforms to share research products.” This increases the chances that other researchers will find and be able to utilize your research product—this is represented by multiple locations (i.e., shed and house). Rule 4 involves the need to secure all necessary permissions a priori. Many datasets have data use agreements that restrict usage. These restrictions can sometimes prevent researchers from performing certain types of analyses or publishing in certain journals (e.g., journals that require all data to be openly accessible); therefore, we represent this rule as a key that can lock or unlock the door of your research. Rule 5 discusses the privacy issues that surround source data. Researchers need to understand what they can and cannot do (i.e., the privacy rules) with their data. Privacy often requires allowing certain users to have access to sections of data while restricting access to other sections of data. Researchers need to understand what can and cannot be revealed about their data (i.e., when to open and close the curtains). Rule 6 is to facilitate reproducibility whenever possible. Since communication is the forte of reproducibility, we depicted it as two researchers sharing a giant scroll, because data documentation is required and is often substantial. Rule 7 is to “think global.” We conceptualize this as a cloud. This cloud allows the research property (i.e., the house and shed) to be accessed across large distances. Rule 8 is to publicize your work. Think of it as “shouting from the rooftops.” Publicizing is critical for enabling other researchers to access your research product. Rule 9 is to “stay realistic.” It is important for researchers to “stay grounded” and resist the urge to overstate the claims made by their research. Rule 10 is to be engaged, and this is depicted as a person waving an “I heart research” sign. It is vitally important to stay engaged and enthusiastic about one’s research. This enables you to draw others to care about your research.


Tab1 Boland PLOSCompBio2017 13-1.png

Table 1. Example sources and sites for each of the ten simple rules

Definitions

In this paper, we use the term "research product" to include all results from research. This includes algorithms, developed software tools, databases, raw source data, cleaned data, and various metadata generated as a result of the research activity. We differentiate this from "data," which comprises the primary "facts and statistics collected together for analysis" for that particular collaboration. Therefore, data could include genetic data or clinical data. By these definitions, developed software tools are not "data" but "research products." Novel genetic sequences collected for analysis would be considered "raw source data," which is a type of "research product."

Rule 1: Make software open-source

The cornerstone of facilitating multi-site collaborations is to enhance data sharing and make software open-source.[12] By allowing the source code to be open, researchers allow others to both reproduce their work and build upon it in novel ways. To engage in multi-site collaborations, it is necessary for collaborators to have access to code in a repository that is shared among collaborators (although, this could be private and not open to the general public). When the study is complete and the paper is under review and/or published, a stable copy of the code should be made available to the general public. Internal sharing allows the code to be developed, while public sharing of a stable version allows the code to be refined and built upon by others.

Many researchers still limit access to their work despite the known advantages of making software open-source upon publication (e.g., higher impact publications[5]). For example, they allow users to interact with their algorithm by inputting data and receiving results on a web platform, while the backend algorithm often remains inaccessible. Masum et al. advocate the reuse of existing code in their Ten Simple Rules for cultivating open science.[13] However, this is often easier said than done. As long as the back-end algorithms remain hidden, open science will not be possible. Therefore, it is essential for researchers interested in participating in multi-site collaborations to make their software code and algorithms open. Because making software truly "open" can be complex, Prlić and Proctor provide Ten Simple Rules to assist researchers in making their software open-source.[12] Truly open-source software is an essential component in collaborations.[13] Openness also has advantages for the researchers themselves. With more eyes on the source code, others within the community can refine the code, leading to greater identification and correction of errors. There are several methods for sharing software code. If you use the R platform, then libraries can be shared with the entire open-source community via CRAN (https://cran.r-project.org/) and Bioconductor, which is specifically for biologically related algorithms (https://www.bioconductor.org/). Code can also be shared on Github with issue trackers for error detection.

Rule 2: Provide open-source data

Deposit source data in appropriate repositories

Whenever possible, it is important to make source data available. Openness benefits your collaborators by allowing them to perform additional analyses easily. Source data could include not only processed or cleaned data used in algorithms but also raw data files. These files can often be very large; therefore, they are often stored in some external site or data warehouse. The National Center for Biotechnology Information (NCBI) maintains the Sequence Read Archive (SRA) (https://www.ncbi.nlm.nih.gov/sra) and the Gene Expression Omnibus (GEO) (https://www.ncbi.nlm.nih.gov/geo/); both are great places to deposit source data, if appropriate.

In addition to raw data files, it is also helpful to provide intermediate data files at various stages of processing. If comparing your results to those in the literature, it can also be useful to provide a meta-analysis with publications (along with PubMed IDs) that detail those publications that support and refute the results you obtained.

Data sharing is vitally important for multi-site collaborations by allowing researchers to compare results from across vastly different study populations, which increases the generalizability of the findings.[14] While a multi-site research project is still ongoing, data can be shared in a private shared space until all necessary data quality checks have been conducted and the findings have been published. After publication, data can be deposited in GEO, SRA, ClinVar (https://www.ncbi.nlm.nih.gov/clinvar/), and any other domain-specific sites that are appropriate for source data deposition.

Consider middle-ground data sharing approaches for sensitive data

Raw source data is not always fully shareable with the public. This can be because of data use restrictions (see rule 4) or privacy concerns (see rule 5). Alternative mechanisms exist for sharing portions of data with the research community. For example, the database for Genotypes and Phenotypes or dbGaP (https://www.ncbi.nlm.nih.gov/gap) provides data holders with two levels of access: open and controlled. The open selection allows for broad release of nonsensitive data online, whereas the controlled release allows sensitive datasets to be shared with other investigators, provided certain restrictions are met. This increases the ability for researchers to share portions of their data that would not be shareable otherwise.

In addition to the restricted data sharing option provided by dbGaP, others have looked at ways of developing middle-ground approaches for sharing sensitive raw data or metadata. Several of these mid-level approaches use federated access systems that allow researchers to query databases containing sensitive data while preventing direct access to the data itself. An example within the United States is the Shared Health Research Information Network (SHRINE), which provides a federated system that is Health Insurance Portability and Accountability Act (HIPAA) compliant.[15] International groups have also seen success in this area. BioGrid Australia (https://www.biogrid.org.au/) allows researchers to access hundreds of thousands of health records through a linked data platform where individual data holders maintain control of their data.[16] Researchers can then be provided with authorized access to certain elements within the data while restricting access to private sections of the medical data. These mid-level approaches facilitate collaboration both within the institution (i.e., across departments) and across institutions by allowing researchers to access sensitive data indirectly. They can even match patients to similar patients (for association analyses) while maintaining stringent privacy constraints.[17] Others provide summary statistics computed over large cohorts (e.g., ExAC browser/database), which maintains privacy while providing others with important information about the populations that can be used in subsequent analyses and comparisons.

Rule 3: Use multiple platforms to share research products

To collaborate with researchers from different backgrounds, it is often necessary to use multiple platforms when sharing data (as different disciplines often have different policies). Using multiple platforms allows individuals from diverse backgrounds to have access to your research product. General phrases like "open data" and "open science" are used commonly in the research community but provide little direction.[13] Research products take many different forms, including 1) raw source data regardless of collection type (e.g., health data, genomic data, survey data, and epidemiological data), 2) software code (mentioned in rule 1), and 3) metadata elements and results of computations used to generate figures published in scientific research. Some data types cannot be fully shared (e.g., EHR data; see rule 5), but most algorithms and summary results/statistics are shareable.

Each of these types of open data necessitates a different platform for data sharing. Figshare (https://figshare.com/) allows users to share data involving published figures. Github (https://github.com/) allows users to share code that is in development or published. For code that is well developed, open-source packages can be created, for example, an R library, which can be deposited in CRAN or Bioconductor. R libraries can be shared immediately on Github without any code checking — this is advisable for code that is still in development. However, when code is finalized, it can be submitted to Bioconductor as an R library. Approved libraries are vetted to ensure the code works well. Vignettes are also good to write to help new users get used to the R package. When collaborating across multiple sites, it is also important to have vignettes and sample source data to help users learn how to use the code even if R is not their language of choice. Data formats, differences among formats, and programming languages are important to consider when sharing data across multiple platforms. Different platforms often have different required formats. While it may seem tedious to translate code, source data, and documentation across multiple formats and data schemas, it can be very helpful, and it will increase the number of users that will find your data and results interesting.

To facilitate communication among members of a collaborative effort, there are many options, including Google forums and wiki webpages, among others. Others have specially designed websites for the sole purpose of allowing users to browse and download the data directly; one such website is the ExAC Browser (http://exac.broadinstitute.org/), which integrates data obtained from 17 different consortiums (http://exac.broadinstitute.org/about).[18]

Rule 4: Secure necessary permissions/data use agreements a priori

Some datasets have provisos that affect publication, and these need to be addressed a priori. For example, the ability for researchers to publish an algorithm that uses a government dataset can depend on the department that generated the data. For example, certain National Aeronautics and Space Administration (NASA) datasets stipulate that data usage requires users to add certain NASA employees to subsequent publications. This is an important stipulation. Others may disallow the deposition of data into an "open" platform as part of their data use agreements (http://above.nasa.gov/Documents/NGA_Data_Access_Agreement_new.pdf). These stipulations can hinder researchers attempting to produce transparent science.

Other datasets have data use agreements as an added layer to ensure that patients are protected. For example, the Surveillance, Epidemiology, and End Results (SEER) dataset linked with Medicare (i.e., SEER-Medicare dataset) requires that users submit the intended publication to their offices for pre-submission approval. This can seem burdensome to researchers; however, it is a condition of the data use agreement and, therefore, must be complied with. Researchers need to be aware of all provisos when including such data in their studies. Before publishing — or providing data in any type of platform whether open, restricted, or closed — it is important to secure all necessary provisions and data use agreements.

Rule 5: Know the privacy rules for your data

Data come with many caveats. For this reason, it is important to understand what you can and cannot do (i.e., the privacy rules) with your data. Keeping and maintaining data privacy is different from data use agreements (DUA, see rule 4). For example, data that is not sensitive may have restrictive DUAs for other reasons (e.g., data from a collaborator in industry). Also, privacy rules often involve your own source data, whereas DUAs become necessary when using data from collaborators or a government source.

Certain datasets, e.g., genomic and EHR data, may be impossible to fully publish on an open platform due to HIPAA privacy rules and other privacy concerns related to patient re-identifiability (http://www.hhs.gov/hipaa/for-professionals/privacy/). Therefore, it is important to know the privacy stipulations of all data used in your collaborations and how this affects the ability to share results among members of the team (especially when members of the team are at different institutions). Methods that anonymize patient information while allowing patient-level data sharing may be the way of the future.[19] However, institutional-specific policies and/or country-specific laws can limit or prevent usage of such methods. This is an important item to consider and discuss with all collaborators at the outset of any collaboration. We discuss some methods that can be used to provide some forms of sensitive data in a shareable federated space in rule 2.

Rule 6: Facilitate reproducibility

Another aspect of both data sharing and enabling multi-site collaborations is reproducibility. Sandve et al. provide Ten Simple Rules for facilitating research reproducibility in general.[20] Keeping track of research results and how data were generated is vital for reproducibility.[20] This site-level record keeping becomes vital when engaging in multi-site collaborations. If one aspect of a methodology is not conducted in the same way at one site, the overall results can be affected in drastic ways. In other words, reproducibility is a core requirement for successful collaborations.

In genetics and computational biology, the issue of standardizing results from across different types of gene sequencing platforms is a major issue.[21] Researchers that use a mixture of clinical and genetic data (for Phenome-Wide Association Studies, PheWAS[22]) often depend on local EHR terminology systems for identifying patient populations. Therefore, standard phenotype definitions are required and must be harmonized across multiple sites to ensure that the definitions are accurate at each site.[23] Several multi-site collaborations have developed platforms that provide links to all necessary documentation, code, and data schemas to help facilitate this process[24], including the eMERGE network. This step is integral to data sharing and enabling multi-site collaborations.

Rule 7: Think global

The importance of thinking globally cannot be overstated. Health care, genetics, climate, and all aspects of science affect the world as a whole. Therefore, it is important to think globally when performing scientific research. Most software languages are designed to be agnostic to the local language of the country. However, understanding and using these languages requires adequate documentation and user manuals to be provided in the local languages of the programmers/implementers. Despite this, open-source languages often provide user manuals in certain languages. For example, R is a popular open-source language yet has official documented translations in only four languages: English, Russian, German, and Chinese (https://www.r-project.org/other-docs.html). Problems can surface when collaborators in different regions run into difficulties with running R. This affects data sharing on a global scale and should be considered when collaborating on an international venue.

Translational mechanisms may also be necessary to understand and to harmonize country-specific terminology. This is especially important as definitions for obesity and many psychiatric conditions vary widely across the globe.[25] Even seemingly simple biological features (e.g., tall versus short) can be difficult to translate in global terms. For example, an average height Norwegian may appear to be tall in a different country. Translating biological features to common absolute metrics (e.g., height) helps to alleviate ambiguities that can occur from categorical variables. Certain diseases, especially psychiatric conditions, are extremely important to study at the multi-site level to increase the generalizability of the results.[14] However, psychiatric conditions are more difficult to translate without a thorough knowledge of how the condition is defined in the underlying country or region.[25] Solutions often involve using concrete measures, e.g., brain imaging analysis, versus subjective measures such as depression presence or absence.[14]

There are many layers to thinking on a global scale. There are mechanical differences (i.e., the software language and documentation) and also the conceptual differences (i.e., country- or region-specific medical definitions). Organizations such as the World Health Organization work tirelessly to integrate different conceptual interpretations of diseases into a standard guideline. Using these guidelines and not a country-specific guideline helps your research reach the broader scientific community.

Several groups have successfully integrated data across multiple countries and provided their data in an open form. The Max Planck Institute for Demographic Research (MPIDR) in Germany collaborated with two separate groups to produce two databases containing international data. Both datasets contain integrated results from over 30 countries. Additionally, all finished data (after cleaning) is made available to users in an open format via two specially designed databases: the Human Fertility Database (http://www.humanfertility.org/cgi-bin/main.php)[26] and the Human Mortality Database (http://www.mortality.org/).[27] Only cleaned data are returned to users in a standardized format, allowing users to easily compare countries with one another. The MPIDR collaborated with the Vienna Institute of Demography (Austria) in creating the Human Fertility Database and the University of California, Berkeley for the Human Mortality Database. They provide a good example of a group that successfully harmonized definitions across countries by overcoming international barriers, and they provided data back to researchers in an easily usable and standardized format. The group provides detailed descriptions of how they harmonized various timescales across countries in a methods document (http://www.humanfertility.org/Docs/methods.pdf) that could easily be submitted as a research report (see Rule 6).

Rule 8: Publicize your work

Publishing all aspects of your work in the appropriate venues is vital for maintaining a multi-site collaboration. This enables each aspect of your research to be assessed by appropriate peer reviewers. Publishing different aspects of your work in separate papers in separate journals allows your contributions to be seen by those most able to learn from your work. Remember, it is important to make your research work available to those who can benefit from your results. Depending on your findings, this can include methodologists, clinicians, epidemiologists, geneticists, and others.

New journals have been developed recently to facilitate open science, which are focused on certain aspects of research. For instance, there are several journals that do not require novelty as a requirement such as PLOS ONE, Scientific Reports, and Cell Reports. These journals are good choices for research results that may be part of a larger research project or collaborative but are not inherently novel. Other journals, such as Scientific Data and Database, are good choices for publishing a resource containing your collected research source data. It is often advisable to publish in data-focused journals simultaneously with an algorithm or results-focused paper that highlights the novel aspects of your research. In some cases, data can be published afterwards if it is part of a large collaborative and the database or user-interface is in production at the time that the main contribution is published.

Publishing in multiple venues is highly important for those engaged in multi-site collaborations, because these projects often involve a tremendous investment of time and resources from across many different organizations. Therefore, it is vital to highlight each and every research contribution that the collaboration has generated to facilitate further engagement from the community. If you are able to provide all raw source data on an open platform, there are new journals designed specifically to facilitate open science such as F1000 (https://f1000research.com/) that may be worth considering. F1000 is also a great source for intermediate results such as posters, which collaborators may have presented at various conferences while working towards the final finished paper. After publication, some collaborative groups effectively utilize blogging (both macro and micro) to communicate with other researchers and the general public. However, it is also important not to overstate the claims in any paper submission/publication or media regarding that publication but to stay focused on the individual contribution of that particular work.

Rule 9: Stay realistic, but aim high

When performing quality research, and collaborating with others, it is important not to overstate the claims of your research — either in publication or online. It is vitally important to resist the urge to overstate the claims and to remain both humble and grounded. This is critical in collaborations because if a researcher overstates the claim in a paper, or worse, shares data publicly that he or she is unable to do legally (e.g., via the stipulations in a DUA), then the paper may be retracted. This could result in irreparable damage to the collaborative group.

This rule also links back to rule 2: making the source data available. This allows others in the research community to check your work interactively, which can help prevent overstating research claims.[28] A site exists that posts retracted journal articles on a public forum, retractionwatch.com. The site includes not only instances of plagiarism and fabrication of data but also papers that are retracted due to human error on the part of an experiment (e.g., a protocol was not followed exactly as specified in the paper) or on the part of the analysis (e.g., the wrong type of statistical test was performed, making the conclusions not substantiated by the data).

So, stay realistic, but do not be afraid to challenge the status quo. Some of the most respected research today was research that challenged the current understanding of the leading scientists at that point in time; this includes the seminal works on Pangaea and even that DNA is composed of a double helix. These concepts were earth-shattering at the time and could have been completely wrong, but the researchers backing them were not afraid to make their theories, data, and results public. These are the things that change science. So, remain humble, do not intentionally overstate the claims of your research, but at the same time do not be afraid to challenge the current mindset and way of thinking. You may be completely off, or you may just be a groundbreaking innovator.

Rule 10: Be engaged

Be engaged with those using your research, your data, and your code. Communicate with them using various software social platforms such as Github, figshare, and so forth. Respond readily when users have questions and concerns. Attempt to follow the motto "release early, release often." Engage with researchers in non-traditional ways. For example, several collaborative efforts have created their own gear, e.g., t-shirts, to engage the community. One such collaborative is the open-source statistical modeling language STAN (http://mc-stan.org/). They have created their own line of STAN "swag" (http://mc-stan.org/shop/) to facilitate user engagement. Communicate often with the research community to convince them your research is worth caring about. The bottom line in collaboration is to care deeply about your research. If you care and you make it known that you care deeply about the problem, then it becomes possible to convince others that your research is important.

Concluding remarks

Collaborations, especially large, multi-site collaborations, contain many pitfalls that must be overcome. In this paper, we present 10 simple rules that will help researchers share their data and methods to facilitate successful and meaningful multi-site collaborations. We describe these rules and highlight several successful multi-site collaborations.

Funding

MRB was supported by NLM T15 LM00707 from Jul 2014–Jun 2016 and by the NCATS, NIH, through TL1 TR000082, formerly the NCRR, TL1 RR024158 from Jul 2016–Jun 2017. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Competing interests

The authors have declared that no competing interests exist.

References

  1. ↑ Reichman, O.J.; Jones, M.B.; Schildhauer, M.P. (2011). "Challenges and opportunities of open data in ecology". Science 331 (6018): 703–5. doi:10.1126/science.1197962. PMID 21311007. 
  2. ↑ Bozeman, B.; Fay, D.; Slade, C.P. (2013). "Research collaboration in universities and academic entrepreneurship: the-state-of-the-art". The Journal of Technology Transfer 38 (1): 1–67. doi:10.1007/s10961-012-9281-8. 
  3. ↑ Brown, P.; Morello-Frosch, R.; Brody, J.G. (2008). "IRB Challenges in Multi-Partner Community-Based Participatory Research". Proceedings of The American Sociological Association Annual Meeting 2008: 1-31. https://www.brown.edu/research/research-ethics/irb-challenges-multi-partner-community-based-participatory-research. 
  4. ↑ Vicens, Q.; Bourne, P.E. (2007). "Ten simple rules for a successful collaboration". PLOS Computational Biology 3 (3): e44. doi:10.1371/journal.pcbi.0030044. PMC PMC1847992. PMID 17397252. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1847992. 
  5. ↑ 5.0 5.1 Jones, B.F.; Wuchty, S.; Uzzi, B. (2008). "Multi-university research teams: shifting impact, geography, and stratification in science". Science 322 (5905): 1259-62. doi:10.1126/science.1158357. PMID 18845711. 
  6. ↑ Börner, K.; Contractor, N.; Falk-Krzesinski, H.J. et al. (2010). "A multi-level systems perspective for the science of team science". Science Translational Medicine 2 (49): 49cm24. doi:10.1126/scitranslmed.3001399. PMC PMC3527819. PMID 20844283. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3527819. 
  7. ↑ Gottesman, O.; Kuivaniemi, H.; Tromp, G. et al. (2013). "The Electronic Medical Records and Genomics (eMERGE) Network: Past, present, and future". Genetics in Medicine 15 (10): 761-71. doi:10.1038/gim.2013.72. PMC PMC3795928. PMID 23743551. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3795928. 
  8. ↑ Feng, Q.; Wei, W.Q.; Chung, C.P. et al. (2016). "The effect of genetic variation in PCSK9 on the LDL-cholesterol response to statin therapy". The Pharmacogenomics Journal. doi:10.1038/tpj.2016.3. PMC PMC4995153. PMID 26902539. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4995153. 
  9. ↑ Karch, C.M.; Ezerskiy, L.A.; Bertelsen, S. et al. (2016). "Alzheimer's Disease Risk Polymorphisms Regulate Gene Expression in the ZCWPW1 and the CELF1 Loci". PLOS One 11 (2): e0148717. doi:10.1371/journal.pone.0148717. PMC PMC4769299. PMID 26919393. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4769299. 
  10. ↑ Malik, R.; Traylor, M.; Pulit, S.L. et al. (2016). "Low-frequency and common genetic variation in ischemic stroke: The METASTROKE collaboration". Neurology 86 (13): 1217-26. doi:10.1212/WNL.0000000000002528. PMC PMC4818561. PMID 26935894. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4818561. 
  11. ↑ Stang, P.E.; Ryan, P.B.; Racoosin, J.A. et al. (2010). "Advancing the science for active surveillance: rationale and design for the Observational Medical Outcomes Partnership". Annals of Internal Medicine 153 (9): 600–6. doi:10.7326/0003-4819-153-9-201011020-00010. PMID 21041580. 
  12. ↑ 12.0 12.1 Prlić, A.; Procter, J.B. (2012). "Ten simple rules for the open development of scientific software". PLOS Computational Biology 8 (12): e1002802. doi:10.1371/journal.pcbi.1002802. PMC PMC3516539. PMID 23236269. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3516539. 
  13. ↑ 13.0 13.1 13.2 Masum, H.; Rao, A.; Good, B.M. et al. (2013). "Ten simple rules for cultivating open science and collaborative R&D". PLOS Computational Biology 9 (9): e1003244. doi:10.1371/journal.pcbi.1003244. PMC PMC3784487. PMID 24086123. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3784487. 
  14. ↑ 14.0 14.1 14.2 Pearlson, G. (2009). "Multisite collaborations and large databases in psychiatric neuroimaging: Advantages, problems, and challenges". Schizophrenia Bulletin 35 (1): 1–2. doi:10.1093/schbul/sbn166. PMC PMC2643967. PMID 19023121. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2643967. 
  15. ↑ Weber, G.M.; Murphy, S.N.; McMurry, A.J. et al. (2009). "The Shared Health Research Information Network (SHRINE): A prototype federated query tool for clinical data repositories". JAMIA 16 (5): 624-30. doi:10.1197/jamia.M3191. PMC PMC2744712. PMID 19567788. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2744712. 
  16. ↑ Merriel, R.B.; Gibbs, P.; O'Brien, T.J. et al. (2011). "BioGrid Australia facilitates collaborative medical and bioinformatics research across hospitals and medical research institutes by linking data from diverse disease and data types". Human Mutation 32 (5): 517-25. doi:10.1002/humu.21437. PMID 21309032. 
  17. ↑ Boyle, D.I.; Rafael, N. (2011). "BioGrid Australia and GRHANITE: Privacy-protecting subject matching". Studies in Health Technology and Informatics 168: 24-34. PMID 21893908. 
  18. ↑ Lek, M.; Karczewski, K.J.; Minikel, E.V. et al. (2016). "Analysis of protein-coding genetic variation in 60,706 humans". Nature 536 (7616): 285–91. doi:10.1038/nature19057. PMC PMC5018207. PMID 27535533. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5018207. 
  19. ↑ El Emam, K.; Rodgers, S.; Malin, B. (2015). "Anonymising and sharing individual patient data". BMJ 350: h1139. doi:10.1136/bmj.h1139. PMC PMC4707567. PMID 25794882. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4707567. 
  20. ↑ 20.0 20.1 Sandve, G.K.; Nekrutenko, A.; Taylor, J. et al. (2013). "Ten simple rules for reproducible computational research". PLOS Computational Biology 9 (10): e1003285. doi:10.1371/journal.pcbi.1003285. PMC PMC3812051. PMID 24204232. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3812051. 
  21. ↑ Bammler, T.; Beyer, R.P.; Bhattacharya, S. et al. (2005). "Standardizing global gene expression analysis between laboratories and across platforms". Nature Methods 2 (5): 351-6. doi:10.1038/nmeth754. PMID 15846362. 
  22. ↑ Denny, J.C.; Ritchie, M.D.; Basford, M.A. et al. (2010). "PheWAS: demonstrating the feasibility of a phenome-wide scan to discover gene-disease associations". Bioinformatics 26 (9): 1205-10. doi:10.1093/bioinformatics/btq126. PMC PMC2859132. PMID 20335276. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2859132. 
  23. ↑ Newton, K.M.; Peissig, P.L.; Kho, A.N. et al. (2013). "Validation of electronic medical record-based phenotyping algorithms: Results and lessons learned from the eMERGE network". JAMIA 20 (e1): e147-54. doi:10.1136/amiajnl-2012-000896. PMC PMC3715338. PMID 23531748. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3715338. 
  24. ↑ Pathak, J.; Wang, J.; Kashyap, S. et al. (2011). "Mapping clinical phenotype data elements to standardized metadata repositories and controlled terminologies: The eMERGE Network experience". JAMIA 18 (4): 376-86. doi:10.1136/amiajnl-2010-000061. PMC PMC3128396. PMID 21597104. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3128396. 
  25. ↑ 25.0 25.1 Diener, E.; Oishi, S.; Lucas, R.E. (2003). "Personality, culture, and subjective well-being: Emotional and cognitive evaluations of life". Annual Review of Psychology 54: 403-25. doi:10.1146/annurev.psych.54.101601.1450561. PMID 12172000. 
  26. ↑ "The Human Fertility Database". Max Planck Institute for Demographic Research and Vienna Institute of Demography. http://www.humanfertility.org/cgi-bin/main.php. Retrieved 06 October 2016. 
  27. ↑ "The Human Mortality Database". University of California, Berkeley and Max Planck Institute for Demographic Research. http://www.mortality.org/. Retrieved 06 October 2016. 
  28. ↑ Senn, S.J. (2009). "Overstating the evidence: Double counting in meta-analysis and related problems". BMC Medical Research Methodology 9: 10. doi:10.1186/1471-2288-9-10. PMC PMC2653069. PMID 19216779. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2653069. 

Notes

This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added.

Retrieved from "https://www.limswiki.org/index.php?title=Journal:Ten_simple_rules_to_enable_multi-site_collaborations_through_data_sharing&oldid=29514"