Journal:Health sciences libraries advancing collaborative clinical research data management in universities

Full article title	Health Sciences Libraries Advancing Collaborative Clinical Research Data Management in Universities
Journal	Journal of eScience Librarianship
Author(s)	Bardyn, Tania P.; Patridge, Emily F.; Moore, Michael T.; Koh, Jane J.
Author affiliation(s)	University of Washington
Primary contact	Email: bardyn at uw dot edu
Year published	2018
Volume and issue	7(2)
Page(s)	e1130
DOI	10.7191/jeslib.2018.1130
ISSN	2161-3974
Distribution license	Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International
Website	https://escholarship.umassmed.edu/jeslib/vol7/iss2/4/
Download	https://escholarship.umassmed.edu/cgi/viewcontent.cgi?article=1130&context=jeslib (PDF)

Abstract

Purpose: Medical libraries need to actively review their service models and explore partnerships with other campus entities to provide better-coordinated clinical research management services to faculty and researchers. TRAIL (Translational Research and Information Lab), a five-partner initiative at the University of Washington (UW), explores how best to leverage existing expertise and space to deliver clinical research data management (CRDM) services and emerging technology support to clinical researchers at UW and collaborating institutions in the Pacific Northwest.

Methods: The initiative offers 14 services and a technology-enhanced innovation lab located in the Health Sciences Library (HSL) to support the University of Washington clinical and research enterprise. Sharing of staff and resources merges library and non-library workflows, better coordinating data and innovation services to clinical researchers. Librarians have adopted new roles in CRDM, such as providing user support and training for UW’s Research Electronic Data Capture (REDCap) instance.

Results: TRAIL staff are quickly adapting to changing workflows and shared services, including teaching classes on tools used to manage clinical research data. Researcher interest in TRAIL has sparked new collaborative initiatives and service offerings. Marketing and promotion will be important for raising researchers’ awareness of available services.

Conclusions: Medical librarians are developing new skills by supporting and teaching CRDM. Clinical and data librarians better understand the information needs of clinical and translational researchers by being involved in the earlier stages of the research cycle and identifying technologies that can improve healthcare outcomes. At health sciences libraries, leveraging existing resources and bringing services together is central to how university medical librarians will operate in the future.

Keywords: clinical research data management, sharing workflows, virtual reality, library innovation space, national network of libraries of medicine, REDCap, librarian roles

Introduction

Medical libraries confront an array of challenges in the ways they have traditionally operated to support their clinical and research missions. Library directors work with resource constraints and pressure to reduce costs, implement new services and applications, and improve outcomes. For shared workflows, medical libraries require active review of their service models and exploration of partnerships with other campus departments and entities. Supporting clinical research data management (CRDM) and uses of emerging technologies in healthcare is a good solution for many medical libraries interested in evolving their service model.

Increased attention is being paid to clinical data that may be developed and maintained by hospitals and companies. This data, based on patients’ contacts and electronic health records, could provide detailed information to those involved in population health research in and outside hospitals. CRDM expands beyond patient records and literature searches into data visualization, survey creation, bioinformatics consultation, and fostering the use of emerging technologies, like virtual and augmented reality.

Initiatives like the Translational Research and Information Lab (TRAIL), founded at the University of Washington’s (UW) Health Sciences Library (HSL), are vital to the future of medical libraries. Established in late 2016, to improve clinical research support, the five-partner initiative explores how best to coordinate data and innovation services for clinical researchers. TRAIL offers 14 online and in-person services and an innovation space in the library. It has provided the groundwork for the pooling of partner resources and extended collaboration between leading medical research-related campus departments and services, establishing a definitive support system for translational clinical research. (Figure 1).

Figure 1. Rendering of a VR cardiac surgery case conference in TRAIL

Cross-partner staff sharing and IT support has also provided HSL librarians with opportunities to spread their expertise beyond the confines of the library, benefiting the clinical research community and libraries as a whole.

Background

University of Washington Health Sciences Library

The UW HSL supports four hospitals in the Puget Sound, the WWAMI (Washington, Wyoming, Alaska, Montana, and Idaho) region, and six health sciences schools: Medicine, Dentistry, Public Health, Pharmacy, Nursing, and Social Work. The mission is to advance scholarship, research, education, and health care by anticipating information needs, providing essential resources and services, and facilitating learning for the greater health sciences community. The 35 full-time employees include librarians and staff who teach students, researchers, and clinicians how to think critically, how to access and interpret available information, and what to do if little or no evidence exists.

As with most medical libraries, HSL’s collection budget is almost exclusively dedicated to electronic material, with existing print material seldom circulating or used, except by request from other institutions through an interlibrary loan service.

Translational Research and Information Lab (TRAIL)

In assessing UW Libraries services during a triennial survey, a recurring theme of the HSL emerged: health sciences researchers and faculty desired more support from the library in their research. More specifically, faculty indicated needs for services that support managing, archiving, and preserving research data.

To fill that research support gap, the Associate Dean and Director of the Library explored partnerships with other departments. Meetings convened with UW’s Chief Information Research Officer and the leadership of Institute for Translational Health Sciences (ITHS), Department of Biomedical Information and Medical Education (BIME), and UW Medicine Information Technology Services (ITS) explored how best to address this gap and provide better-coordinated clinical research management services to faculty and researchers. Subsequent meetings were held with the Chair of BIME and the Associate Director for the National Network of Libraries of Medicine, Pacific Northwest Region (NNLM PNR), to discuss a partnership to share the provision of CRDM, team science services and staffing to aid campus and regional researchers. Those partnerships link to the National Library of Medicine’s (NLM) data initiative and develop a mechanism for bringing NLM programs more directly to researchers affiliated with the Clinical and Translational Science Awards (CTSA).

The partner leadership agreed on an overarching mandate: to provide operational coordination of innovation, clinical research and data management, team science, and informatics services to health sciences researchers. Six guiding principles were established: collaboration, quality, assessment, diversity, education, and access (Figure 2).

Figure 2. TRAIL guiding principles

Together, these partners officially launched the TRAIL initiative at the UW campus in 2016 by pooling staffing and designing a space in the library to house a technological-enhanced and collaborative innovation space to support over 4,900 faculty and researchers in 30 departments at the University. An Assistant Director of Clinical Research & Data Services (CRDS) was hired to lead the initiative and implement three goals in Year 1:

1. to provide an innovation space for UW-affiliated researchers to meet and collaborate;
2. to introduce, triage, and provide consultations on research data management tools (e.g., REDCap); and
3. to merge workflows and staffing to develop a new service model for supporting researchers at UW.

Methodology

Building a researcher-focused space

HSL floor space is predominantly dedicated to general student study space, offering solo quiet zones, general open study areas, and bookable group study rooms. However, those student-focused spaces aren’t conducive to supporting researchers needing dedicated, technology-enhanced meeting and collaborative spaces.

TRAIL’s first step was to develop a suitable, innovative, researcher-focused, and technology-enhanced innovation lab within HSL. Due to the limiting infrastructure, building a completely new room was deemed too expensive. With study spaces in high demand, particularly around exams, it was more logical to convert an existing office space, rather than take over a student study area.

An 838-sq. ft. room, centrally located on the lower of HSL’s two floors, was selected to become TRAIL. A 10-person committee from the five partner organizations selected the final design and technology choices after consulting with administrators, librarians, IT staff, and key faculty stakeholders. The space was designed by an interior designer. CompView, an audiovisual integrator based in Seattle, helped decide what technologies to incorporate into the lab, with renovations overseen by the Assistant Director of CRDS. Current elements in TRAIL include:

• Data wall: The space features six 55” LCD panels configured as a data wall in a three-wide-by-two-high arrangement. The data wall can be set up in various configurations, such as one image displayed across all six screens, a separate image for each of the six, or a hybrid of one image across four screens with separate images across the other two. Each individual panel is 1920×1080 HD, with a total display resolution of 5760×2160, and offering more pixels than a 4K display. An HSL IT staff member is assigned to assist with setup and troubleshoot issues for first-time users.

• Workstation: The data wall is supported by its own dedicated internet-equipped work station with USB ports to allow users to upload presentations and other files. Users can project their own laptops and mobile devices onto the data wall using standard display methods (e.g., HDMI, VGA, etc.), allowing for various display arrangements to be showcased across the six screens.

• Seating: TRAIL can comfortably seat up to 40 guests at any time but is ideally suited to hosting smaller gatherings of 5-15 people. A semicircular campfire-style lounge couch—the space’s signature furniture accent—energizes researchers with a collaborative seating arrangement that allows up to eight people to converse and brainstorm while using the data wall.

• White boards: The room’s east wall is covered in a white board for larger team discussions, with a medium-sized board situated in a separate corner to support break-out groups.

• Virtual reality gear: HSL is currently exploring virtual reality to host cardiothoracic pre-surgical consultations in TRAIL. The project includes a HTC Vive, an Oculus Rift, and an MSI Titan gaming laptop. See the next subsection for more details.

The space can be booked by UW clinical researchers using an online form on the HSL website, which automatically generates an email response to the requester. An HSL administrative coordinator receives the request, confirms the space is available at the desired time, and sends an email with the requested information to the TRAIL ticketing system. A librarian monitoring the ticketing system manages the room request and closes the ticket after the event. A calendar of upcoming bookings is also available on the HSL website.

TRAIL used a two-phase qualitative case study design to evaluate the data visualization space. In the first phase, reservation forms were analyzed to measure use of the data visualization services. In the second phase, follow-up surveys were analyzed to assess user feedback. In the first year and a half, the room was booked for 156 separate events, with approximately a third of these reservations used for clinical or health sciences data visualization. Clinical researchers indicated they are more likely to use the data wall in the HSL space because it is state-of-the-art technology and is located closer to the hospital and labs than any other data wall on campus. One unanticipated outcome was ensuring that the team and researchers using the space have process documents, support, and other documentation resources to accomplish data visualization functions. As a result, TRAIL implemented a concierge model, whereby the researchers are assigned a library staff person who works with them to conduct training and use of the data wall. To complement the concierge model, IT staff programmed skills into an Amazon Echo so it could be used by researchers to help operate the data wall. The data visualization evaluation model could offer other academic and medical libraries ideas to provide university-wide or systems-level clinical research training opportunities.

Designing clinical research support services

Nine trends identified by the American Library Association’s Center for the Future of Libraries helped lay the foundation for the development of services:

• Collective impact
• Connected learning
• Connected toys
• Creative placemaking
• Data everywhere
• Haptic technology
• Sharing economy
• Virtual reality
• Voice control

Through a series of discussions with key user stakeholders, a set of 14 researcher-driven services were devised to foster innovation and improve faculty and researcher access to CRDM support. The services are:

1. REDCap: Most requested TRAIL service. Research Electronic Data Capture (REDCap) is a rapidly evolving web tool for building and managing online surveys and databases. Developed at Vanderbilt University by translational researchers and for translational researchers, REDCap features a high degree of customizability for forms and advanced user right control. It also offers free, unlimited survey functionality, a sophisticated export module with support for all the popular statistical programs while supporting HIPAA compliance. HSL’s implementation of REDCap and use of librarians as support is discussed in more detail in the following section.

2. Data wall: Second-most requested TRAIL service. TRAIL has six 55” LCD panels configured as a data wall for presentations, teaching sessions, and collaborative meetings. Participants can use each panel separately, all six panels as one screen, or as one large four-panel screen and two one-panel screens. An HSL IT staff member can help set up and monitor the data wall during a user meeting or event.

3. Virtual reality: In the fall of 2017, HSL received an Institute of Museum and Library Services (IMLS) grant to develop a virtual reality space for cardiothoracic presurgical consultations in TRAIL. Through the project, HSL has worked with leading surgeons, radiologists, nurses, and researchers on using patient-specific imaging in a virtual reality environment. TRAIL has been outfitted with both an HTC Vive with full room-scale capabilities and an Oculus Rift headset, both driven by an MSI Titan laptop with an NVIDIA GeForce 1080 graphics card. HSL staff collaborated with UW Medicine ITS to install three software packages—Xcelera, ORCA PowerChart, and GE AW Server Access—in TRAIL to allow clinicians to access and analyze confidential patient information and scan imagery directly from the UW Medicine servers.

4. Mobile application development consultations: Using experience from developing a mobile app for use by Pacific Northwest first responders, HSL staff provide consultations and support on logistics of creating mobile applications for Android and iPhones. Consultation topics can include assistance with design elements, such as logo, icon, and website creation, configuring a development workstation, and publishing projects on Google Play and the Apple App Store.

5. Technology tools: Third-most requested TRAIL service. Researchers can learn to use devices like Amazon Echo, Samsung 360, and Apple TV through TRAIL technology with HSL IT support, allowing them to explore their effectiveness for future research projects. Product developers are eager to expand the reach of voice-controlled technology across audiences, including healthcare.

6. Computing environments: UW Medicine ITS staff help researchers develop strategies to store data using cloud-based environments.

7. Omics data analysis: Datasets generated by “-omics” disciplines (genomics, proteomics, epi-genomics, etc.) are usually very large, making it difficult for most researchers to analyze the resulting data in a meaningful way. The BIME department helps researchers connect to the proper resources and people with the right expertise to extract the most knowledge from datasets.

8. Accessing UW electronic health record (EHR) data: The UW medical system generates a tremendous amount of information on a daily basis. This treasure trove of data can be used for clinical research in a number of interesting and unique ways. Consultations discuss best practices for accessing the data warehouse.

9. Biomedical informatics consultations: The biomedical informatics team is a resource for service, education, and innovation in the management of clinical and translational biomedical data. It assists researchers with extracting useful data from electronic medical record data sources, and hosts and supports several free self-service tools for research data capture and exploration of de-identified cohorts.

10. Clinical study cohort discovery: LEAF is a cohort-discovery tool developed at UW that is ideal for quickly identifying cohorts of patients without the need for extensive programming experience. It also has export capabilities via REDCap for researchers with appropriate approval.

11. Biostatistical consultations: Partners’ biostatistical experts can provide coordinated collaboration for researchers and trainees at every stage of the research study. Users can set up a free initial consultation to receive expert advice on topics such as study design, sample size estimation, data collection, and multicenter study coordination.

12. Clinical research support: TRAIL connects researchers to the Translational Research Unit (TRU), the Pediatric Clinical Research Center (PCRC), the Regional Clinical Dental Research Center (RCDRC), and the Gene and Cell Therapy Lab (GCTL).

13. Population health research consultations: A Care Management and Population Health Librarian assists in systematic reviews, searching databases for population health information, and finding datasets.

14. Bioethics consultations: Staff provides advice to researchers, research staff, and personnel involved in protection of human subjects, and research participants who have questions about any aspect of a research study and assist when there are ethical issues that could benefit from in-depth conversation and analysis related to the development, implementation, or analysis of clinical and translational research.

Partner leadership discussed and decided which partner(s) was the ideal point of contact for each service (Figure 3).

Figure 3. TRAIL services launched in January 2018

Research Data Management Services Aggregation: REDCap

REDCap was an established service to researchers prior to the founding of TRAIL and has become the focal point for CDRM as it grows on campus. Launched at UW in 2012, REDCap is a web-based application developed by Vanderbilt University to capture translation data for clinical research and create databases and projects.

Highly-secure, intuitive, and fully HIPAA-compliant, REDCap offers research and data collection tools and database management instruments for researchers. Projects are self-sufficient, and secure databases can be used for normal data entry or for surveys across multiple distinct time points. They are workflow-based and focus on collecting data and exporting it to statistical programs and other data analysis software. REDCap is designed to provide a secure environment so that research teams can collect and store highly sensitive information. Many medical libraries have started using REDCap for assessment and capturing of data for projects. UW clinicians use the platform to create projects and clinical trials such as a basic database format, multiple surveys, and randomized clinical trials.

As of March 20, 2018, the UW REDCap instance has 8,581 active users. A total of 7,750 projects have been created, featuring 38,423 data collection surveys that have been completed more than 700,000 times. When starting a new project, a drop-down list asks users the purpose for creating a REDCap project. At the University of Washington, research is listed as the most popular response (Figure 4).

Figure 4. UW REDCap Project Purposes (December 2016 – March 2018)

A single REDCap administrator, based in ITHS and BIME, was solely responsible for overseeing the entire UW REDCap instance, including monitoring servers, configuring updates, championing REDCap both internally and externally, and answering all user support questions.

With increasing interest in REDCap, the daunting workload proved too high for a single administrator to handle, and ITHS leadership began exploring various avenues to provide additional support while managing a limited budget that prevented the hiring of new staff. As TRAIL’s mandate and purpose began to take shape, HSL’s Associate Dean and Director suggested an existing, highly-trained cohort of liaison librarians to support REDCap. The skillsets meshed perfectly; HSL librarians offer interdisciplinary breadth and are experienced hands at answering information and data queries for solving user and researcher issues. From a library perspective, REDCap support is most akin to answering online reference questions. Furthermore, HSL’s large staff made it feasible to share the support workload and carve it into existing workflows without negatively impacting service performance.

A pilot for REDCap support began at HSL in fall 2016 with two librarians trained to answer researcher questions on basic functions in REDCap. The liaison librarians were quick to learn the basics of REDCap support, so the REDCap Lead created a 10-to-12-hour training module to bring additional librarians into the workflow.

To automate functions of the REDCap team, TRAIL leadership explored the implementation of a ticketing system to allow the librarians to log into one central website rather than answer queries through email without clear records of which questions had been answered and by whom. During the pilot period, HSL tested various ticketing systems before settling on osTicket, an open-source platform modeled on reference ticketing systems used by librarians to answer directions, ready reference, and reference questions. In January 2017, HSL IT staff configured and launched osTicket, consolidating REDCap support for the entire campus into a single online ticketing system.

With the success of the support pilot program, the number of librarians providing REDCap support grew in January 2017 from two to seven. New hires, including librarians, received training with the REDCap Lead. Five librarians are each assigned one day of the week to monitor the ticketing queue and answer questions, with two backup librarians providing additional support and stepping in to cover days others miss due to sickness, scheduling conflicts, or vacation.

Answering questions typically takes 1.5 to 2 hours for the on-call librarian, with almost 11,000 tickets containing questions about CRDM answered between January 2017 and February 2018. Tickets can range in complexity from simple (e.g., approving the creation of a new project), to intermediate (e.g., reviewing proposed project changes to ensure no user’s data is unintentionally corrupted or lost), to complex (e.g., best practices for setting up a multi-arm longitudinal study). If the librarian cannot answer the question quickly (under 20 minutes) or competently, they reassign the ticket to the REDCap Lead, who also provides in-person consultations to support project design.

With its ever-growing popularity, REDCap classes focusing on uses for secure clinical data collection are the most requested TRAIL service, with multiple librarians serving as instructors. All classes are held at HSL; smaller classes, with fewer than 20 attendees, are held in TRAIL using the data wall, with larger classes with more than 21 attendees held in a multipurpose room. In the first year, a series of drop-in beginner and advanced courses were offered to researchers, with three health sciences librarians and the REDCap Lead teaching 10 classes to 85 registrants. The curriculum was changed in January 2018 to offer four weekly classes held every month: a beginner Intro to REDCap, and a rotating set of topics for three expert classes each focusing on an advanced REDCap feature, such as branching logic, data import and export, and database management. Enrollment between the old and new curricula will be impacted in 2018.

Discussion

Medical libraries’ IT staff and librarians need to evolve into service organizations to provide shared technological and CRDM support and an understanding of how to use clinical research data administrative systems. The value of merging departmental workflows equates to the “value of doing business right.” In other words, leveraging existing resources and bringing services and systems together is central to how universities should operate.

This commitment is reflected in TRAIL’s shared staffing, which is comprised of experts in the fields of clinical librarianship, data management, systems and technology, and public health informatics. The sharing of staff has stimulated conversations for more engagement on application design and deployment, including adding library leadership to the UW Medicine ITS Governance Board.

The UW clinical librarians have begun to play an active role in engaging clinicians involved in research to use TRAIL to visualize data, host multidisciplinary working meetings, conduct tests of emerging technologies, and utilize simulation. Through outreach to the community, researchers quickly realized the space could be used to host their virtual reality pre-surgery sessions with clinicians bringing their own equipment to the space. Webinars and conferences such as the Research Data Access & Preservation (RDAP) Summit, NIH Big Data to Knowledge (BD2K), and NNLM Data-Driven Discovery (RD3) represent examples of training that exists to train librarians for roles in data management.

Having ITHS’ Director of Education and NNLM PNR research and data coordinators actively involved in the TRAIL initiative has helped connect researchers to the greater data services librarian community in the Pacific Northwest. Both ITHS and NNLM PNR are regional organizations whose programs build capacity in quality CRDM and data production. NNLM PNR is participating in a national effort by all eight regional offices of the NLM to provide resources and training to data professionals. To that end, the NLM is building a "community of practice" of librarians and other information professionals who have knowledge of research data management (RDM) practices, services, and tools that support the management of research data across the lifecycle. This reflects NLM’s vision to strengthen its role in fostering the future generation of professionals in biomedical informatics, data science, library sciences, and related disciplines through sustained and focused training efforts.

Conclusion

Collaborative, library-led initiatives like TRAIL are vital to supporting universities’ clinical data research efforts. In uniting leading on-campus health sciences organizations, such initiatives build off the strengths of each partner. The sharing of staff and resources aid leadership in identifying and supporting the needs of clinical researchers, re-conceptualizing existing spaces, and adopting innovative, technology-enhanced services. As new services and facilities are introduced, dedicated librarians will grow to develop additional skillsets and carve new roles in CRDM, allowing them to better serve the diverse cross-discipline user base and best position their libraries to thrive amid evolving future campus clinical research priorities.

Acknowledgements

Funding statement

This publication was supported by the National Library of Medicine of the National Institutes of Health under Award Number UG4LM012343 and National Center For Advancing Translational Sciences of the National Institutes of Health under Award Number UL1 TR002319. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Disclosure

The authors report no conflict of interest.

References

• American Library Association (2018). "Trends". Library of the Future. http://www.ala.org/tools/future/trends. Retrieved 13 July 2017. 

• Atlassian (2017). "Atlassian - Software Development and Collaboration Tools". https://www.atlassian.com/. Retrieved 19 July 2017. 

• Bardyn, T. (7 June 2017). "This university’s interactive lab is revolutionizing research". eCampus News. eCampus News Online. https://www.ecampusnews.com/2017/06/07/university-of-washington-lab/. Retrieved 21 July 2017. 

• Bardyn, T.; Mooney, S.; Patridge, E.; Ruhl, D. (May 2017). "Information Resources, Translational Research, Library Convergence: The Newly Emerging Collaboration Lab Model Called the Translational Research and Information Lab (TRAIL)". 2017 GBA/GIP/GIR Joint Spring Meeting Agenda. Association of American Medical Colleges. http://www.cvent.com/events/2017-group-on-business-affairs-group-on-institutional-planning-and-group-on-information-resources-jo/agenda-3b6cf2f32ee441c39c915e1ca2bd8efe.aspx. 

• Best Practical Solutions, LLC (2017). "Best Practical". https://bestpractical.com/. Retrieved 19 July 2017. 

• BMC Software, Inc (2017). "“Remedy 9 - IT Service Management Suite". https://www.bmc.com/it-solutions/remedy-itsm.html. Retrieved 19 July 2017. 

• Brigham, T.J. (2017). "Reality Check: Basics of Augmented, Virtual, and Mixed Reality". Medical Reference Services Quarterly 36 (2): 171–8. doi:10.1080/02763869.2017.1293987. 

• BD2K Training Coordinating Center (2017). "Data Science Series". https://bigdatau.ini.usc.edu/data-science-seminars. Retrieved 19 July 2017. 

• Delawska-Elliott, B.; Belcinksi, D. (1 August 2017). "Hospital Libraries Matter". NLM Musings from the Mezzanine. U.S. National Library of Medicine. https://nlmdirector.nlm.nih.gov/2017/08/01/hospital-libraries-matter/. Retrieved 27 September 2017. 

• de Veer, B. (2018). "REDCap support through TRAIL collaboration". REDCap Consortium. 

• Gonzalo, J.D.; Lucey, C.; Wolpaw, T.; Chang, A. (2017). "Value-Added Clinical Systems Learning Roles for Medical Students That Transform Education and Health: A Guide for Building Partnerships Between Medical Schools and Health Systems". Academic Medicine 92 (5): 602–07. doi:10.1097/ACM.0000000000001346. PMID 27580433. 

• Institute of Translational Health Sciences (26 June 2017). "Limiting Survey Responses". REDCap Tip of the Month. https://www.iths.org/blog/news/redcap-tip/limiting-survey-responses/. Retrieved 21 July 2018. 

• Kraemer, R. (2014). "Advancing Without New Resources". EDUCAUSE Review 49 (4): 44–52. https://er.educause.edu/articles/2014/7/advancing-without-new-resources. 

• Leonard, M.A. (21 June 2017). "Humphreys Honored, Brennan Delivers Keynote at Medical Library Conference". NLM in Focus. U.S. National Library of Medicine. https://infocus.nlm.nih.gov/2017/06/21/humphreys-honored-brennan-delivers-keynote-at-medical-library-conference/. Retrieved 27 September 2017. 

• Main, D.S.; Felzien, M.C.; Magid, D.J. et al. (2012). "A Community Translational Research Pilot Grants Program to Facilitate Community-Academic Partnerships: Lessons From Colorado's Clinical Translational Science Awards". Progress in Community Health Partnerships: Research, Education, and Action 6 (3): 381–7. doi:10.1353/cpr.2012.0036. 

• Marantz, P.R.; Currie, B.; Shamoon, H. (2010). "Transforming Research Environments through Institutional Partnerships: Creation of an academic home for clinical and translational research". Clinical and Translational Science 3 (1): 12–13. doi:10.1111/j.1752-8062.2010.00173.x. 

• Martinez, L.S.; Russell, B.; Rubin, C.L. et al. (2012). "Clinical and Translational Research and Community Engagement: Implications for Researcher Capacity Building". Clinical and Translational Science 5 (4): 329–32. doi:10.1111/j.1752-8062.2012.00433.x. 

• Green, E.; Krumholz, H.; Altman, R. et al. (11 June 2015). "National Library of Medicine (NLM) Working Group - Final Report, June 11 2015" (PDF). National Institutes of Medicine. https://acd.od.nih.gov/documents/reports/Report-NLM-06112015-ACD.pdf. 

• National Institutes of Health (11 May 2016). "NIH appoints Patricia Flatley Brennan, R.N., Ph.D., to lead the National Library of Medicine". https://www.nih.gov/news-events/news-releases/nih-appoints-patricia-flatley-brennan-rn-phd-lead-national-library-medicine. 

• National Network of Libraries of Medicine (2017). "NNLM RD3: Resources for Data-Driven Discovery". National Institutes of Health. https://nnlm.gov/data. Retrieved 27 September 2017. 

• National Network of Libraries of Medicine Evaluation Office (March 2017). "NNSC Goals and Indicators Update" (PDF). https://staff2.nnlm.gov/sites/default/files/shared/Committees/NNSC/NNSC_UPDATE_3_2_17_.pdf. Retrieved 17 July 2017. 

Notes

This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. The original paper did not use any inline citations, so it's not clear what text the references correspond to.