Journal:Electronic laboratory notebooks in a public–private partnership

Full article title	Electronic laboratory notebooks in a public–private partnership
Journal	PeerJ Computer Science
Author(s)	Vaas​, Lea A.I.; Witt​, Gesa; Windshügel, Björn; Bosin, Andrea; Serra, Giovanni; Bruengger, Adrian; Winterhalter, Mathias; Gribbon, Philip; Levy-Petelinkar, Cindy J.; Kohler, Manfred
Author affiliation(s)	Fraunhofer Institute for Molecular Biology and Applied Ecology, University of Cagliari, Basilea Pharmaceutica International AG, Jacobs University Bremen, GlaxoSmithKline
Primary contact	Email: manfred dot kohler at ime dot fraunhofer dot de
Editors	Baker, Mary
Year published	2016
Volume and issue	2
Page(s)	e83
DOI	10.7717/peerj-cs.83
ISSN	2167-8359
Distribution license	Creative Commons Attribution 4.0 International
Website	https://peerj.com/articles/cs-83/
Download	https://peerj.com/articles/cs-83.pdf (PDF)

Abstract

This report shares the experience during selection, implementation and maintenance phases of an electronic laboratory notebook (ELN) in a public–private partnership project and comments on users' feedback. In particular, we address which time constraints for roll-out of an ELN exist in granted projects and which benefits and/or restrictions come with out-of-the-box solutions. We discuss several options for the implementation of support functions and potential advantages of open-access solutions. Connected to that, we identified willingness and a vivid culture of data sharing as the major item leading to success or failure of collaborative research activities. The feedback from users turned out to be the only angle for driving technical improvements, but also exhibited high efficiency. Based on these experiences, we describe best practices for future projects on implementation and support of an ELN supporting a diverse, multidisciplinary user group based in academia, NGOs, and/or for-profit corporations located in multiple time zones.

Keywords: public–private partnership, open access, Innovative Medicines Initiative, electronic laboratory notebook, New Drugs for Bad Bugs, IMI, PPP, ND4BB, collaboration, sharing information

Introduction

Laboratory notebooks (LNs) are vital documents of laboratory work in all fields of experimental research. The LN is used to document experimental plans, procedures, results and considerations based on these outcomes. The proper documentation establishes the precedence of results, particularly for inventions of intellectual property (IP). The LN provides the main evidence in the event of disputes relating to scientific publications or patent application. A well-established routine for documentation discourages data falsification by ensuring the integrity of the entries in terms of time, authorship, and content. LNs must be complete, clear, unambiguous and secure. A remarkable example is Alexander Fleming’s documentation, leading to the discovery of penicillin.

The recent development of many novel technologies brought up new platforms in life sciences requiring specialized knowledge. As an example, next-generation sequencing and protein structure determination are generating datasets, which are becoming increasingly prevalent especially in molecular life sciences. The combination and interpretation of these data requires experts from different research areas, leading to large research consortia.

In consortia involving multidisciplinary research, the classical paper-based version of a LN is an impediment to efficient data sharing and information exchange. Most of the data from these large-scale collaborative research efforts will never exist in a hard copy format but will be generated in a digitized version. An analysis of this data can be performed by specialized software and dedicated hardware. The classical application of a LN fails in these environments. It is commonly replaced by digital reporting procedures, which can be standardized. Besides the advantages for daily operational activities, an electronic laboratory notebook (ELN) yields long-term benefits regarding data maintenance. These include, but are not limited to, items listed in Table 1. The order of mentioned points is not expressing any ranking. Besides general tasks, some specific tasks have to be facilitated, especially in the field of drug discovery. One such specific task is searching for chemical structures and substructures in a virtual library of chemical structures and compounds (see Table 1, last item in column “Potentially”). Enabling such a function in an ELN hosting reports about wet-lab work dealing with known drugs and/or compounds to be evaluated would allow dedicated information retrieval for the chemical compounds or (sub-) structures of interest.

Table 1. Long-term benefits of an electronic laboratory notebook (ELN) compared to a paper based LN Definitely Potentially • Create (standard) protocols for experiments • Create and share templates for experimental documentation • Share results within working groups • Amend/extend individual protocols • Full complement of data/information from one experiment is stored in one place (in an ideal world) • Storage of data from all experiments in a dedicated location • Search functionality (keywords, full text) • Protect intellectual property (IP) by timely updating of experimental data with date/time stamps • Exchange protocols/standard operating procedures (SOPs) • Remote access of results/data from other working groups • Ensure transparency within projects • Discuss results online • Control of overall activity by timely planning of new experiments based on former results • Search for chemical (sub)structures within all chemical drawings in experiments

Interestingly, although essential for the success of research activities in collaborative settings, the above mentioned advantages are rarely realized by users during daily documentation activities and institutional awareness in academic environment is often lacking.

Since funding agencies and stakeholders are becoming aware of the importance of transparency and reproducibility in both experimental and computational research, the use of digitalized documentation, reproducible analyses and archiving will be a common requirement for funding applications on national and international levels.

A typical example for a large private-public partnership is the Innovative Medicines Initiative (IMI) New Drugs for Bad Bugs (ND4BB) program (see Fig. 1 for details). The program’s objective is to combat antibiotic resistance in Europe by tackling the scientific, regulatory, and business challenges that are hampering the development of new antibiotics.

Figure 1. Structural outline of the New Drugs for Bad Bugs (ND4BB) framework

The TRANSLOCATION consortium focus on (i) improving the understanding of the overall permeability of Gram-negative bacteria, and (ii) enhancing the efficiency of antibiotic research and development through knowledge sharing, data sharing and integrated analysis. To meet such complex needs, the TRANSLOCATION consortium was established as a multinational and multisite public–private partnership (PPP) with 15 academic partners, five pharmaceutical companies and seven small- and medium-sized enterprises (SMEs).

In this article we describe the process of selecting and implementing an ELN in the context of the multisite PPP project TRANSLOCATION, comprising about 90 bench scientists in total. Furthermore we present the results from a survey evaluating the users’ experiences and the benefit for the project two years post-implementation. Based on our experiences, the specific needs in a PPP setting are summarized and lessons learned will be reviewed. As a result, we propose recommendations to assist future users avoiding pitfalls when selecting and implementing ELN software.

Methods

Selection and implementation of an ELN solution

The IMI project call requested a high level of transparency enabling the sharing of data to serve as an example for future projects. The selected consortium TRANSLOCATION had a special demand for an ELN due to its structure — various labs and partners spread widely across Europe needed to report into one common repository — and due to the final goal, data was required to be stored and integrated into one central information hub, the ND4BB Information Centre. Fortunately, no legacy data had to be migrated into the ELN.

The standard process for the introduction of new software follows a highly structured multi-phase procedure, as outlined in Fig. 2.

Figure 2. Schematic outline of stepwise procedure for the implementation of a new system

For the first step, we had to manage a large and highly heterogeneous user group (Fig. 3) that would be using the ELN, scheduled for roll-out within six months after project launch. All personnel of the academic partners were requested to enter data into the same ELN, potentially leading to unmet individual user requirements, especially for novices and inexperienced users.

Figure 3. Technical and organizational challenge: schematic overview of paths for sharing research activity results within a public–private partnership on antimicrobial research

As a compromise for step 1 (Fig. 2), we assembled a collection of user requirement specifications (URS) based on the experiences of one laboratory that had already implemented an ELN. We further selected a small group of super users based on their expertise in documentation processes, representing different wet laboratories and in silico environments. The resulting URS was reviewed by IT and business experts from academic as well as private organisations of the consortium. The final version of the URS is available as Supplemental File 1.

In parallel, based on literature (Rubacha, Rattan & Hossel, 2011) and internet searches, presentations of widely used ELNs were evaluated to gain insight into state-of-the-art ELNs. This revealed a wide variety of functional and graphical user interface (GUI) implementations differing in complexity and costs. The continuum between simple out-of-the-box solutions and highly sophisticated and configurable ELNs with interfaces to state-of-the-art analytical tools were covered by the presentations. Notably, the requirements specified by super users also ranged from “easy to use” to “highly individually configurable.” Based on this information, it was clear that the ELN selected for this consortium would never ideally fit all user expectations. Furthermore, the exact number of users and configuration of user groups were unknown at the onset of the project. The most frequently or highest prioritized items of the collected user requirements are listed in Table 2. We divided the gathered requirements into ‘core’ meaning essential and ‘non-core’ meaning ‘nice to have, but not indispensable.’ Further, we list here only the items, which were mentioned by more than two super users from different groups. The full list of URS is available in Supplemental File 1.

Table 2. Overview of user requirements organized as ‘core user requirements’ for essential items, and ‘non-core user requirements’ representing desirable features Core user requirements Non-core user requirements • System set-up and implementation should be fast and simple • Access from different platforms should be possible: Windows, Linux, Mac OS • Low training requirements (for high level of acceptance • Hosted system with state-of-the-art security settings • Simple user management (only limited support by project members possible) • Suitable for both chemical (including e.g., drawings of molecules) and biological (including e.g., capture fluorescent images) experiments • Low costs, especially for long-term usage in the academic area • Conform with Good Laboratory Practise • User management with dedicated access permissions (expectation: all users working on the same project, but in different work packages) • Workflow management • Order management • Chemical structure handling • Dedicated tree structure for storing experiments • Legally-binding procedures (signatures) • Modular expandability • Appropriate integrated analytical features • Social networking and collaborative (chat) features • Storage for large sets of “raw” data for re-analysis

Based on the user URS, a tender process (Step 2, Fig. 2) was initiated in which vendors were invited to respond via a request for proposal (RFP) process. The requirement of the proposed ELN to support both chemical and biological research combined with the need to access the ELN by different operating systems (Windows, Linux, Mac OS) (see Fig. 3) reduced the number of appropriate vendors. Their response provided their offerings aligned to the proposed specifications.

Key highlights and drawbacks of the proposed solutions were collected as well as approximations for the number of required licenses and maintenance costs. The cost estimates for licenses were not comparable because some systems require individual licenses whereas others used bulk licenses. At the time of selection, the exact number of users was not available.

Interestingly, the number of user specifications available out-of-the-box differed by less than 10 percent between systems with the lowest (67) and highest (73) number of proper features. Thus, highlights and drawbacks became a more prominent issue in the selection process.

For the third step, (Fig. 2), the two vendors meeting all the core user requirements and the highest number of non-core user requirements were selected to provide a more detailed online demonstration of their ELN solution to a representative group of users from different project partners. In addition, a quote for 50 academic and 10 commercial licenses was requested. The direct comparison did not result in a clear winner; both systems included features that were instantly required, but each lacked some of the essential functionalities.

For the final decision, only features which were different in both systems and supported the PPP were ranked between 1 = low (e.g., academic user licenses are cheap) and 5 = high (e.g., cloud hosting) as important for the project. The decision between the two tested systems was then based on the higher number of positively ranked features, which revealed most important after presentation and internal discussions of super users. The main drivers for the final decision are listed in Table 3. In total, the chosen system got 36 positive votes on listed features meeting all high ranked demands listed in Table 3, while the runner-up had 24 positive votes on features. However, if the system had to be set up in the envisaged consortium it turned out to be too expensive and complex in maintenance.

Table 3. List of drivers for final decision about which ELN solution to be set up and run in the project consortium Main drivers for the final decision • Many end users were unfamiliar with the use of ELNs, therefore the selected solution should be intuitive • Easy to install and maintain • Minimal user training required • Basic functionalities available out-of-the-box (import of text, spreadsheet, PDF, images and drawing of chemical structures), with as few configurations as possible • ELN does not apply highly sophisticated checking procedures, which would require a high level of configuration, restricting users to apply their preferred data format (users should take responsibility for the correct data and the correct format of the data stored in the ELN instead) • Web interface available to support all operating systems to avoid deploying and managing multiple site instances • Vendor track record—experience of the vendor with a hosted solution as an international provider • Sustainability: ∘ affordable for academic partners also after the five year funding duration of the project (based on the maturity of the vendor, number of installations/users, the state-of-the-art user interface and finally also the costs) ∘ easy to maintain (minimal administrative tasks, mainly user management) ∘ support of different, independent user groups ∘ configurable private and public templates for experiments • Proposed installation time-frame • Per user per year costs for academics and commercial users

The complete process, from the initial collection of URS data until the final selection of the preferred solution, took less than five months.

Following selection, the product was tested before specific training was offered to the user community. Parallel support frameworks were rolled out at this time, including a help desk as a single point of contact (SPoC) for end users.

The fourth step (Fig. 2) was the implementation of the selected platform. This deployment was simple and straightforward because it was available as software as a service (SaaS) hosted as a cloud solution. Less than one week after signing the contract, the administrative account for the software was created and the online training of key administrators commenced. The duration of training was typically less than two hours, including tasks such as user and project administration.

To accomplish step 5 (Fig. 2), internal training material was produced based on the experiences made during the initial introduction of the ELN to the administrative group. This guaranteed that all users would receive applicable training. During this initial learning period, the system was also tested for the requested user functionalities. Workarounds were defined for missing features detected during the testing period and integrated into the training material.

One central feature of an ELN in its final stage is the standardization of minimally required information. These standardizations include, but are not limited to:

• Define required metadata fields per experiment (e.g., name of the author, date of creation, experiment type, keywords, aim of experiment, executive summary, introduction).
• Define agreed naming conventions for projects.
• Define agreed naming conventions for titles of experiments.
• Prepare (super user agreed) list of values for selection lists.
• Define type of data which should be placed in the ELN (e.g., raw data, curated data).

We did not define specific data formats since we could not predict all the different types of data sets that would be utilized during the lifetime of the ELN. Instead, we gave some best practise advice on arranging data (Supplemental Files 3 and 4), facilitating its reuse by other researchers. The initially predefined templates, however, were only rarely adopted, with some groups using nearly the same structure to document their experiments. More support especially for creating templates may help users to document their results more easily.

For the final phase, the go live, high user acceptance was the major objective. A detailed plan was created to support users during their daily work with the ELN. This comprised the setup of a support team (the project specific ELN-Helpdesk) as a SPoC and detailed project-specific online training, including documentation for self-training. As part of the governance process, we created working instructions describing all necessary administrative processes provided by the ELN-Helpdesk.

Parallel to the implementation of the ELN-Helpdesk, a quarterly electronic newsletter was rolled out. This was used advantageously to remind potential users that the ELN is an obligatory central repository for the project. The newsletter also provided a forum for users to access information and news, messaging to remind them the value of this collaborative project.

Documents containing training slide sets, frequently asked questions (FAQs) and best practice spreadsheet templates (Supplemental File 4) have been made available directly within the ELN to give users rapid access to these documents while working with the ELN. In addition, the newsletter informed all users about project-specific updates or news about the ELN.

Results

Operation of the ELN

Software operation can be generally split into technical component/cost issues and end-user experiences. The technical component considers stability, performance and maintenance, whereas the end-user experience is based on the capability and usability of the software.

Technical solution

The selected ELN, hosted as a SaaS solution with a cloud-based service centre, provided a stable environment with acceptable performance, e.g., login <15 sec, opening an experiment with five pages <20 sec (for further technical details, please see Supplemental File 1). During the evaluation period of two years, two major issues emerged. The first involved denial of access to the ELN for more than three hours, due to an external server problem, which was quickly and professionally solved after contacting the technical support, and the other was related to the introduction of a new user interface (see below).

The administration was simple and straightforward, comprising mainly minor configurations at the project start and user management during the runtime. One issue was the gap in communication regarding the number of active users, causing a steady increase in number of licenses.

A particular disadvantage using the selected SaaS solution concerned system upgrades. There was little notice of upcoming changes, and user warnings were hidden in a weekly mailing. To keep users updated, weekly or biweekly mails about the ELN were sent to the user community by the vendor. Although these messages were read by users initially, interest diminished over time. Consequently, users were confused when they accessed the system after an upgrade and the functionality or appearance of the ELN had changed. On the other side, system upgrades were performed over weekends to minimize system downtime.

The costs per user were reasonable, especially for the academic partners for whom the long-term availability of the system, even after project completion, could be assured. This seemed to be an effect of the competitive market that caused a substantial drop in price during the last years.

User experience

In total, more than 100 users were registered during the first two years runtime, whereas the maximum number of parallel user accounts was 87; i.e., 13 users left the project for different reasons. The number of 87 users is composed of admins (n = 3), external reviewers (n = 4), project owners (n = 26) which are reviewing and countersigning as principal investigators (PIs), and normal users (n = 41). Depending on the period, the number of newly entered experiments per month ranged between 20 and 200 (Fig. 4, blue bars). The size of the uploaded or entered data was heterogeneous and comprised experiments with less than 1 MB, i.e., data from small experimental assays, but also contained data objects much larger than 100 MB, e.g., raw data from mass spectroscopy. Interestingly, users structured similar experiments in different ways. Some users created single experiments for each test set, while other combined data from different test sets into one experiment.

Figure 4. Overview of usage of the ELN and workload for the helpdesk over time: Y-axis on the left shows number of experiments created per month (blue bars) overlaid by number of help desk tickets created per month (orange line) scaled on the y-axis on the right.

In an initial analysis, we evaluated user experience by the number of help desk tickets created during the live time of the system (Fig. 4, orange line).

During the initial phase (2013.06–2013.10) most of the tickets were associated with user access to the ELN. However, after six months (2013.12–2014.01), many requests were related to functionality, especially those from infrequent ELN users. In February 2015, the vendor released an updated graphical user interface (GUI), resulting in a higher number of tickets referring to modified or missing functions and the slow response of the system. The higher number of tickets in August/September 2015 were related to a call for refreshing assigned user accounts. However, overall the number of tickets was within the expected range (<10 per month).

There was a clear decline in frequent usage during the project runtime. The ELN was officially introduced in June 2013, and the number of experiments increased during the following month as expected (Fig. 4, blue bars). The small decrease in November 2013 was anticipated as a reaction to a new release, which caused some issues on specific operating system/browser combinations and language settings. Support for Windows XP also ended at this time. Some of the issues were resolved with the new release (December 2013). The increase in new experiments in October 2014 and subsequent decline in November 2014 is correlated to a reminder sent out to the members of the project to record all activities into the ELN. The same is true for September/October 2015. The last quarter of each year illustrates year-end activities, including the conclusion of projects and the completion of corresponding paperwork, as reflected in the chart (Fig. 4, orange line).

Overall, the regular documentation of experiments in the ELN appeared to be unappealing to researchers. This infrequent usage prompted us to carry out a survey of user acceptance in May/June 2015 (detailed description of analysis methods including KNIME workflow and raw data are provided in Supplemental File 2). The primary aim was to evaluate user experiences compared to expectations in more detail. In addition, the administrative team wanted to get feedback to determine what could be done to the existing support structure to increase usage or simplify the routine documentation of laboratory work. Overall, 77 users (see Table 4) were invited to participate in the survey. Two users left the project during the runtime of the survey. We received feedback from 60 (=80%) out of the remaining 75 users. Two questionnaires were rejected due to less than 20% answered questions. The number of evaluated questionnaires is 58 (see Supplemental File 5).

Table 4. Number of users of the ELN and participants in the survey Description n Remarks Maximum number of parallel users of the ELN 87 Number of parallel users at the time point of the survey 84 Administrators 3 Not invited to participate in survey External reviewers 4 Not invited to participate in survey Project owners = Principal investigators 26 Invited to participate in survey Normal user 51 Invited to participate in survey Deactivated users during the runtime of the questionnaire 2 Questionnaires returned 60 Questionnaires rejected due to insufficient (less than 20%) number of answered questions 2 Evaluated questionnaires 58

There are also some limitations of the survey which should be discussed. The number of invited active ELN users was low (n = 77), thus we refused to collect detailed demographic data in order to ensure full anonymity of the participants; by doing so we expected a higher participation rate and especially more detailed answers to the free text questions. In addition, some interesting analysis could not be answered by the questionnaire due to the low number of returned forms (n = 58). For example only six users had some experience with ELNs. Three out of the six users found the ELN is changing the way of personal documentation positively, while the others didn’t answered the question or gave a neutral answer. Thus, we did report these results as not representative.

It should also be mentioned that this survey reflects the situation of this specific PPP project. The results cannot be easily transferred to other projects. It would be of interest if the same survey would give different or the same results (i) either in other projects (ii) during the time course of this project.

A summary of the most important results of the survey is presented in Table 5.

Table 5. Overview about most important results from survey, which was sent out to 77 users from 18 academic and SME organisations: A set of 58 comprehensively answered questionnaires was considered for evaluation. Major results from survey • Most user never used an ELN before (51 out of 58 users replying to the survey stated “I never used an ELN before this project”; no info from remaining 19 invited users) • Most users (76%) are using a paper notebook in addition to the ELN • Many users (n = 23) would not recommend using an ELN again • No of Operating systems: Linux = 7, Mac OS = 14, Windows = 37 • ELN typically used ∘ Rarely or sometimes with <1 hour per session (53%) ∘ Frequently <1 hour per session (16%) ∘ Sometimes or frequently 1–2 hours per session (9%) • Frequent users (n = 13) realized an increase in quality of documentation (46%) and 38% would recommend this software to colleagues while even three out the 13 frequent users would not use an ELN again if they could decide • Rarely users (n = 19) are skeptical about ELN functionality (42%) • While 52% of the Mac and Linux users are satisfied about the performance 35% of the Windows users are unhappy compared to 27% which are satisfied about the performance of the system • Helpdesk support in general is O.K. (36%), but some users (10%) seem to be not satisfied, especially with training (47%) • Most users demand higher speed (n = 15) and/or better user interface (n = 14)

Despite the perceived advantages of an ELN compared to traditional paper-based LNs as described above, users encountered several drawbacks during their usage of this ELN. Users criticized the provision of templates and cloned experiments, which were considered to impede the accurate documentation of procedural deviations. The standardized documentation of experimental procedures made it difficult to detect deviations or variations because they are not highlighted. Careful review of the complete documentation was required in order to check for missing or falsified information.

For many users (44 out of 58 = 76%), a paper-based LN was still the primary documentation system. They established a habit of copying the documentation into the ELN only after the completion of successful experiments rather than using the ELN online in real time. This extra work is also a major source of dissatisfaction and could create difficult situations in case of discrepancies between the paper and the electronic version when intellectual property needs to be demonstrated. In these cases, failed experiments were not documented in the ELN, although comprehensive documentation is available offline. For failed experiments, the effort to document the information in a digital form was not considered worthwhile by the users. In other cases, usage of the ELN for documentation of experimental work was hindered due to performance issues by technically outdated lab equipment.

The survey helped to understand what factors where contributing to the low usage of the ELN. Additional results of the survey grouped by operating system or usage are shown in Tables 6 and 7. For a more detailed analysis, see Supplemental File 6.

Table 6. Summary of acceptance of ELN and user experience given the mainly used OS plus an overview of the main result from survey for the different user-groups a Those columns do not add up to 58 because some participants stated that they are both wet-lab and in-silico researchers. Frequently used platform to access the ELN No. of users Percentage of all users No. of wet-lab researchersa No. of in-silico researchersa Mac 14 24% 5 8 Windows 37 64% 30 12 Linux 7 12% 1 5 Summarized result based on the answers by OS: • Windows users mainly conduct wet lab work while Linux and Mac users perform in silico work. • Windows users find the software too slow and too labor-intensive. • Windows users know the functionality of the ELN better, as they are using the system more frequently. • Mac OS and Linux users are more comfortable with the speed of the software, but they would not use or recommend it again. This may be related to the specific in silico work which might not be supported by the ELN sufficiently.

Table 7. Overview of self-assessed frequency and usage-time of the ELN given the used OS plus summarized results of the survey for the different user-groups a This column does not add up to 100% due to rounding errors. Self-assessed frequency of the ELN Self-assessed length of usage-time of the ELN Frequently used platform to access the ELN No. of users Percentagea Percentage/OS Rarely <1 hour Linux 2 3% 29% Sometimes <1 hour Linux 5 9% 71% Rarely <1 hour Mac 3 5% 21% Rarely 1–2 hours Mac 4 7% 29% Sometimes <1 hour Mac 5 9% 36% Frequently <1 hour Mac 2 3% 14% Rarely <1 hour Windows 9 16% 24% Rarely 1–2 hours Windows 1 2% 3% Sometimes <1 hour Windows 7 12% 19% Sometimes 1–2 hours Windows 8 14% 22% Sometimes >2 hours Windows 1 2% 3% Frequently <1 hour Windows 7 12% 19% Frequently 1–2 hours Windows 4 7% 11% Summarized result based on frequency of usage: • In silico users enter into the ELN less frequently, which is not unexpected as computational experiments generally run for a longer period of time than wet lab experiments. • More frequent users operate the ELN online during their lab work. • Frequent users would like to have higher performance (this might be related to Windows). • Better quality documentation was associated with more frequent use. • Frequent users are not disrupted by documenting their work in the ELN, they like the software and would use an ELN in future. • Frequent users of an ELN obtain a positive effect on the way documentation is prepared. • More frequent users like the software and feel comfortable about using the software while Infrequent users find the ELN complex and are frustrated about functionality. • Infrequent users are disappointed about the quality of search results.

Notes

This presentation is faithful to the original, with only a few minor changes to presentation, including regionalizing spelling. In some cases important information was missing from the references, and that information was added. The original lists references in alphabetical order; this version lists them in order of appearance due to the nature of the wiki. Supplemental file names have been clarified in the text.