Journal:Usability evaluation of laboratory information systems

Abstract

Background: Numerous studies have revealed widespread clinician frustration with the usability of electronic health records (EHRs) that is counterproductive to adoption of EHR systems to meet the aims of healthcare reform. With poor system usability comes increased risk of negative unintended consequences. Usability issues could lead to user error and workarounds that have the potential to compromise patient safety and negatively impact the quality of care. While there is ample research on EHR usability, there is little information on the usability of laboratory information systems (LIS). Yet, an LIS facilitates the timely provision of a great deal of the information needed by physicians to make patient care decisions. Medical and technical advances in genomics that require processing of an increased volume of complex laboratory data further underscore the importance of developing a user-friendly LIS. This study aims to add to the body of knowledge on LIS usability.

Methods: A survey was distributed among LIS users at hospitals across the United States. The survey consisted of the 10-item System Usability Scale (SUS). In addition, participants were asked to rate the ease of performing 24 common tasks with an LIS. Finally, respondents provided comments on what they liked and disliked about using the LIS to provide diagnostic insight into the perceived usability of an LIS.

Results: The overall mean SUS score of 59.7 for the LIS evaluated is significantly lower than the benchmark of 68 (P < 0.001). All evaluated LIS received mean SUS scores below 68 except for Orchard Harvest (78.7). While the years of experience using the LIS was found to be a statistically significant influence on mean SUS scores, the combined effect of years of experience and LIS used did not account for the statistically significant difference in the mean SUS score between Orchard Harvest and each of the other evaluated LIS.

Conclusions: The results of this study indicate that overall usability of an LIS is poor. Usability lags that of systems evaluated across 446 usability surveys.

Keywords: Health information technology usability, human–computer interaction, laboratory information systems

Introduction

The Health Information Technology for Economic and Clinical Health Act was enacted in 2009 to promote the adoption of health information technology in a nationwide effort to improve the coordination, efficiency, and quality of care. However, numerous studies have revealed widespread clinician frustration with the usability of electronic health records (EHRs) that is counterproductive to the adoption of EHR systems to meet the aims of healthcare reform. According to the survey results released by The American College of Physicians and American EHR Partners, clinician dissatisfaction with the ease-of-use of certified EHRs had increased from 23% to 37% from 2010 to 2012. A 2013 RAND Corporation study sponsored by the American Medical Association found that poor usability of EHRs is a significant source of physician professional dissatisfaction. Findings from a 2015 study conducted by consulting firm Accenture revealed that while 90% of the 601 physician respondents from the United States indicated that an easy-to-use data entry system is important for improving the quality of patient care, 58% of U.S. physician respondents reported that their EHRs are “hard to use.”

With poor system usability comes increased risk of negative unintended consequences. Usability issues could lead to user error and workarounds that have the potential to compromise patient safety and negatively impact quality of care.

While there is ample research on EHR usability, there is little information on the usability of laboratory information systems (LIS). Yet, an LIS facilitates the timely provision of a great deal of the information needed by physicians to make patient care decisions. Medical and technical advances in genomics, requiring processing of an increased volume of complex laboratory data, further underscores the importance of developing a user-friendly LIS. This study aims to add to the body of knowledge on LIS usability.

Methods

Institutional Review Board approval for this study was obtained from The College of St. Scholastica. A survey was built and hosted using the online survey tool Survey Gizmo (http://www.surveygizmo.com). All responses were completely confidential, and no IP addresses or other identifying information were collected.

Survey design

To gain an accurate description of the general traits of study participants, the survey included questions on participants' LIS vendor, length of time using the LIS, job title, bed size of employing hospital, and age. Demographic questions were followed by the administration of the System Usability Scale (SUS), a well-established, highly reliable, and valid instrument in measuring usability of systems. The SUS required respondents to rate their level of agreement/disagreement with 10 statements on a five-point Likert scale ranging from strongly disagree (1) to strongly agree (5).

LIS instructional materials developed by vendors were analyzed to compile a list of 24 basic laboratory tasks. Respondents were asked to rate the ease of performing each task on a five-point Likert scale ranging from very difficult to very easy. Respondents were asked to select N/A for any tasks not applicable to their job role.

The survey concluded with a set of questions asking participants to list three aspects of their LIS that they liked and three aspects they disliked. Participants also were asked to submit any additional comments they wished to express about their LIS' usability.

Participant selection

The survey was distributed through e-mail among pathologists, medical laboratory managers, medical laboratory technicians (MLTs), medical laboratory scientists (MLSs), and other laboratory professionals who currently use an LIS in a hospital setting to collect, process, and analyze specimens in support of patient care. The 2013 HIMSS Analytics Database was used to generate a list of names and e-mail addresses of laboratory managers and pathology department chairs/directors at hospitals across the United States. These individuals were asked to e-mail the survey invitation to their laboratory staff and colleagues. The survey also was distributed among the members of professional associations, including the Association for Pathology Informatics, American Medical Technologists (MTs), American Society of Clinical Laboratory Science, and State Associations of Pathologists. Finally, the survey was posted in medical laboratory-specific social media groups (e.g., Reddit's MedLabProfessionals subreddit, LinkedIn's Pathology Laboratory Manager Forum, the Laboratory Informatics Medical and Professional Scientific Community, the Student Doctor Network Forum, and LIS user groups).

Statistical analysis

Statistical analysis was performed in Microsoft Excel and R Studio. Surveys where the respondent answered “Never used it” to the question “How long have you used your LIS?” were disqualified from the study. In addition, surveys with incomplete SUS responses were disqualified. As the purpose of the study was to evaluate clinician and laboratory staff perceptions of usability, responses from LIS analysts and other information technology (IT) staff were excluded.

System Usability Scale scoring

A response quota of 28 for each LIS was set to have an 80% chance of detecting a 10-point difference between the benchmark mean and each LIS SUS score mean at the 95% confidence level. For each LIS yielding ≥28 responses, a composite SUS score was calculated. Responses were assigned numeric values ranging from 1 for strongly disagree to 5 for strongly agree. For each SUS set of responses, the number one was subtracted from the numeric value of each odd-numbered question. The numeric value of each even-numbered question was subtracted from the number five. The sum of these values was then multiplied by 2.5 to produce the SUS composite score, which had a possible range of 0–100.

The mean SUS score for each LIS was compared to a benchmark mean SUS score of 68, which was derived from the previous research from 446 usability surveys and over 5000 SUS responses. Scores above 68 were considered above average and scores below 68 considered below average. The composite SUS scores were then converted into percentile ranks and graded on a curve to assign letter grades.

To determine if demographic variables influenced SUS scores, one-way ANOVA tests were run to determine if there were statistically significant differences in mean SUS scores between respondents with different roles, ages, employing hospital sizes, and years of experience using their LIS. Finally, the overall mean SUS score for all evaluated LIS was compared to the benchmark score of 68 using one-sample, two-tailed t-tests.

Validation with KLAS Research results

To validate the observed SUS scores, the average SUS scores for each LIS were compared against the KLAS ease-of-use scores. KLAS is a widely-known research firm that collects, analyzes, and reports data reflecting the opinions of healthcare professionals on their experience with the products and services of health-care information technology vendors. The firm issues report cards for each LIS that includes a score for ease-of-use. Using a simple linear regression, the ease-of-use scores from KLAS report cards compiled from data collected from August 2015 to August 2016 were compared with the SUS scores for each LIS evaluated in this study to determine if the SUS scores trended in the same direction as KLAS scores.

Task ratings analysis

To analyze the ratings of the ease of performing the 24 common laboratory tasks, the frequency of each response for each task was calculated to determine the top three tasks rated “very difficult” and top three tasks rated “difficult” for each LIS. The responses also were analyzed to determine what tasks were most frequently rated either “very difficult” or “difficult” across all LIS. Conversely, the data were analyzed to determine the top three tasks rated “very easy” and top three tasks rated “easy” for each LIS and across all LIS.

Free text analysis

Respondents were asked to name three things they liked about using an LIS, three things they disliked about using a LIS, and provide any additional comments regarding their experience with an LIS. QI Macros software (by KnowWare International, Inc., Denver, Colorado) was used in Microsoft Excel to generate word counts and two-word phrase counts to identify themes in the free-text responses. The frequency of the themes was calculated.

Results

Respondent demographics

Respondent roles

There were 327 responses to the survey. Of the total responses, there were 259 qualifying responses analyzed. Sixty-six respondents (25.5%) were laboratory managers, 49 (18.9%) were MLS, 47 (18.1%) were MT, 35 (13.5%) were MLT, and 32 (12.4%) were pathologists. The remaining participants were phlebotomists (7, 2.7%), residents (6, 2.3%), cytotechnologists (5, 1.9%), fellows (3, 1.2%), a health information manager (1, 0.4%), histology technicians (2, 0.8%), processing technicians (2, 0.8%), laboratory supervisors (2, 0.8%), a laboratory assistant (1, 0.4%), and a registered nurse (1, 0.4%).

Hospital size

Just over half of the respondents (54%) reported that they work at a hospital with 400 or more beds. Eight percent are employed at a hospital with 250–399 beds, 14% work in a hospital with 100–249 beds, and 24% work in a hospital with <100 beds.

Age

Ninety-four (36.3%) of respondents were 50–59 years of age, 52 (20.1%) were 30–39 years of age, 46 (17.8%) were 40–49 years of age, 39 (15.1%) were 60–69 years of age, 25 (9.7%) were 18–29 years of age, and one respondent was 80–89 years of age. There were no respondents aged 70–79, and two respondents did not give their age.

Years using the laboratory information system

Most respondents (43.6%) reported using an LIS for five or more years. Fifty-five (21.2%) have been using an LIS for three to four years, 52 (20.1%) have been using an LIS for one to two years, and 39 (15.1%) have been using an LIS for less than one year.

System Usability Scale scores

The overall mean SUS score for all evaluated LIS was 59.7. The one-sample, two-tailed t-test revealed that this mean score is significantly lower than the benchmark score of 68 (P< 0.001).

A one-way ANOVA (F-value =0.806, P = 0.686) revealed no significant difference in mean SUS scores between the role of the participant, and a two-way ANOVA revealed no evidence of a statistically significant interaction effect (P = 0.389) of the independent variables of the LIS used and respondent role on mean SUS scores. Similarly, a one-way ANOVA (F-value = 1.156, P = 0.327) revealed no significant difference in mean SUS scores between respondents from different hospital sizes, and a two-way ANOVA revealed no evidence of a statistically significant interaction effect (P = 0.198) of the independent variables of LIS used and hospital size on mean SUS scores. A one-way ANOVA (F-value =2.021, P =0.063) revealed that there was not a significant difference in mean SUS scores between the age brackets.

The mean SUS score was higher for respondents who have been using their LIS for five or more years as compared to the other duration groups, as seen in Table 1.

Table 1: Mean System Usability Scale scores by years using laboratory information system

Notes

This presentation is faithful to the original, with only a few minor changes to presentation and updates to spelling and grammar. In some cases important information was missing from the references, and that information was added.