Journal:Improving data quality in clinical research informatics tools

Abstract

Maintaining data quality is a fundamental requirement for any successful and long-term data management project. Providing high-quality, reliable, and statistically sound data is a primary goal for clinical research informatics. In addition, effective data governance and management are essential to ensuring accurate data counts, reports, and validation. As a crucial step of the clinical research process, it is important to establish and maintain organization-wide standards for data quality management to ensure consistency across all systems designed primarily for cohort identification, allowing users to perform an enterprise-wide search on a clinical research data repository to determine the existence of a set of patients meeting certain inclusion or exclusion criteria. Some of the clinical research tools are referred to as de-identified data tools.

Assessing and improving the quality of data used by clinical research informatics tools are both important and difficult tasks. For an increasing number of users who rely on information as one of their most important assets, enforcing high data quality levels represents a strategic investment to preserve the value of the data. In clinical research informatics, better data quality translates into better research results and better patient care. However, achieving high-quality data standards is a major task because of the variety of ways that errors might be introduced in a system and the difficulty of correcting them systematically. Problems with data quality tend to fall into two categories. The first category is related to inconsistency among data resources such as format, syntax, and semantic inconsistencies. The second category is related to poor extract, transform, and load (ETL) and data mapping processes.

In this paper, we describe a real-life case study on assessing and improving the data quality within a healthcare organization. This paper compares between the results obtained from two de-identified data systems—TranSMART Foundation's i2b2 and Epic's SlicerDicer—and discuss the data quality dimensions specific to the clinical research informatics context, and the possible data quality issues between the de-identified systems. This work closes by proposing steps or rules for maintaining data quality among different systems to help data managers, information systems teams, and informaticists at any healthcare organization to monitor and sustain data quality as part of their business intelligence, data governance, and data democratization processes.

Keywords: clinical research data, data quality, research informatics, informatics, management of clinical data

Introduction

Data is the building block in all research, as research results are only as good as the data upon which the conclusions were formed. However, researchers may receive minimal training on how to use the de-identified data systems and methods common to clinical research today for achieving, assessing, or controlling the quality of research data. (Nahm, 2012; Zozus et al., 2019) De-identified data systems are defined as systems or tools that allow users to drag and drop search terms from a hierarchical ontology into a Venn diagram-like interface. Investigators can then perform an initial analysis on the de-identified cohort. However, de-identified data systems have no features to indicate or assist in identifying the quality of data in the system; these systems only provide counts.

Another issue involves the level of knowledge a clinician has about informatics in general and clinical informatics in particular. Without knowledge of informatics concepts, clinicians may not be able to identify quality issues in informatics systems. This requires some background in informatics, the science of how to use data, information, and knowledge to improve human health and the delivery of healthcare services (American Medical Informatics Association, 2022), as well as clinical informatics, the the application of informatics and information technology to deliver healthcare services. For example, clinicians increasingly need to turn to patient portals, electronic medical records (EMRs), telehealth tools, healthcare apps, and a variety of data reporting tools (American Medical Informatics Association, 2022) as part of achieving higher-quality health outcomes.

The case presented in this paper focuses on the quality of data obtained from two de-identified systems: TranSMART Foundation's i2b2 and Epic's SlicerDicer. The purpose of this paper is to discuss the quality of the data (counts) generated from the two systems, understand the potential causes of data quality issues, and propose steps to improve the quality and increase the trust of the generated counts by comparing the accuracy, consistency, validity, and understandability of the outcomes from the two systems. The proposed quality improvement steps are broadly applicable and contribute towards adding generic and essential steps to automate data curation and data governance to tackle various data quality problem. These steps should help data managers, information systems teams, and informaticists at a healthcare organization monitor and sustain data quality as part of their business intelligence, data governance, and data democratization processes.

The remainder of this paper is organized as follows. In the following section, we introduce the importance of data quality to clinical research informatics, followed by details of the case study, study method, and materials used. Afterwards, findings are presented and the proposed steps to ensure data quality are discussed. At the end, conclusions are drawn and future work discussed.

Importance of data quality to clinical research informatics

Data quality refers to the degree data meets the expectations of data consumers and their intended use of the data. (Pipino et al., 2002; Halimeh, 2011; AbuHalimeh and Tudoreanu, 2014). In clinical research informatics, this depends on the parameters of the study conducted. (Nahm, 2012; Zozus et al., 2019)

The significance of data quality lies in how the data is perceived and used by its consumer. Identifying data quality involves two stages: first, highlighting which characteristics (i.e., dimensions) are important (Figure 1) and second, determining how these dimensions affect the population in question. (Halimeh, 2011; AbuHalimeh and Tudoreanu, 2014)

Fig. 1 De-identified data quality dimensions (DDQD)

This paper focuses on a subset of data quality dimensions, which we term "de-identified data quality dimensions" (DDQD). We think these dimensions, described in Table 1, are critical to maintaining data quality in de-identified systems.

Table 1. Definitions of de-identified data quality dimensions (DDQD) Quality dimension Definition Accuracy Refers to the degree to which information accurately reflects an event or object described, i.e., how well does a piece of information reflect reality? Completeness Refers to the extent to which data is not missing and of sufficient amount for the task at hand, i.e., does it fulfill data consumers' expectations and is the needed amount known? Consistency Refers to the extent to which data is applicable and helpful to the task at hand, i.e., does information stored in one place match relevant data stored elsewhere? Timeliness Refers to the extent to which data is sufficiently relevant for the task at hand, i.e., is the most up-to-data available when you need it? Validity Refers to information that doesn't conform to a specific format or doesn't follow business rules, i.e., is the information in a specific format, does it follow business rules, or is it in a usable format? Understandability Refers to the degree the data can be comprehended, i.e., can the user understand the data easily?

The impact of quality data and management is in performance and efficiency gains and the ability to extract new understanding. On the other hand, poor clinical research informatics data quality can cause inefficiencies and other problems throughout an organization. This impact includes the quality of research outcomes, healthcare services, and decision-making.

Quality is not a simple scalar measure but can be defined on multiple dimensions, with each dimension yielding different meanings to different information consumers and processes. (Halimeh, 2011; AbuHalimeh and Tudoreanu, 2014) Each dimension can be measured and assessed differently. Data quality assessment implies providing a value for each dimension that clearly says something about how much of the dimension or quality feature is achieved to enable adequate understanding and management. Data quality and the discipline of informatics are undoubtedly interconnected. Data quality depends on how data are collected, processed, and presented; this is what makes data quality important and sometimes complicated because data collection and processing varies from one study to another. Clinical informatics data can include different data formats and types and can come from different resources.

Notes

This presentation is faithful to the original, with only a few minor changes to presentation, grammar, and punctuation. In some cases important information was missing from the references, and that information was added. Numerous links that were originally posted inline in the text were turned into full citations for this version, adding significantly to the total citation count.