Journal:Sigma metrics as a valuable tool for effective analytical performance and quality control planning in the clinical laboratory: A retrospective study

Full article title	Sigma metrics as a valuable tool for effective analytical performance and quality control planning in the clinical laboratory: A retrospective study
Journal	National Journal of Laboratory Medicine
Author(s)	Karaattuthazhathu, Anupapama R.; Sathi, P.P.; Nair, Lathi; Ramees, P.M.; Manayani, Reenu; Benny, Elizabeth; Benny, Mary
Author affiliation(s)	KMCT Medical College
Primary contact	anupamarojith at gmail dot com
Year published	2023
Volume and issue	12(2)
Page(s)	PO14 - PO18
DOI	10.7860/NJLM/2023/60440.2714
ISSN	2277-8551
Distribution license	Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International
Website	https://njlm.net/article_FULLTEXT.aspx
Download	https://www.jcdr.net//articles/PDF/ (PDF)

Abstract

Introduction: For the release of precise and accurate reports of routine tests, its necessary to follow a proper quality management system (QMS) in the clinical laboratory. As one of the most popular QMS tools for process improvement, Six Sigma techniques and tools have been accepted widely in the laboratory testing process. Six Sigma gives an objective assessment of analytical methods and instrumentation, measuring the outcome of a process on a scale of 0 to 6. Poor outcomes are measured in terms of defects per million opportunities (DPMO).

Aim: To do the performance assessment of each clinical laboratory analyte by Six Sigma analysis and to plan and chart out a better, customized quality control (QC) plan for each analyte, according to its own sigma value.

Materials and methods: This was a retrospective observational study, conducted from January 2022 to June 2022 in the Department of Central Laboratory, KMCT Medical College, Kozhikode, Kerala, India. The precision and accuracy of 26 parameters in both hematology and biochemistry were assessed via internal quality control (IQC) and external quality assurance (EQAS) programs and analysis, and their performance was assessed by sigma analysis.

Results: Clinical chemistry parameters showed an average percentage of coefficient of variation (CV%) of 2.65% and 2.30% for all the parameters in L2 (normal level) and L3 (abnormal levels), respectively. In hematology, the average CV% came out as 1.30% (high level), 1.82% (low level), and 1.35% (normal level). These values indicate excellent precision for all parameters in both clinical chemistry and hematology, with CV% below 3.00%. It was observed in the month of May, due to reconstitution errors, that bias percent showed a setback in a few chemistry parameters, due to which the sigma was lowered. Parameters with < 3 sigma metrics (poor performance) occupy 37%, 3-6 sigma metrics (good performance) occupy 29%, and > 6 sigma metrics (world-class performance) occupy 34% of all the 26 parameters of clinical chemistry and hematology. Mean standard deviation (SD) for biochemistry parameters were calculated using the daily IQC data.

Conclusion: With the present study, sigma metric analysis provides a benchmark for the laboratory to design a protocol for IQC, address poor assay performance, and assess the efficiency of the existing laboratory processes. It is on the basis of strict QC measures and sigma analysis the Department of Central Laboratory, KMCT Medical College was able to achieve world-class performance in many analytes of clinical chemistry and hematology disciplines. However, a few analytes like alkaline phosphatase, alanine transaminase, aspartate transaminase, and total protein needed more stringent EQAS monitoring and modified QC measures.

Keywords : Six sigma, Total analytical error, Westgards rules

Introduction

In a healthcare system, laboratory diagnostics plays a very important role in accurate diagnosis and prompt disease management. The total testing process in a clinical laboratory includes pre-analytical, analytical, and post-analytical phases. Quality control (QC) measures can be applied successfully in any of the three phases. In the analytical phase, quality improvement can be done through internal quality control (IQC) and external quality assurance (EQAS) measures to ensure precision and accuracy of result reporting. IQC involves running a control sample with an identical matrix to the patient's specimen, with the sample having an established concentration range. The range should be available in high, low, or normal levels of the analyte, covering the medical decision points. EQAS or peer group programs involve periodically reporting proficiency testing samples, which are supplied by an external agency at a predefined time interval. The values obtained are compared with those obtained in other laboratories that are participating in the same program and are interpreted as SD Index or Z-score. While IQC determines the precision (expressed as a coefficient of variation or CV) of the testing process, EQAS measures its accuracy (expressed as bias). The use of sigma metrics is about identifying non-conformities or errors using a technique to quantify and then minimize those defects. It is the benchmarking scale where all the process defects in all three phases of a total testing process are measured and judged.

The Six Sigma method of sigma metrics was put to use decades ago, in Motorola by Sir Bill Smith, the father of Six Sigma in 1986. The first paper to express its application in clinical laboratory processes was published in 2000.

Six Sigma measures the outcomes of key process aspects, on a scale of 0 to 6. The poor outcomes are measured in terms of defects per million opportunities (DPMO). The number of defects or errors of the laboratory in any area of the total testing process can be counted or estimated and converted to the DPMO ratio. The level of sigma metrics and corresponding DPMO is shown in (Table/Fig 1).

Table/Figure 1. The level of sigma metrics and corresponding DPMO.

This work has the following objectives:

• To determine the precision of 36 parameters included in the hematology and clinical chemistry sections, quantitated by percentage coefficient variation (CV%); • To assess the accuracy of the analytes, quantitated by the bias percentage of each analyte after EQAS analysis; • To determine the performance assessment of each analyte by Six Sigma analysis; and • To plan and chart out a better, customized QC plan for each analyte according to its own sigma value.

Using the sigma metrics protocol, the authors were able to effectively evaluate the analytical process control procedures in the clinical laboratory and could verify any shortcomings in the procedures. Thus, accuracy, precision, and error detection rate of the analytical phase of the total testing process may be detected and rectified.

Materials and methods

This was a retrospective observational study, conducted for a period of six months (January 2022 to June 2022) in the Department of Central Laboratory, KMCT Medical College, Kozhikode, Kerala, India. The biochemical analytes included in the study are albumin, bilirubin (total and direct), total protein, calcium, glucose, urea, creatinine, uric acid, HDL, triglycerides, cholesterol, phosphorus, aspartate transaminase, and alanine transaminase. Hematology parameters included hemoglobin (Hb), red blood cell (RBC) count, hematocrit, mean corpuscular hemoglobin concentration (MCHC), mean corpuscular volume (MCV), mean corpuscular hemoglobin (MCH), and coefficient of variation of red cell distribution width (RDW-cv), standard deviation of red cell distribution width (RDW-SD), total white blood cell (WBC) count, platelet count, and mean platelet volume (MPV).

Study procedure

The biochemistry IQCs were performed daily using L2 (normal) and L3 (pathological) levels, thrice a day. Hematological QC was performed thrice a day using three levels (High, Normal, Low).

The chemistry parameters’ QC samples were run in Randox's RX Imola and sigma were calculated on a six monthly cumulative basis. The hematology parameters’ QC samples were run in an Horiba ABX Pentra XL 80 analyzer and the CV% of each was calculated on a monthly basis, with sigma calculated on a cumulative six-month basis. In the laboratory, the IQC data of all the disciplines were interpreted daily by Levy-Jenning’s charts and Westgard’s rules. Daily IQC outliers were detected and appropriate prompt corrective and preventive actions were taken. The patients’ specimens were analyzed only when the IQC results were within control limits. The authors adopted the set of Westgard’s multi-rules in the laboratory as 1-2s as warning rules, and 2-2s, 1-3s, and R4s as rejection rules.

The QC practices, such as the control of material storage, reconstitution, and analysis, were done as per the manufacturer’s instructions. EQAS data was obtained by running monthly control samples from the Randox International Quality Assessment Scheme (RIQAS).

Statistical analysis

Statistical analysis was performed using Microsoft Office Excel 2010. Mean standard deviation (SD) for biochemistry parameters was calculated using the daily IQC data. The CV% was calculated using the formula:

$CV\%={\frac {\text{SD X 100}}{\text{Laboratory mean}}}$.

Bias percentage and SD index were calculated using RIQAS data for each analyte. Bias percentage was calculated using the formula:

$Bias\%={\frac {\text{Peer group mean - Laboratory mean}}{\text{Peer group mean}}}$,

where the peer group mean is the mean of all QC values of laboratories enrolled in the RIQAS program. The allowable total analytical error (TEa%) for each analyte was referenced from the proposed Clinical Laboratory Improvement Amendments (CLIA) 2024 guidelines and the consolidated analytical performance requirements. Sigma metrics for analytes were calculated using the formula:

${\text{Sigma value}}={\frac {TEa\%-Bias\%}{CV\%}}$.

Results

Monthly CV% of level 2 (L2) controls for the chemistry analytes, from January to June 2022, and six-month cumulative CV% for each are tabulated and summarized in (Table/Fig 2).

Table/Figure 2. Monthly CV% of 16 chemistry analytes and six-month cumulative CV% of each analyte, for level 2 QC. CV: Coefficient of variation; ALB: Albumin, TP: Total protein; ALP: Alkaline phosphatase; AST: Aspartate transaminase; ALT: Alanine transaminase; TB: Total bilirubin; DB: Direct bilirubin; GLU: Glucose, CHOL: Cholesterol; TG: Triglycerides; UA: Uric acid; CREA: creatinine; CA: Calcium; HDL: High-density lipoprotein; PHOS: Phosphorus.

Monthly CV% of Level 3 (L3) controls for the chemistry analytes, from January to June 2022, and six-month cumulative CV% for each are tabulated and summarized in (Table/Fig 3).

Table/Figure 3. Monthly CV% of 16 chemistry analytes and six-month cumulative CV% of each analyte, for level 3 QC. CV: Coefficient of variation; ALB: Albumin, TP: Total protein; ALP: Alkaline phosphatase; AST: Aspartate transaminase; ALT: Alanine transaminase; TB: Total bilirubin; DB: Direct bilirubin; GLU: Glucose, CHOL: Cholesterol; TG: Triglycerides; UA: Uric acid; CREA: creatinine; CA: Calcium; HDL: High-density lipoprotein; PHOS: Phosphorus.

Monthly CV%, from January 2022 to June 2022, of hematological parameters for all three levels (High, Low, and Normal) were calculated, and six-month cumulative CV% were calculated for each parameter, summarized in (Table/Fig 4), (Table/Fig 5), and (Table/Fig 6), respectively.

Table/Figure 4–6. Monthly CV% of 10 hematology parameters and six-month cumulative CV% of each, for level high qc (Table/Fig. 4), low (Table/Fig. 5), and normal (Table/Fig. 6).

Monthly bias percentage, from January 2022 to June 2022, and six-month cumulative bias percentage of chemistry and hematological parameters were calculated, summarized in (Table/Fig 7) and (Table/Fig 8), respectively.

Table/Fig. 7–8 : Monthly bias percentage of 16 chemistry analytes and six-month cumulative bias percentage of each (Table/Fig. 7), and monthly bias percentage of 10 hematology analytes and six-month cumulative bias percentage of each (Table/Fig. 8).

The cumulative CV% and cumulative bias percentage of each analyte in both chemistry and hematology are clubbed together, and the allowable TEa% of all the chemistry and hematological parameters according to the proposed CLIA 2024 guidelines are summarized in (Table/Fig 9) and (Table/Fig 10), respectively.

Table/Fig. 9–10 : Percentage coefficient of variance (CV%), bias, and the allowable total analytical error (TEa) of chemistry analytes (Table/Fig. 9), and percentage coefficient of variance (CV%), bias, and the allowable total analytical error (TEa) of hematology analytes (Table/Fig. 10).

Average CV% was calculated, and with the above-mentioned formula, sigma metrics was calculated for each clinical chemistry and hematology parameter, as seen in (Table/Fig 11) and (Table/Fig 12), respectively. The analytes were broadly categorized into sigma < 3 (unacceptable), sigma 3-6 (good), and sigma > 6 (excellent), using a pie diagram (Table/Fig 13). Upon analysis, parameters with < 3 sigma metrics (poor performance) occupied 37%, 3-6 sigma metrics (good performance) occupied 29%, and > 6 sigma metrics (world-class performance) occupied 34% of all the 26 parameters of clinical chemistry and hematology.

Table/Fig. 11–13 : Sigma metrics of clinical chemistry and hematology parameters (Table/Fig. 11 and 12, respectively), and the distribution of six-month cumulative sigma metrics of clinical chemistry and hematology parameters (Table/Fig. 13).

Discussion

In regards to the QC protocol implemented in most of the laboratories, the number of times and number of levels is scheduled based on national accreditation bodies. However, as per good laboratory practices (GLP), each and every laboratory should design a customized individualized quality control plan (IQCP) protocol, based on sigma analysis. By maintaining six standard deviations between the parameter average and its upper and lower limits, the incorporation of sigma metrics results in the reduction of laboratory errors.

According to the analysis of our analyte parameters, the CV% of both the chemistry and hematology parameters were well within the target set up by our lab policy, according to the Westgard desirable specifications. The CV% ranged from 1.30% (albumin) to 4.35% (direct bilirubin) for level 2, and 0.91% (albumin) to 4.63% (direct bilirubin) for level 3 chemistry controls, with the average CV% being 2.65% and 2.30% for all the parameters in L2 and L3 levels, respectively. In hematology, the average CV% came out as 1.30% (high level), 1.82% (low level), and 1.35% (normal level). These values indicate excellent precision for all parameters in both clinical chemistry and hematology, in that the average CV% in both disciplines is below 3.00%.

It was found that the bias percentage, which denotes the accuracy of the analysis, was found to be increased, especially for all the chemistry parameters in the month of May 2022. This contributed to the reduced sigma metrics (less than 3) of many parameters, which include ALP, ALT, AST, total protein, glucose, urea, creatinine, uric acid, and phosphorus. Good and excellent performances, as per the sigma metrics, were exhibited by albumin, total cholesterol, bilirubin (total and direct), calcium, and HDL. All the hematological parameters except RDW and low-level hematocrit showed good and excellent performance as per sigma metrics.

The reduced sigma in major chemistry analytes was identified and categorized as a major incident in the lab, studied, and root cause analysis was done, finding the reduced sigma being due to the reconstitution and mixing error of the RIQAS proficiency testing sample in May 2022.

Similar studies were conducted by Kashyap et al. and Zhou et al., including 15 biochemistry parameters and 16 parameters from both biochemistry and hematology, respectively. The variations in sigma values for a few analytes between our study and these other studies can be due to the difference in the methodology of different parameters, traceability of calibrators used, instruments used, QC material used, reconstitution protocols followed by different laboratories, other pre-analytical and analytical conditions, and the analytical performance requirements of each parameter followed by different laboratories. Parameters whose sigma showed a shift between levels (e.g., albumin, total bilirubin, and hematocrit) should be evaluated with discretion. This indicates that Westgard's multi-rules have to be implemented more stringently in them.

Westgard recently described statistical QC procedures based on sigma metrics-run size matrix and Westgard sigma rules with run size, which includes three parameters: 1) selection of appropriate Westgard sigma rules; 2) the total number of control measurements per statistical quality control event; and 3) frequency of events (run size) of patient samples between SQC events.

The frequency of IQC and the criteria for rejection of each QC run for each of the categories mentioned earlier were designed as follows: Tests > 6 sigma value (excellent tests)- evaluate with two-level QC once a day and 1-3 SD rule. Tests with sigma values between 3 and 6-evaluate with two-level QC once a day (1-2.5 SD rule). Tests with sigma values < 3 sigma-evaluate with two or three-level QC two times a day plus a combination of Westgard rules (1-3S/2-2S/R4S/4-1S).

While analyzing the IQC and EQAS outliers, the quality of test results is dependent on various factors such as reagents quality, type and quality of QC materials, types of analyzers, the methodology followed, environmental conditions, training, and personal competency of laboratory staff performing the tests. Hence, during the root causes analysis and implementation of various corrective and preventive measures, various aspects associated with methodology, materials, personnel, equipment, and working conditions were investigated. Laboratory staff training (for reagent preparation and control material reconstitution, instrument maintenance, reagent handling, and storage) and a periodic competency assessment program was introduced to improve their attitude and knowledge in order to improve precision for parameters/analytes having low sigma values. Further, standard operating procedures were also reviewed for those analytes having low sigma values and rewritten in a simpler and more user-friendly manner.

Limitations of this analysis

Even though the authors had calculated the sigma values of each analyte on a six-month basis, the effectiveness of QC planning using the new QC strategy, for those with low sigma values as mentioned above, is not included in the present paper.

Conclusion

Being an easy and effective tool for implementing quality assurance in the analytical phase of the laboratory, Six Sigma metrics help us to design a QC program and formulate the most ideal methodology for a particular analyte. It is also preferable to keep in mind the probability of false rejection and error detection, while validating the tests against Westgard's rules. However, sigma metrics can be utilized to plan the QC frequency accordingly, thereby upgrading the QMS of a clinical laboratory, which does approximately 80% of tests in-house and thus delivers test results accurately with reliability in a timely fashion. On the basis of sigma metrics analysis, it may be concluded that the Department of Central Laboratory, KMCT Medical College, Kozhikode was able to achieve satisfactory results, with world-class performance of many analytes, as a roadmap towards preparation for National Accreditation Board for Testing and Calibration Laboratories (NABL) accreditation.

Acknowledgements

The authors acknowledge the laboratory technical staffs for their cooperation and assistance.

Ethics

Ethics committee approval and informed consent was obtained for this study.

Competing interests

None declared.

Notes

This presentation is faithful to the original, with changes to presentation, spelling, and grammar as needed. The PMCID and DOI were added when they were missing from the original reference. No citation was given for RIQAS in the original; a citation was provided for this version. No other changes were made in accordance with the "NoDerivatives" portion of the content license.