ORIGINAL RESEARCH

User Performance Evaluation of Four Blood GlucoseMonitoring Systems Applying ISO 15197:2013Accuracy Criteria and Calculation of Insulin DosingErrors

Guido Freckmann . Nina Jendrike . Annette Baumstark . Stefan Pleus .

Christina Liebing . Cornelia Haug

Received: January 24, 2018 / Published online: March 3, 2018� The Author(s) 2018. This article is an open access publication

ABSTRACT

Introduction: The international standard ISO15197:2013 requires a user performance evalu-ation to assess if intended users are able toobtain accurate blood glucose measurementresults with a self-monitoring of blood glucose(SMBG) system. In this study, user performancewas evaluated for four SMBG systems on thebasis of ISO 15197:2013, and possibly relatedinsulin dosing errors were calculated. Addi-tionally, accuracy was assessed in the hands ofstudy personnel.Methods: Accu-Chek� Performa Connect (A),Contour� plus ONE (B), FreeStyle Optium Neo(C), and OneTouch Select� Plus (D) were eval-uated with one test strip lot. After familiariza-tion with the systems, subjects collected acapillary blood sample and performed an SMBGmeasurement. Study personnel observed the

subjects’ measurement technique. Then, studypersonnel performed SMBG measurements andcomparison measurements. Number and per-centage of SMBG measurements within± 15 mg/dl and ± 15% of the comparisonmeasurements at glucose concentrations \100and C 100 mg/dl, respectively, were calculated.In addition, insulin dosing errors weremodelled.Results: In the hands of lay-users three systemsfulfilled ISO 15197:2013 accuracy criteria withthe investigated test strip lot showing 96% (A),100% (B), and 98% (C) of results within thedefined limits. All systems fulfilled minimumaccuracy criteria in the hands of study person-nel [99% (A), 100% (B), 99.5% (C), 96% (D)].Measurements with all four systems were withinzones of the consensus error grid and surveil-lance error grid associated with no or minimalrisk. Regarding calculated insulin dosing errors,all 99% ranges were between dosing errors of- 2.7 and ? 1.4 units for measurements in thehands of lay-users and between - 2.5 and? 1.4 units for study personnel. Frequent lay-user errors were not checking the test strips’expiry date and applying blood incorrectly.Conclusions: Data obtained in this study showthat not all available SMBG systems compliedwith ISO 15197:2013 accuracy criteria whenmeasurements were performed by lay-users.Trial Registration: The study was registered atClinicalTrials.gov (NCT02916576).

Enhanced content To view enhanced content for thisarticle go to https://doi.org/10.6084/m9.figshare.5909179.

Electronic supplementary material The onlineversion of this article (https://doi.org/10.1007/s13300-018-0392-6) contains supplementary material, which isavailable to authorized users.

G. Freckmann � N. Jendrike � A. Baumstark � S. Pleus� C. Liebing (&) � C. Haug

Institut fur Diabetes-Technologie, Forschungs- undEntwicklungsgesellschaft mbH an der UniversitatUlm, Ulm, Germanye-mail: christina.liebing@idt-ulm.de

Diabetes Ther (2018) 9:683–697

https://doi.org/10.1007/s13300-018-0392-6

Funding: Ascensia Diabetes Care DeutschlandGmbH.

Keywords: Insulin dosing errors; ISO 15197;Lay-user; Self-monitoring of blood glucose;Surveillance error grid; System accuracy; Userperformance evaluation

INTRODUCTION

Accurate measurement results obtained withblood glucose (BG) monitoring systems inten-ded for self-testing by diabetes patients areimportant for correct insulin dosing calcula-tions and adequate therapeutic decisions [1–4].Several self-monitoring of blood glucose(SMBG) measurements per day are recom-mended especially for patients on intensiveinsulin regimens [5]. Regular SMBG is also rec-ommended for patients with less intensiveinsulin therapy [6].

Previous accuracy evaluations often analyzedaccuracy of SMBG systems in the hands of pro-fessionals who are trained in the use of SMBGsystems [7–10]. However, accuracy achieved byprofessionals does not necessarily reflect accu-racy achieved by lay-users, i.e., people withdiabetes [11–14]. The International Organiza-tion for Standardization’s (ISO) standard ISO15197:2013, clause 8 [15], requires a user per-formance evaluation to assess if intended usersare able to obtain accurate BG measurementresults with an SMBG system. ISO 15197:2013was harmonized with the regulations of theEuropean Union as EN ISO 15197:2015 [16].This harmonization had no impact on therequirements and procedures in ISO15197:2013; changes were made to the fore-word and an informative annex. According toISO 15197 clause 8, a system’s user performanceshall be evaluated with one test strip lot apply-ing the following acceptance criteria: at least95% of SMBG measurement results shall fallwithin ± 15 mg/dl (0.83 mmol/l) of comparisonmeasurement results at glucose concentra-tions\100 mg/dl (5.55 mmol/l) and within± 15% at glucose concentrations C 100 mg/dl(5.55 mmol/l).

In this study, accuracy of four SMBG systemsin the hands of intended users was evaluated onthe basis of testing procedures and acceptancecriteria of ISO 15197:2013, clause 8. Accuracywas also evaluated in the hands of trained per-sonnel. Potential insulin dosing errors werecalculated by using a model described in detailby Baumstark et al. [17] in order to estimatepossible influences of SMBG accuracy on gly-cemic control.

METHODS

The study was performed between Septemberand October 2016 at the Institut fur Diabetes-Technologie, Forschungs- und Entwicklungsge-sellschaft mbH an der Universitat Ulm (IDT),Germany. The study was conducted in compli-ance with the German Medical Devices Act,approved by the responsible Ethics Committeein Stuttgart, Germany and exempted fromapproval by the German Federal Institute forDrugs and Medical Devices. All procedures fol-lowed were in accordance with the ethicalstandards of the responsible committee onhuman experimentation (institutional andnational) and with the 1964 Declaration ofHelsinki, as revised in 2013. Informed consentwas obtained from all patients for being inclu-ded in the study.

SUBJECTS

To obtain at least 100 evaluable data sets, intotal 158 subjects were enrolled. Table 1 showsdemographic data for subjects included in dataanalysis. Included subjects did not use theinvestigated SMBG systems for the last 3 yearsaccording to their own statement.

Before study participation, a study physicianreviewed the subjects’ self-reported medicalhistory and medication, the inclusion andexclusion criteria for study participation (e.g.,pregnancy or lactation period, severe acute dis-ease, and/or chronic disease), and checked forinterfering substances given in the manufac-turers’ labelling.

684 Diabetes Ther (2018) 9:683–697

TEST SYSTEMS

In this study, four SMBG systems were evaluated(Table 2): Accu-Chek� Performa Connect (systemA), Contour� plus ONE (system B), FreeStyleOptium Neo (system C), and OneTouch Select�

Plus (system D). All systems were CE-marked.System B is a new system; systems A, C and D areavailable in Eastern Europe and Asia. All systemsdisplayed plasma-equivalent glucose concentra-tions in milligrams per deciliter (mg/dl).

Meters and test strips of system A were pur-chased from a pharmacy in Austria. For systemB, five meters and test strips were purchasedfrom a pharmacy in Poland; 115 meters wereprovided by the manufacturer. Meters of systemC were bought online from amazon.co.uk, andtest strips were purchased from a pharmacy inthe UK. Meters and test strips of system D werepurchased from a pharmacy in Germany.

Meters and test strips were stored, main-tained, and used in compliance with the man-ufacturer’s labelling. Control measurementsaccording to the manufacturer’s labelling wereperformed daily prior to the test procedure andfor each test strip vial to ensure the properfunction of each system.

Since, for systems B and D, the same meterwas used for the familiarization and the

measurement procedure (as described inSect. ‘‘Test Procedure’’), control measurementswere also performed retrospectively after themeasurement procedure to confirm the properfunction of the system.

INSTRUCTIONS FOR USE

According to ISO 15197:2013, clause 8, subjectsshall be given the instructions for use and anytraining material that are routinely providedwith the SMBG system.

For system A, C, and D, instructions for useroutinely provided with the SMBG system wereavailable. For system B, instructions for use wereprovided by the manufacturer. Since not allinstructions for use were available in German,relevant information, i.e., preparation and per-formance of BG measurements, possible errorsources, and error messages, was summarized inGerman in a comparable layout for each system.

COMPARISON MEASUREMENTMETHODS

Comparison measurements were performedwith a glucose oxidase (GOD) method (YSI 2300

Table 1 Subjects demographics

Included subjects System A System B System C System Dn 5 100 n5 100 n5 100 n5 100

Male, female (n) 54, 46 60, 40 59, 41 58, 42

Mean age (min–max) (years) 60.8 (29–80) 60.3 (27–79) 60.7 (25–80) 61.7 (25–80)

Diabetes type 1, 2 (n) 36, 64 35, 65 38, 62 36, 64

Education level (n):

Secondary education grade 70 66 68 67

University entrance qualification 11 17 17 17

University degree 19 17 15 16

Time since diabetes diagnosis,

mean (min–max) (years)

17.8 (1–66) 18 (1–55) 19.3 (1–66) 19.3 (1–66)

Subjects performing at least 1 SMBG

measurement per week (n)95 94 95 94

For each system, 100 different subjects were included in data analysis

Diabetes Ther (2018) 9:683–697 685

Table2

System

characteristics

System

Manufacturer’s

comparison

metho

d

Enzym

etest

strips

Measurementcond

itions

Manufacturer

#Nam

eTeststriplot

(expirydate)

Tem

perature

(�C)

Hum

idity

(%)

Hem

atocrit

(%)

AAccu-Chek

Performa

Connect

475098

(2017-

09)

Hexokinase

Glucose

dehydrogenase

8–44

10–9

010–6

5Roche

DiabetesCareGmbH

,

Mannh

eim,G

ermany

BContour

plus

ONE

DP6

ELHD01C

(2018-05)

Glucose

oxidase

Glucose

dehydrogenase

5–45

10–9

30–

70Bayer

Consumer

CareAG,B

asel,

Switzerland

CFreeStyle

Optium

Neo

4500167624

(2017-12)

Glucose

oxidase

Glucose

dehydrogenase

15–4

010–9

015–6

5AbbottDiabetesCareLtd.,Witney,

Oxon,

UK

DOneTouch

Select

Plus

3997403(2017-

08)

Glucose

oxidase

Glucose

oxidase

10–4

410–9

030–5

5LifeScan

Europe,Divisionof

Cilag

GmbH

InternationalZug,

Switzerland

686 Diabetes Ther (2018) 9:683–697

STAT Plus glucose analyzer, YSI Incorporated,Yellow Springs, OH, USA) and a hexokinase(HK) method (Cobas Integra� 400 plus, RocheInstrument Center, Rotkreuz, Switzerland).Comparison measurements were performed induplicate with both comparison methods. Bothmethods provided BG values in milligrams perdeciliter (mg/dl). For the YSI 2300 STAT Plusglucose analyzer and the Cobas Integra 400 plusinstruments, conformity to traceabilityrequirements of ISO 17511 [18] was confirmedby the manufacturers. Trueness and precision ofthe two analyzers were verified by regularinternal and external quality control measuresas required by the German national guideline(Rili-BAK) [19]. In addition, daily quality con-trol measurements were performed applyingIDT internal standard operating procedures byusing National Institute of Standards andTechnology (NIST) standard reference material(SRM) 965b. The manufacturer’s comparisonmeasurement method and/or the method usedby the manufacturer for SMBG accuracy evalu-ation as indicated in the manufacturer’s label-ling is shown in Table 2.

TEST PROCEDURE

The test procedure was performed on the basisof ISO 15197:2013, clause 8, with one test striplot for each system. For each subject, testingprocedures were performed with three differentSMBG systems; the evaluation of one systemwas completed before the evaluation of anothersystem began. The number of SMBG systemsused by the same subject was limited to threebecause the procedures were very time-con-suming and attention-demanding for individ-ual subjects. To minimize possible order effects,the systems were rotated according to a prede-fined pattern. The familiarization procedure (asdescribed below) was performed with a desig-nated training meter and the measurementprocedure (as described below) with a desig-nated test meter. Test meters were cleaned anddisinfected after each subject. According to thesystems’ labelling, system B and D might beinfectious after use even if they were disin-fected. For these systems, every subject got their

own meter for the familiarization and themeasurement procedure.

For each subject, the capillary blood’shematocrit value was checked to be within therange indicated in the SMBG system’s labelling.For this purpose, samples were collected inheparinized capillaries and the capillaries werecentrifuged. Hematocrit values were deter-mined by using an alignment chart.

Familiarization with SMBG Systems

As requested by ISO 15197:2013, each subjecthad the opportunity to review the instructionsfor use and to practice testing with a givensystem in a manner that represents how lay-users learn to use a new SMBG system. For thispurpose, subjects were allowed to review theinstructions for use and to perform up to threetraining measurements with control solution.

No additional instructions, training, andassistance were provided to the subjects. Inaddition, the familiarization procedure wassupervised by study personnel to prevent anyinfluence on a subject, e.g., by other subjects.

Measurements

Measurements were performed in a controlledlaboratory setting (temperature, 19.8–24.3 �C;humidity, 28.5–56.4%). Each subject wasallowed to use the SMBG system’s instructionsfor use. Measurements were supervised bytrained study personnel.

Each subject washed and dried their handsbefore the measurement procedure. Then, thesubject collected his/her own capillary bloodsample from a fingertip by skin puncture andperformed a measurement using the test meter.For hygiene reasons, the skin puncture wasperformed with a single-use lancing device. Forsystems A, B, and D, study personnel removedeach test strip from the vial and put it on theclosed vial. For system C each test strip waspackaged separately and was removed from thepackage by the subject. Test strip vials or pack-ages were changed after approximately 10subjects.

Diabetes Ther (2018) 9:683–697 687

After the measurement with a given testsystem, subjects were asked if they thought thatthey performed the measurement correctly.Subjects repeated the measurement up to threetimes if they reported making a mistake or if novalid result was obtained. The first quantitativemeasurement result for which a subject did notreport making a mistake in the measurementprocedure was included in the accuracyanalysis.

After the SMBG measurement by the subject,study personnel collected a capillary bloodsample from the subject’s fingertip and per-formed measurements in duplicate with thesame SMBG system, consecutively using thesubject’s test meter and an additional meter.Before and after these duplicate SMBG mea-surements, samples for comparison measure-ments (first and second comparisonmeasurement) were collected in lithium hep-arin tubes, aliquots were centrifuged within7 min, and measurements were performed onseparated plasma.

Human Factors

The subjects’ measurement technique wasobserved and documented by study personnelwith regard to the manufacturer’s labelling, i.e.,preparation of the measurement, insertion oftest strips, blood sampling, application ofblood, and mistakes in device handling. Errorsdocumented only by the study personnel butnot by the subjects did not lead to exclusionfrom accuracy analysis.

DATA ANALYSIS

Data Exclusions

Data were excluded from analysis for the fol-lowing reasons: the subject reported making amistake in his/her measurement procedure andthe measurement was repeated; the number of100 valid measurement results was alreadyreached; the subject obtained no valid resultand the measurement was repeated; hemolysisin plasma samples for comparison

measurements, errors regarding comparisonmeasurements; quality control measurementresults obtained with the comparison methodwere outside predefined limits or no validmeasurement result was obtained; the differ-ence between the first and second comparisonmeasurements exceeded the acceptance criteriafor sample stability (B 4 mg/dl at glucose con-centrations\100 mg/dl or B 4% at glucoseconcentrations C 100 mg/dl).

User Performance Evaluation

For each system, data from 100 subjects(n = 100) were included in the evaluation.Accuracy was evaluated by comparing theSMBG system’s measurement result with therespective comparison measurement result (firstcomparison measurement) according to ISO15197:2013, clause 8, by calculating the num-ber and percentage of results within ± 15 mg/dlof the mean comparison measurement result atBG concentrations\100 mg/dl and within± 15% at BG concentrations C 100 mg/dl. Inaddition, the percentage of results within theclinically acceptable zones A and B of the con-sensus error grid (CEG) and the bias (systematicmeasurement difference) according to Blandand Altman [20] were calculated.

Data are also presented in radar plots[21, 22], an alternative approach to visualize themeasurement accuracy of an SMBG system, andin surveillance error grids (SEG) [23], which isan alternative approach to assess the clinicalrisk associated with SMBG systems’ accuracy.

Evaluation of Human Factors

For each system, the occurrence of user errors inperforming the measurement procedure wasanalyzed.

Evaluation of Accuracy in the Handsof Study Personnel

For each system, data from 100 subjects(n = 200, duplicate measurements) were inclu-ded in the evaluation. Accuracy was evaluatedwith one test strip lot for each system by

688 Diabetes Ther (2018) 9:683–697

comparing the study personnel’s SMBG mea-surement result with the respective mean com-parison measurement result (first and secondcomparison measurement). The same parame-ters as for lay-users were calculated and arepresented as described above.

Calculation of Possibly Related InsulinDosing Errors

All data included in the accuracy evaluation(lay-users n = 100, study personnel n = 200)were assessed regarding possibly related insulindoses based on the model described in detail byBaumstark et al. [17]. In this model, only short-term insulin doses (i.e., meal-related doses anddoses for BG correction), but not long-term(basal) insulin doses were covered.

The model utilizes the mealtime insulindosing formula used by the Diabetes TeachingCenter at the University of California SanFrancisco (UCSF), San Francisco, CA [24]. Insu-lin doses were calculated for comparison

method results and SMBG system results andthe following fixed parameters: 60 g carbohy-drate intake, 1 unit per 15 g insulin-to-carbratio, a target BG value of 100 mg/dL, and aninsulin sensitivity of 25 mg/dL per unit.

RESULTS

Accuracy in the Hands of Lay-users

Accuracy results for each system when evalu-ated against the manufacturer’s comparisonmethod are presented in Table 3, Fig. 1 (differ-ence plot and radar plot), and Fig. 2 (SEG).Results when evaluated against the alternativecomparison method are provided in Supple-mental Table 1 and Supplemental Fig. 1).

Three of the four systems fulfilled ISO15197:2013 accuracy criteria in the hands oflay-users with the investigated test strip lotshowing 96% (A), 100% (B), and 98% (C) ofresults within ± 15 mg/dl and ± 15% of the

Table 3 Accuracy results based on ISO 15197:2013 criteria, resultswithin consensus error grid (CEG) zones A and B, results withinsurveillance error grid zones with ‘‘no risk’’ or ‘‘slight lower risk’’,

relative bias according to Bland and Altman, and calculated insulindosing error

System User Results within Relativebias

Calculated insulin dosing error

–15 mg/dl/–15%

CEGzonesA 1 B

SEG zones

Norisk

Slight,lowerrisk

Median(50thpercentile)

99% ranges betweenthe 0.5th and99.5th percentile

Risk score

0–0.5 > 0.5–1.0% % n % Units

A Lay-users 96 100 100 0 - 7.6 - 0.9 - 2.7 to - 0.1

Study personnel 99 100 197 3 - 6.8 - 0.8 - 2.4 to 0.0

B Lay-users 100 100 100 0 - 2.2 - 0.3 - 1.0 to ? 0.3

Study personnel 100 100 200 0 - 2.2 - 0.2 - 1.0 to ? 0.2

C Lay-users 98 100 97 3 ? 2.8 ? 0.3 - 0.6 to ? 1.4

Study personnel 99.5 100 198 2 ? 1.6 ? 0.2 - 0.7 to ?1.4

D Lay-users 92 100 97 3 - 5.6 - 0.6 - 2.4 to ?0.6

Study personnel 96 100 193 7 - 5.3 - 0.7 - 2.5 to ? 0.3

Results are shown for measurements performed by lay-users (n = 100) and by study personnel (N = 200) when evaluatedby using the manufacturer’s comparison method (hexokinase for system A, glucose oxidase for systems B, C, and D)

Diabetes Ther (2018) 9:683–697 689

690 Diabetes Ther (2018) 9:683–697

comparison method results at BG concentra-tions \100 mg/dl and C 100 mg/dl, respec-tively. One system (D) did not fulfill ISO15197:2013 accuracy criteria with the investi-gated test strip lot showing 92% of resultswithin the limits (± 15 mg/dl and ± 15%). Allfour systems showed 100% of results withinCEG zones A and B. Regarding the SEG, allmeasurements with systems A and B werewithin the ‘‘no risk’’ zone (risk score 0–0.5). Therelative bias ranged from - 7.6% (system A) to? 2.8% (system C) with a smallest bias of- 2.2% (system B). Median modelled insulindosing errors ranged from - 0.9 units (systemA) to ? 0.3 units (system C) with all 99% rangesbetween dosing errors of - 2.7 and ? 1.4 units(Table 3, Fig. 3).

Human Factors

Frequently observed lay-user errors were notchecking the test strip’s expiry date and

incorrect blood application, e.g., application ofan insufficient blood amount, pressing the fin-gertip onto the test strip, or applying the bloodincorrectly onto the test field (Table 4). Becauseall test strips were used in this study before theexpiry date, this error had no impact on mea-surement results.

Accuracy in the Hands of Study Personnel

Accuracy results for one lot of each systemwhen evaluated against the manufacturer’scomparison method are presented in Table 3and Fig. 1 (difference plot and radar plot).Results when evaluated against the alternativecomparison method are provided in Supple-mental Table 1 and Supplemental Fig. 1.

All four systems fulfilled ISO 15197:2013accuracy criteria in the hands of study person-nel with the investigated test strip lot showing99% (A), 100% (B), 99.5% (C), and 96% (D) ofresults within ± 15 mg/dl and ± 15% of thecomparison method results at BG concentra-tions \100 mg/dl and C 100 mg/dl, respec-tively. All four systems showed 100% of resultswithin CEG zones A and B. Regarding the SEG,all measurements with system B were within the‘‘no risk’’ zone (risk score 0–0.5). The relativebias ranged from - 6.8% (system A) to ? 1.6%(system C) with a smallest bias of ? 1.6% (sys-tem C). Median modelled insulin dosing errorsranged from - 0.8 units (system A) to? 0.2 units (system C) all with 99% rangesbetween dosing errors of - 2.5 and ? 1.4 units.

DISCUSSION

In this study, system accuracy of four SMBGsystems in the hands of lay-users, i.e., diabetespatients, was evaluated with one test strip lot onthe basis of testing procedures and accuracycriteria of ISO 15197:2013, clause 8. Addition-ally, accuracy in the hands of study personneltrained in operating the SMBG systems wasevaluated with one test strip lot for each of thefour systems. According to ISO 15197, userperformance evaluation shall demonstrate thatintended users are able to obtain accuratemeasurement results when operating the SMBG

bFig. 1 Difference plots (left side) and radar plots (rightside) for the tested lot of each of the four SMBG systemswhen evaluated against the respective manufacturer’scomparison method (hexokinase for system A, glucoseoxidase for system B, C, D). Measurements performed bylay-users (n = 100) are displayed in blue squares, measure-ments performed by study personnel (n = 200) aredisplayed in orange triangles. Difference plots: ISO15197:2013 accuracy limits (± 15 mg/dl for BG concen-trations\100 mg/dl and ± 15% for BG concentra-tions C 100 mg/dl) are displayed in solid lines. Radarplots: Data points show differences between SMBGmeasurement results and the respective comparison mea-surement result, absolute differences for BG concentra-tions\100 mg/dl and relative differences for BGconcentrations C 100 mg/dl. The absolute values of thedifferences define the location of the data points, i.e., thedistance from the center of the plot, and the sign of thedifferences indicates the hemisphere (positive sign, upperhemisphere; negative sign, lower hemisphere). The direc-tion with respect to the center of the plot in which thedata point lies depends on the comparison method result.In radar plots, high accuracy is represented by tightlygrouped data points close to the center of the plot. Thecircle in dark green highlights the system accuracy limits ofISO 15197:2013 (± 15 mg/dl/± 15%)

Diabetes Ther (2018) 9:683–697 691

system, given only the instructions for use andtraining materials routinely provided with thesystems.

Three out of four systems fulfilled accuracycriteria in the hands of lay-users with the eval-uated test strip lot showing 96% (system A),100% (system B), and 98% (system C) of resultswithin ± 15 mg/dl for BG concentrations\100 mg/dl and ± 15% for BG concentrationsC 100 mg/dl. All systems fulfilled ISO

15197:2013 accuracy criteria in the hands oftrained study personnel. One system (B) showed100% of results in the hand of lay-users andstudy personnel, and similar accuracy results forthis system were obtained in a previous userperformance evaluation [25].

The other three systems showed slightlybetter accuracy results for study personnelmeasurements compared to lay-user measure-ments. Different studies showed that accuracy

Fig. 2 Surveillance error grid analysis for the tested teststrip lot of each of the four SMBG systems. Data areshown for lay-user SMBG measurements (n = 100)

evaluated against the respective manufacturer’s comparisonmethod. Colors indicate associated risk levels ranging fromnone (dark green) to extreme (dark red)

692 Diabetes Ther (2018) 9:683–697

achieved by trained study personnel is oftenbetter than accuracy achieved by lay-persons[11, 12, 26, 27]; however, only few user perfor-mance evaluations were performed on the basisof procedures described in ISO 15197:2013[25, 26, 28–30].

For all four systems, all measurements werewithin zones of the CEG and SEG that areassociated with no or minimal risk, irrespectiveof the user group. ISO 15197:2013, clause 6.3,requires a CEG analysis for system accuracyevaluation. The CEG is developed for type 1diabetes patients and is based on data collectedin 1994 [31]. The CEG is divided into eightzones indicating the estimated risk to thepatient. The SEG also assesses the clinical riskbut is based on much more recent survey data[23]. The SEG is designed as an assessment toolof SMBG systems’ postmarket risk. SEG riskzones are color-coded and data points areassigned to continuous individual risk scores.

For both comparison measurement methodsused in our study, compliance with traceabilityrequirements was confirmed by the manufac-turers. In this study, the comparison measure-ment method/system applied had no impact asto whether the data obtained comply with ISO15197:2013 accuracy criteria or not. However,differences in accuracy data depending on theapplied comparison method/system were foundwhich is consistent with results of two previousISO 15197-based studies performed at ourinstitute [9, 32]. This influence might be

Fig. 3 Modelled insulin dosing errors. Bars in blue (lay-users) and orange (study personnel) indicate ranges inwhich 99% of all dose errors were found, with the whitecircle showing the median dose error. Data are shownwhen evaluated against the respective manufacturer’scomparison method (hexokinase for system A, glucoseoxidase for system B, C, D)

Table 4 Human factors: number of errors in the measurement technique of 100 lay-users observed by study personnel

Number of lay-user errors

SystemA

SystemB

SystemC

SystemD

Preparation of the measurement—testa strip’s expiry date was not checked 90 97 87 101

Insertion of test strips—e.g., not inserted correctly, kinked, bent 7 4 5 4

Blood sampling—e.g., blood drop too small, too old, smeared 3 3 11 8

Blood application—blood was applied incorrectly, e.g., insufficient amount,

fingertip was pressed onto the test strip; blood was not applied onto the test

field correctly

17 20 38 35

Device handling—e.g., blood application before meter was ready for use, test strip

was removed before completion of the measurement, fingertip was not removed

from the test trip after the signal tone

8 6 12 7

Number of incorrectly performed measurements as reported by subjects 4 2 17 7

a Only non-expired test strips were used in this study; therefore, not checking the test strip’s expiry date had no impact onmeasurement results

Diabetes Ther (2018) 9:683–697 693

reduced if manufacturers would adjust calibra-tion of their reference methods [33].

This study design has several limitations.According to ISO 15197:2013, the user perfor-mance evaluation shall also include the evalu-ation of the instructions for use and themessages displayed on the meter by a ques-tionnaire in order to assess whether theinstructions and messages are adequate andeasy to understand. Since not all instructionsfor use were available in German, subjectsreceived a summarized instruction for use (inGerman) for each system with relevant infor-mation for the preparation and performance ofBG measurements. Therefore, instructions foruse and the messages displayed on the respec-tive meters were not evaluated. In this study,only one lot was evaluated on the basis of ISO15197:2013, clause 8. According to ISO15197:2013, clause 6.3, system accuracy evalu-ation in the hands of study personnel shall beperformed with three test strip lots, since lot-to-lot variations between multiple test strip lotsused for the same system can markedly affect asystem’s accuracy results [11, 34, 35].

Regarding the subjects’ measurement tech-nique, not checking the expiry date of test stripsbefore the measurement was the most frequenterror. It might be that subjects assumed thatthey were only provided with non-expired teststrips. Because test strips were used in this studybefore their expiry date, this error had noimpact on measurement results. However, dif-ferent studies showed that the use of deterio-rated test strips can affect SMBG measurementresults [36, 37].

For all four systems, an incorrect bloodapplication, e.g., applying an insufficientamount of blood, pressing the fingertip ontothe test strip, or applying the blood not ontothe test field, was the most frequent error duringthe measurement procedure. Since incorrectblood application can potentially affect SMBGmeasurement results, adequate SMBG trainingand education should be an integral part ofdiabetes therapy [36, 38].

Accurate SMBG measurement results arecrucial for adequate insulin dosing decisions bypatients on insulin therapy. Modelling analysesshowed that inaccurate BG measurements can

lead to insulin dosing errors [1, 3, 4, 39, 40]which can adversely affect glycemic control andincrease the risk of long-term complications inpatients on intensive insulin therapy. In thisstudy, negative median insulin dosing errors(too small insulin doses) were calculated forthree systems (A, B, D) and positive medianinsulin dosing errors (too high insulin doses)were calculated for one system (C). Except forone system (D), median calculated insulin dos-ing errors were larger when SMBG measure-ments were performed by lay-persons. Inaddition, the four systems showed differences inthe width of the error range (99% rangesbetween the 0.5th percentile and the 99.5thpercentile).

CONCLUSIONS

In this user performance evaluation, three outof four systems fulfilled ISO 15197:2013 accu-racy criteria with the investigated test strip lot.However, the systems showed slight differencesin the number of results within ISO 15197:2013accuracy limits. All four systems fulfilled ISO15197:2013 accuracy criteria with the investi-gated test strip lot in the hands of study per-sonnel, with three systems showing betteraccuracy results when measurements were per-formed by trained study personnel. One system(B) showed 100% of results within the ISO15197:2013 accuracy limits in the hands of lay-users and in the hands of study personnel. Inboth user groups, calculated median insulindosing errors were within ± 0.9 units and ten-ded to be larger when measurements were per-formed by lay-users. Inaccurate SMBGmeasurements can result in insulin dosingerrors and adversely affect glycemic control.Therefore, manufacturers should provide highquality SMBG systems that are easy to use andresistant to user errors.

ACKNOWLEDGMENTS

Funding. This investigator-initiated trialand the medical writing were supported by

694 Diabetes Ther (2018) 9:683–697

Ascensia Diabetes Care Deutschland GmbH,Germany. Article processing charges were fun-ded by the authors’ institute Institut fur Dia-betes-Technologie, Forschungs- undEntwicklungsgesellschaft mbH an der Univer-sitat Ulm (IDT), Ulm.

Medical Writing, Editorial, and OtherAssistance. We would like to thank the studypersonnel and other employees of the IDT whoconducted the study or helped in preparing themanuscript. We would like to thank ScottPardo, PhD, PStat�, for his assistance in mod-eling insulin dosing errors.

Authorship. All named authors meet theInternational Committee of Medical JournalEditors (ICMJE) criteria for authorship for thismanuscript, take responsibility for the integrityof the work as a whole, and have given finalapproval for the version to be published. Allauthors had full access to all of the data in thisstudy and take complete responsibility for theintegrity of the data and accuracy of the dataanalysis.

Previous Presentations. Parts of this study’sresults were presented at the 77th ScientificSessions of the American Diabetes Associationin San Diego, CA, June 9–13, 2017 and at theInternational Diabetes Federation’s congress inAbu Dhabi, UAE, December 4–8, 2017.

Thanking Patients. We thank the partici-pants of the study.

Disclosures. Guido Freckmann is generalmanager of the IDT, which carries out clinicalstudies on the evaluation of BG meters andmedical devices for diabetes therapy on its owninitiative and on behalf of various companies.Guido Freckmann/IDT have received speakers’honoraria or consulting fees from Abbott,Ascensia, Bayer, LifeScan, Menarini Diagnostics,Novo Nordisk, Roche, Sanofi, Sensile, andYpsomed. Nina Jendrike is an employee ofthe IDT. Annette Baumstark is an employee ofthe IDT. Stefan Pleus is an employee of the IDT.Christina Liebing is an employee of the IDT.Cornelia Haug is an employee of the IDT.

Compliance with Ethics Guidelines. Allprocedures followed were in accordance withthe ethical standards of the responsible com-mittee on human experimentation (institu-tional and national) and with the 1964Declaration of Helsinki, as revised in 2013.Informed consent was obtained from allpatients for being included in the study.

Data Availability. The datasets generatedand/or analyzed during the current study areavailable from the corresponding author onreasonable request.

Open Access. This article is distributedunder the terms of the Creative CommonsAttribution-NonCommercial 4.0 InternationalLicense (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercialuse, distribution, and reproduction in anymedium, provided you give appropriate credit tothe original author(s) and the source, provide alink to the Creative Commons license, andindicate if changes were made.

REFERENCES

1. Boyd JC, Bruns DE. Quality specifications for glu-cose meters: assessment by simulation modeling oferrors in insulin dose. Clin Chem. 2001;47:209–14.

2. Raine CH III, Schrock LE, Edelman SV, et al. Sig-nificant insulin dose errors may occur if bloodglucose results are obtained from miscoded meters.J Diabetes Sci Technol. 2007;1:205–10.

3. Virdi NS, Mahoney JJ. Importance of blood glucosemeter and carbohydrate estimation accuracy. J Dia-betes Sci Technol. 2012;6:921–6.

4. Breton MD, Kovatchev BP. Impact of blood glucoseself-monitoring errors on glucose variability, riskfor hypoglycemia, and average glucose control intype 1 diabetes: an in silico study. J Diabetes SciTechnol. 2010;4:562–70.

5. ADA. Standards of medical care in diabetes—2017.Diabetes Care. 2017;40:1–135.

6. Elgart JF, Gonzalez L, Prestes M, Rucci E, Gagliar-dino JJ. Frequency of self-monitoring blood glucoseand attainment of HbA1c target values. Acta Dia-betol. 2016;53:57–62.

Diabetes Ther (2018) 9:683–697 695

7. Klaff LJ, Brazg R, Hughes K, et al. Accuracy evalua-tion of contour next compared with five bloodglucose monitoring systems across a wide range ofblood glucose concentrations occurring in a clinicalresearch setting. Diabetes Technol Ther.2015;17:8–15.

8. Brazg RL, Klaff LJ, Sussman AM. New generationblood glucose monitoring system exceeds interna-tional accuracy standards. J Diabetes Sci Technol.2016;10:1414–5.

9. Freckmann G, Link M, Schmid C, et al. Systemaccuracy evaluation of different blood glucosemonitoring systems following ISO 15197:2013 byusing two different comparison methods. DiabetesTechnol Ther. 2015;17:635–48.

10. Hasslacher C, Kulozik F, Platten I. Analytical per-formance of glucose monitoring systems at differ-ent blood glucose ranges and analysis of outliers ina clinical setting. J Diabetes Sci Technol.2014;8:466–72.

11. Kristensen GB, Monsen G, Skeie S, Sandberg S.Standardized evaluation of nine instruments forself-monitoring of blood glucose. Diabetes TechnolTher. 2008;10:467–77.

12. Littmann K, Petersen ER, Pussinen C, et al. Evalua-tion of OneTouch Verio(�), a new blood glucoseself-monitoring system for patients with diabetes.Scand J Clin Lab Invest. 2013;73:286–92.

13. Bailey TS, Wallace JF, Parkes JL, et al. Performanceof a new blood glucose monitoring system in thehands of intended users. Diabetes Technol Ther.2012;14:783–9.

14. Solvik UO, Risa M, Jacobsen CE, Monsen G, Sand-berg S. Performance of 10 systems for self-moni-toring of blood glucose by trained healthcareprofessionals and in the hands of the users. ClinChem. 2015;61:772–4.

15. International Organization for Standardization.In vitro diagnostic test systems—requirements forblood-glucose monitoring systems for self-testing inmanaging diabetes mellitus. ISO 15197:2013.

16. International Organization for Standardization.In vitro diagnostic test systems—requirements forblood-glucose monitoring systems for self-testing inmanaging diabetes mellitus (ISO 15197:2013). ENISO 15197:2015.

17. Baumstark A, Jendrike N, Pleus S, Haug C, Freck-mann G. Evaluation of accuracy of six blood glu-cose monitoring systems and modeling of possiblyrelated insulin dosing errors. Diabetes TechnolTher. 2017;19:580–8.

18. International Organization for Standardization.In vitro diagnostic medical devices—measurementof quantities in biological samples—metrologicaltraceability of values assigned to calibrators andcontrol materials. EN ISO 17511:2003.

19. Bundesarztekammer. Richtlinie der Bun-desarztekammer zur Qualitatssicherung laboratori-umsmedizinischer Untersuchungen. DeutschesArzteblatt. 2014;111:1583-618.

20. Bland JM, Altman DG. Statistical methods forassessing agreement between two methods of clin-ical measurement. Lancet. 1986;1:307–10.

21. Simmons DA. How should blood glucose metersystem analytical performance be assessed? J Dia-betes Sci Technol. 2015;10:178–84.

22. Pardo S, Dunne N, Simmons DA. Using radar plotsto demonstrate the accuracy and precision of 6blood glucose monitoring systems. J Diabetes SciTechnol. 2017;11:966–9.

23. Klonoff D, Lias C, Vigersky R, et al. The surveillanceerror grid. J Diabetes Sci Technol. 2014;8:658–72.

24. Diabetes Education Online. http://dtc.ucsf.edu/types-of-diabetes/type2/treatment-of-type-2-diabetes/medications-and-therapies/type-2-insulin-rx/calculating-insulin-dose/#mealtime. Diabetes TeachingCenter at the University of California, San Fran-cisco. 2018. Accessed Feb 2018.

25. Bailey TS, Wallace JF, Pardo S, et al. Accuracy anduser performance evaluation of a new, wireless-en-abled blood glucose monitoring system that links toa smart mobile device. J Diabetes Sci Technol.2017;11:736–43.

26. Freckmann G, Baumstark A, Jendrike N, et al.Accuracy evaluation of four blood glucose moni-toring systems in the hands of intended users andtrained personnel based on ISO 15197 require-ments. Diabetes Technol Ther. 2017;19:246–54.

27. Skeie S, Thue G, Nerhus K, Sandberg S. Instrumentsfor self-monitoring of blood glucose: comparisonsof testing quality achieved by patients and a tech-nician. Clin Chem. 2002;48:994–1003.

28. Christiansen M, Greene C, Pardo S, et al. A new,wireless-enabled blood glucose monitoring systemthat links to a smart mobile device: accuracy anduser performance evaluation. J Diabetes Sci Tech-nol. 2017;11:567–73.

29. Bailey T, Wallace JF, Greene C, et al. Accuracy anduser performance evaluation of the Contour(�)Next Link 2.4 blood glucose monitoring system.Clin Chim Acta. 2015;448:139–45.

696 Diabetes Ther (2018) 9:683–697

30. Caswell M, Frank J, Viggiani MT, et al. Accuracy anduser performance evaluation of a blood glucosemonitoring system. Diabetes Technol Ther.2015;17:152–8.

31. Parkes JL, Slatin SL, Pardo S, Ginsberg BH. A newconsensus error grid to evaluate the clinical signif-icance of inaccuracies in the measurement of bloodglucose. Diabetes Care. 2000;23:1143–8.

32. Link M, Schmid C, Pleus S, et al. System accuracyevaluation of four systems for self-monitoring ofblood glucose following ISO 15197 using a glucoseoxidase and a hexokinase-based comparisonmethod. J Diabetes Sci Technol. 2015;9:1041–50.

33. Bailey TS, Klaff LJ, Wallace JF, et al. Fundamentalimportance of reference glucose analyzer accuracyfor evaluating the performance of blood glucosemonitoring systems (BGMSs). J Diabetes Sci Tech-nol. 2016;10:872–5.

34. Baumstark A, Pleus S, Schmid C, et al. Lot-to-lotvariability of test strips and accuracy assessment ofsystems for self-monitoring of blood glucoseaccording to ISO 15197. J Diabetes Sci Technol.2012;6:1076–86.

35. Kristensen GB, Christensen NG, Thue G, SandbergS. Between-lot variation in external quality assess-ment of glucose: clinical importance and effect on

participant performance evaluation. Clin Chem.2005;51:1632–6.

36. Muller UA, Hammerlein A, Casper A, Schultz M.Community pharmacy-based intervention toimprove self-monitoring of blood glucose in type 2diabetic patients. Pharmacy Pract (Granada).2006;4:195–203.

37. Bamberg R, Schulman K, MacKenzie M, Moore J,Olchesky S. Effect of adverse storage conditions onperformance of glucometer test strips. Clin Lab Sci.2005;18:203–9.

38. Bergenstal R, Pearson J, Cembrowski GS, et al.Identifying variables associated with inaccurateself-monitoring of blood glucose: proposed guide-lines to improve accuracy. Diabetes Educ.2000;26:981–9.

39. Bedini JL, Wallace JF, Pardo S, Petruschke T. Perfor-mance evaluation of three blood glucose monitoringsystems using ISO 15197: 2013 accuracy criteria,consensus and surveillance error grid analyses, andinsulin dosing error modeling in a hospital setting.J Diabetes Sci Technol. 2015;10:85–92.

40. Pardo S, Dunne N, Simmons DA. Bolus insulin doseerror distributions based on results from two clini-cal trials comparing blood glucose monitoring sys-tems. J Diabetes Sci Technol. 2017;11:970–4.

Diabetes Ther (2018) 9:683–697 697