multivariate detection of aberrant billing: an evaluation
DESCRIPTION
Multivariate Detection of Aberrant Billing: An Evaluation. Maharaj Singh, Ted Wallace & Martin Schrager National Government Services, Inc. Outline of the study. Outlier defined Multivariate method for detecting Outlier Detecting outlier billing providers An evaluation of the methodology - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/1.jpg)
Multivariate Detection of Multivariate Detection of Aberrant Billing: An Aberrant Billing: An EvaluationEvaluation
Maharaj Singh, Ted Wallace & Martin SchragerMaharaj Singh, Ted Wallace & Martin SchragerNational Government Services, IncNational Government Services, Inc..
![Page 2: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/2.jpg)
2
Outline of the studyOutline of the study• Outlier defined• Multivariate method for detecting
Outlier • Detecting outlier billing providers• An evaluation of the methodology • Other factors considered for
future application• Conclusion
![Page 3: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/3.jpg)
3
An Outlier An Outlier
•An outlier is not ‘Outlier’. May be we haven’t yet found the ‘right’ Distribution.
![Page 4: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/4.jpg)
4
Multivariate MethodMultivariate Method
• We used a Mahalanobis distance as multivariate vector corresponding to each observation in the data set.
![Page 5: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/5.jpg)
5
Mahalanobis DistanceMahalanobis Distance
• Mahalanobis distance is a distance measure introduced by Prasanta Chandra Mahalanobis in 1936.
• It is based on Correlation between variables by which different patterns can be identified and analyzed.
• It is a useful way of determining similarity of an unknown sample set to a known one
![Page 6: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/6.jpg)
6
Mahalanobis distanceMahalanobis distance
• Mahalanobis distance
![Page 7: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/7.jpg)
7
Mahalanobis distance and Multivariate Mahalanobis distance and Multivariate OutliersOutliers
• Mahalanobis D2 is a multidimensional version of a z-score. It measures the distance of a case from the centroid (multidimensional mean) of a distribution, given the covariance (multidimensional variance) of the distribution.
![Page 8: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/8.jpg)
8
DD22
• A case is a multivariate outlier if the probability associated with its D2 is 0.05 or less. D2 follows a chi-square distribution with degrees of freedom equal to the number of variables included in the calculation.
• Mahalanobis D2 requires that the variables be metric, i.e. interval level or ordinal level variables that are treated as metric.
![Page 9: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/9.jpg)
9
Mahalanobis Distance from EllipsoidMahalanobis Distance from Ellipsoid
• Mahalanobis distance measure is based on correlations among the variables by which different patterns can be identified and analyzed.
• The region of constant Mahalanobis distance around the mean forms an ellipsoid when more than two variables are used.
![Page 10: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/10.jpg)
10
Multivariate trimming …Multivariate trimming …
• The χ2 plot for multivariate data is not resistant to the effect of outliers.
• A few discrepant observations can affect the mean vector, and can potentially influence the outcome.
• In order to avoid the effect of a few discrepant observations, we used multivariate trimming which involved an iterative process of setting aside the observations with largest squared distance and the trimmed statistics are computed from the remaining observations.
• At the end of this iterative process, the new squared distance values are computed using the robust statistics.
![Page 11: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/11.jpg)
11
Chi Square plot for the datasetChi Square plot for the datasetχ2
![Page 12: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/12.jpg)
12
The Data Set: Paid ClaimsThe Data Set: Paid Claims
![Page 13: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/13.jpg)
13
Indices for the Id of Indices for the Id of observationsobservations
– Billing providers• Location, • Size • Specialty
– HCPCS used – Primary
Diagnoses
![Page 14: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/14.jpg)
14
Matrix of Utilization Variables: AmountMatrix of Utilization Variables: Amount
• Cost– Charges Billed– Charges denied– Reimbursement
![Page 15: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/15.jpg)
15
Matrix of Utilization Variables: RateMatrix of Utilization Variables: Rate
• Rate – Reimbursement per
beneficiary– Service Units per
beneficiary– Service units per
service dates per beneficiary
![Page 16: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/16.jpg)
16
Matrix of Utilization Variables: VolumeMatrix of Utilization Variables: Volume
• Volume– Number of claims– Number of
beneficiaries– Number of service
units rendered– Number of service
days
![Page 17: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/17.jpg)
17
The cost: Medicare trust The cost: Medicare trust fund $$$fund $$$
• For each observation the amount paid is a function of the rate and the volume.
• However for each observation Id, the rate and volume variables are also highly inter-correlated.
![Page 18: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/18.jpg)
18
Methodology for Paid Claim Methodology for Paid Claim Data SetData Set
![Page 19: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/19.jpg)
19
Data StepsData Steps
• The line-level (detailed) paid claim data was summarized by id (provider-HCPCS combination) with summary of the utilization variables (cost, rate and volume).
![Page 20: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/20.jpg)
20
Principal ComponentsPrincipal Components
• The variables in the matrix of the paid claims dataset were converted into principal components.
• The distance squared was computed as unique sum of squares principal components.
![Page 21: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/21.jpg)
21
Multivariate TrimmingMultivariate Trimming
• The iterative process of multivariate trimming was used.
![Page 22: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/22.jpg)
22
DD22 and Expected Chi Square and Expected Chi Square ValueValue
• Corresponding to the square distance the expected chi square value along with its probability were computed.
χ2
![Page 23: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/23.jpg)
23
Outlier ObservationsOutlier Observations
• The observations with probability < .05 are treated as outliers and are flagged.
• The flagged observations are treated as candidates for probe by medical review and/or treated as potential CERT errors and referred to the Provider Education Unit.
Outlier
![Page 24: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/24.jpg)
24
Outlier Observations PrioritizedOutlier Observations Prioritized
• Finally the outlier observation were prioritized by the magnitude of distance measure, expected chi-square value and the probability associated with measure.
![Page 25: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/25.jpg)
25
Evaluation of Outlier Evaluation of Outlier ClassificationClassification
• Once each observation in the dataset has been classified as an outlier or non-outlier by using chi-square distribution, we used logistic regression to find out the estimate of the goodness of fit of the model.
![Page 26: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/26.jpg)
26
C StatisticsC Statistics
• In order to find out how accurately we were able identity the outlier observations we used C Statistics.
![Page 27: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/27.jpg)
27
C Statistics…C Statistics…
• The value of c statistics varies from 0.5 ( randomly assigning to one of the other category) to 1.0 where the observations are correctly assigned to the categories.
![Page 28: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/28.jpg)
28
An example from Paid An example from Paid Claim DatasetClaim Dataset
![Page 29: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/29.jpg)
29
Think about Some Random Think about Some Random Numbers??Numbers??
4 - 100 - 40 - 60
![Page 30: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/30.jpg)
30
Outlier Proportion of NGS UtilizationOutlier Proportion of NGS Utilization
• Provider HCPC Line 03.46%• Provider Counts 99.49%• Total Reimbursement
37.90%• Total Units 58.78%• Provider HCPC Benes
44.21%• Provider HCPC Claims 48.21%
![Page 31: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/31.jpg)
31
Model EvaluationModel Evaluation
• The outlier model for NGS data was evaluated by using goodness of fit test.
• The NGS combined data set has 930,260 Provider HCPC Lines.
• Of the total lines there were 32,145 were outliers lines.
• The Chi Square for the model was 344144.04 with Probability being < 0.0001.
![Page 32: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/32.jpg)
32
C statistics C statistics
• Association of Predicted Probabilities and Observed Responses
Percent Concordant 94.9% Percent Discordant 3.6%Percent Tied 1.5%
c statistic 0.956
![Page 33: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/33.jpg)
33
Past and Future ApplicationPast and Future Application
![Page 34: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/34.jpg)
34
Current ApplicationCurrent Application
• Used as a single factor in determining multivariate statistical outliers– Problem areas– HCPC codes– Individual Providers
![Page 35: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/35.jpg)
35
Current ApplicationCurrent Application
• Positives– Confidence (statistically valid
methodology)– Consistent methodology regardless of
problem area– Lack of clinical bias
• Negatives– Difficult to interpret– Volume of provider/HCPC
combinations required for valid analysis
– Lack of clinical bias
![Page 36: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/36.jpg)
36
Future ApplicationFuture Application
• Using the squared distance as a factor in determining outlier problem areas
• Using the squared distance as a factor in determining the aberrancy index of a provider
![Page 37: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/37.jpg)
37
Future Application – Problem Future Application – Problem AreasAreas
Problem AreaCERT Factor
Factor 2
Factor 3
Squared Distance Factor
Weight 1
Weight 2
Weight 3
Weight 4
Aggregate Index
Problem Rank
Problem Area 5 10 10 8 0.33872 4 1 2 4 6,735.49 1
Problem Area 10 8 6 9 0.86849 4 1 2 4 5,947.40 2
Problem Area 3 10 8 1 0.75349 4 1 2 4 5,301.40 3
Problem Area 7 10 4 3 0.06234 4 1 2 4 5,024.94 4
Problem Area 9 9 6 3 0.18039 4 1 2 4 4,872.16 5
Problem Area 8 8 1 1 0.43183 4 1 2 4 3,672.73 6
Problem Area 2 6 6 2 0.99527 4 1 2 4 3,798.11 7
Problem Area 6 4 10 1 0.74227 4 1 2 4 3,096.91 8
Problem Area 4 8 2 2 0.22331 4 1 2 4 3,889.32 9
Problem Area 1 2 1 3 0.66469 4 1 2 4 1,765.87 10
![Page 38: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/38.jpg)
38
Future Application – Problem Future Application – Problem ProvidersProviders
ProviderCERT Factor
Factor 2
Factor 3
Squared Distance Factor
Weight 1
Weight 2
Weight 3
Weight 4
Aggregate Index Provider Rank
Provider 4 10 10 8 0.55720 4 1 2 4 6,822.88 1
Provider 6 10 5 7 0.98560 4 1 2 4 6,294.24 2
Provider 2 7 2 10 0.12170 4 1 2 4 5,048.68 3
Provider 10 4 6 10 0.89450 4 1 2 4 4,557.80 4
Provider 7 6 9 6 0.13890 4 1 2 4 4,555.56 5
Provider 1 6 1 8 0.23790 4 1 2 4 4,195.16 6
Provider 5 4 3 5 0.79580 4 1 2 4 3,218.32 7
Provider 8 2 2 5 0.20110 4 1 2 4 2,080.44 8
Provider 3 3 1 2 0.55390 4 1 2 4 1,921.56 9
Provider 9 2 5 1 0.69240 4 1 2 4 1,776.96 10
![Page 39: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/39.jpg)
39
The multivariate modelThe multivariate model
• By using multivariate model only 4% of total Provider-HCPC combinations lines were identified as outliers.– However the 4% of the total lines have
captured almost 100% of the NGS providers and questioned their 40% of their payment in the Quarter 4 of 2007.
![Page 40: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/40.jpg)
40
Testing of the model as Testing of the model as classifierclassifier
• Using multivariate model with multivariate trimming we were able to identify each observation (provider-hcpcs combination) to be as outlier or non-outlier.
• Using this method we were able identify outliers with a very high concordance ( 94.6%).
![Page 41: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/41.jpg)
41
ConclusionConclusion
• We used multivariate statistical method to identify aberrant billing and utilization in the claim data set and tested the validity of the method by using logistic regression.
• We also noted that statistical method alone is not enough and we need to add other factors to add value to the process of identifying the problem areas as well finding the high value target.
![Page 42: Multivariate Detection of Aberrant Billing: An Evaluation](https://reader036.vdocuments.mx/reader036/viewer/2022062321/56813a2c550346895da212f7/html5/thumbnails/42.jpg)
42