The audio portion of the conference may be accessed via the telephone or by using your computer's

speakers. Please refer to the instructions emailed to registrants for additional information. If you

have any questions, please contact Customer Service at 1-800-926-7926 ext. 10.

Presenting a live 90-minute webinar with interactive Q&A

Medicaid Overpayments: Challenging State

Audit Allegations of Overpayment and the

Use of Statistical Sampling and Extrapolation

Today’s faculty features:

1pm Eastern | 12pm Central | 11am Mountain | 10am Pacific

TUESDAY, OCTOBER 11, 2016

David R. Ross, Shareholder, O’Connell and Aronowitz, Albany, N.Y.

Dr. Patricia L. Maykuth, Ph.D, President, Research Design Associates, Decatur, Ga.

Tips for Optimal Quality

Sound Quality

If you are listening via your computer speakers, please note that the quality

of your sound will vary depending on the speed and quality of your internet

connection.

If the sound quality is not satisfactory, you may listen via the phone: dial

1-866-927-5568 and enter your PIN when prompted. Otherwise, please

send us a chat or e-mail sound@straffordpub.com immediately so we can

address the problem.

If you dialed in and have any difficulties during the call, press *0 for assistance.

Viewing Quality

To maximize your screen, press the F11 key on your keyboard. To exit full screen,

press the F11 key again.

FOR LIVE EVENT ONLY

Continuing Education Credits

In order for us to process your continuing education credit, you must confirm your

participation in this webinar by completing and submitting the Attendance

Affirmation/Evaluation after the webinar.

A link to the Attendance Affirmation/Evaluation will be in the thank you email

that you will receive immediately following the program.

For additional information about continuing education, call us at 1-800-926-7926

ext. 35.

FOR LIVE EVENT ONLY

Program Materials

If you have not printed the conference materials for this program, please

complete the following steps:

• Click on the ^ symbol next to “Conference Materials” in the middle of the left-

hand column on your screen.

• Click on the tab labeled “Handouts” that appears, and there you will see a

PDF of the slides for today's program.

• Double click on the PDF and a separate page will open.

• Print the slides by clicking on the printer icon.

FOR LIVE EVENT ONLY

MEDICAID OVERPAYMENTS: Challenging State Audit Allegations of Overpayment and the Use of Statistical Sampling and Extrapolation

P re s e nte d b y : Pa t M ay ku th , P h . D. , P re s i dent , Re s e a rc h D e s i g n A s s o c i ate s , I n c . D av i d R . Ro s s , E s q . , S h a re h o l de r, O ’C o nn e l l & A ro n ow i t z , P.C .

Tu e s d ay, O c t o be r 1 1 , 2 0 1 6

1 : 0 0 p . m . t o 2 : 3 0 p . m .

A Primer on Sampling and Extrapolation

PART I OF THE PRESENTATION

6

Right Idea, Wrong Answer

7

There Are Three Kinds of Lies: Lies, Damned Lies, and Statistics So which is it? The outcome of any statistical study is determined by that characteristic that is sampled

Why are so many polls in this election saying contradictory things?

◦ bias

◦ corrupted frame

◦ probability of selection unknown

◦ polling done for advocacy not to answer the question

◦ sampling error

◦ response bias

8

Dewey Wins the Election!!

Pollsters surveyed via telephone

9

Ultimately, It Comes Down to Two Basic Questions

1. What is the ‘it’ that is being measured?

2. Is ‘it’ being measured correctly?

10

What is Statistics? Statistics is a branch of mathematics dealing with: ◦ the collection,

◦ analysis,

◦ interpretation, and

◦ presentation of masses of quantitative data

11

Terms Universe – dollar amount of claims paid to a Provider in a specific timeframe

Sampling Frame – subset of the universe defined as variables of interest from which the sample will be randomly selected and over which the sample will be extrapolated

Sample – a randomly selected subset of a sampling frame to be audited for overpayments

Unit of Analysis (Sampling Unit) – what is measured in the audit: claim line, claim, beneficiary, provider (must be invariant throughout the audit)

(Please see the List of Terms provided with presentation)

12

Universe (Provider and all claims)

Frame (All claims for a particular time

frame and code)

Sample (50 claims per sample)

01/01/2010

Code: 12743

Sampling Unit (Individual claim: must be within frame)

13

Corrupted Frame

Fra Sampling Frame

Universe

Outside Definition

SAMPLE

14

Characteristics of a Valid Medicaid

Overpayment Study

15

universe frame

Inferential Statistics Method: following rules required

Sample definition

Simple

Select seed & random number

table

Applied to file that is unbiased

Calculate sample

size

Select probability

sample

Independent observations Randomly selected Normally distributed Representative

Stratified

Audit claims in the field

Prior history of overpayment

error

Define the who, what, time and how overpayment is measured

16

Statistics Is Different from Numbers

Statistical analysis that uses probability theory to generate and properly interpret inferences

Probability statistics uses probability distributions for decision making. Probability theory is the mathematical basis of those distributions tested repeatedly

Use subsets of data (sample) to estimate variables in a larger data set (frame/universe)

17

What is Statistical Sampling? For the purpose of today’s discussion: A Statistically Valid Random Sample (“SVRS”) from a universe of paid Medicaid claims ◦ Guards against cherry picking or any bias

A sample has to:

◦ meet the requirements the methodology,

◦ meet chosen sampling error,

◦ be of sufficient size to accurately measure the variable,

◦ be random, and

◦ be representative (without bias)

18

Probability Statistics and SVRS To know the number of samples of the chosen size

that can be created from the frame

Known likelihood of selection of each sampling unit

Proper randomization

Proper execution of sample methodology

Use correct formulae

Accurate measurement of the variable of interest (overpayment)

Medicare reference: MPIM 8.4.2; Cochran Sampling Methodology

19

What is Extrapolation?

Extrapolation takes the results of an audited sample of claims and projects the dollar amount of the overpayment from the sample over the universe of paid claims

The audited sample has a known amount of dollars in error so that amount is projected to the universe for a repayment amount

20

Medicare and Statistical Sampling

Medicare Program Integrity Manual (MPIM) Chapter 8

1 provides nineteen pages of guidance “to

provide instructions for [auditors] for the use of statistical sampling in their reviews to calculate and project (i.e., extrapolate) overpayment amounts to be recovered”

1 https://www.cms.gov/Regulations-and-

Guidance/Guidance/Manuals/Downloads/pim83c08.pdf

21

Valid Outcomes Require Proper Execution

Defined universe

Defined frame

Defined sampling units

Use proper randomization

Accurately measure overpayments

Use the correct formulas for estimation

Test key assumptions of method

Accurately report actual findings

22

How to Properly Execute the Audit Methodology

Choose appropriate sampling unit, methodology and definition for the audit data

Meet the criteria of that statistical model

Exercise knowledgeable statistical oversight and quality control throughout

Document the process so that it can be replicated

Evaluate non-sampling errors and their impact

Calculate the results correctly

Report findings accurately and ethically

23

Random Samples A simple random sample is a sequence of independent, identically distributed (IID) random variables.

The term random sample is ubiquitous in mathematical statistics while the abbreviation IID is just as common in basic probability. Much of basic probability and mathematical statistics deal with random parameters constructed from random samples (sample mean, sample variance, sample covariance, and order statistics).

The point estimate (CMS’s common overpayment estimator) is a type of mean making these fundamental concepts critical to the prediction.

24

25

Representative Samples A small yet complete and accurate picture of the data in the frame.

A subset of a statistical universe that accurately reflects the numerical membership of the entire universe and its distribution. A representative sample is an unbiased indication of what the frame is like.

Representativeness is tested mathematically. When a sample is not representative, the result is known as a sampling error.

26

Representative Sample

Frame

Sample

27

28

Key Requirements for Use of Parametric Statistics

Use a sample that is: ◦ Made up of independent observations

◦ Randomly selected

◦ Normally distributed

◦ Representative of the frame from which it was chosen and over which it will be extrapolated

29

Sample Size Determination Based on Chosen Precision and Confidence

30

RAT-STATS Results

80% 90% 95% 99%

1% 75 77 78 79

2% 63 68 71 75

5% 29* 39 46 56

10% 10* 15* 20* 29*

15% 5* 8* 10* 16*

Confidence Level

Precision Level

* Sample sizes less than 30

31

Normal Distribution

32

What’s the Big Deal About the Normal Distribution?

In probability theory, the normal (or bell-shaped) distribution is continuous probability distribution (a function that tells the probability of a number in some context falling between any two real numbers). The normal distribution is symmetric around the mean. The mean, median and mode are the same number.

The normal distribution is immensely useful because of the Central Limit Theorem - which states that the mean of many random variables independently drawn from the same frame is distributed approximately normally, irrespective of the form of the original distribution. That is, the overpayment means will be randomly distributed if the sample is large ... moving toward infinity.

33

Lower Confidence Level

Upper Confidence Level

Point Estimate

34

35

Mode

Mean

Median

0 12,072.62 566.09 1,274.35

Where is the Confidence Level? One-sided or Two?

Illustration not based on data

36

Frame Distribution

37

38

Sample Frequency in Dollars N 15

Mean 430.93

Median 112.00

Mode 120

Std. Deviation 525.519

Variance 76,170.210

Skewness 0.816

Kurtosis 0.580

Minimum 10

Maximum 1,300

Sum 6,464

39

Non Normal Mean and Standard Deviation

Sample mean +/- 1 standard deviation

Mean = $430.92 + sd = 525.510 = 956.43

430.92 - 525.510 = -94.18

40

Sample Overpayment Frequency in Dollars

N 15

Mean 339.333

Median 100.00

Mode 0.0

Std. Deviation 511.62

Variance 261,763.810

Skewness 1.197

Kurtosis -0.524

Minimum 10

Maximum 1,300

Sum 6,464

41

Amount Paid is a Proxy for Overpayment

Sample selected from amount paid to provider

Sample analyzed using overpayment data

Never know up front what the overpayment amount is going to be unless ◦ There is a known history of overpayment dollar amount; OR

◦ Conduct a probe

Overpayment amounts must meet criteria for using parametric statistics or the confidence levels are destroyed

If overpayments are not correlated with amount paid the analysis cannot proceed

42

RAT-STATS Overpayment Estimation

Formulae:

Confidence Level:

43

RAT-STATS Point Estimate & Confidence Interval

POINT ESTIMATE 27,147

90% CONFIDENCE LEVEL

LOWER LIMIT 10,368

UPPER LIMIT 43,925

PRECISION AMOUNT 16,778

PRECISION PERCENT 61.81%

T-VALUE USED 1.761310135775

Lower 27,147 + 16,778 = $43,925

Upper 27,147 - 16,778 = $ 10,368

Confidence Interval = 16,778 + 16,778 = 33,556

44

Confidence Levels Non Normal Data

Point Estimate +/- ½ Confidence Interval

Lower 27,147 + 16,778 = $43, 925

Upper 27,147 - 16,778 = $ 10,368

Confidence Interval = 16,778 + 16,778 = 33,556

Lower confidence Level ??? Unknowable

45

Poor Audit Design & Execution Produce Only Invalid Results

Statistics in the hands of an inept auditor are like a lamppost to a drunk: they are used more for

support than illumination.

46

Legal Issues Pertaining to Sampling and

Extrapolation

PART II OF THE PRESENTATION

47

Legal Basis for Statistical Sampling for Overpayment Estimation

US HHS “may introduce the results of a statistical sampling study as evidence of the number of violations . . . or the factors considered in determining the amount of [a] civil money penalty. Such statistical sampling study, if based upon an appropriate sampling and computed by valid statistical methods, constitutes prima facie evidence. [T]he burden . . . shifts to the [Provider] to produce evidence reasonably calculated to rebut the findings of the statistical sampling study.”

Excerpt from 45 C.F.R. § 160.536(a)-(b) (emphasis added).

48

What Are the Legal Issues Pertaining to Sampling and Extrapolation?

Understanding what those issues are

Do I need an expert? (Hint: you absolutely do) ◦ Lawyers are not enough (and that’s coming from a

lawyer)

49

Individual States and Statistical Sampling

Unlike Medicare, there is no universal guidance outlining the rules of the process for Medicaid

Your state may have simple or detailed rules, or no rules at all ◦ Texas follows the MPIM and uses RAT-STATS ◦ California and Florida use “generally accepted statistical

standards” ◦ New York has no rules at all

The burden of proof is on the Provider to challenge the statistical sampling and extrapolation ◦ The State’s audit methodology is presumed valid unless

and until the Provider proves otherwise

50

The Role of the Expert:

Why You Should Always Have One A statistical consultant is essential to understanding how the audit was conducted and whether the results are statistically valid ◦ Laypersons cannot understand these concepts

absent training

The expert will review both the process actually used and the results in your case

An expert is the only person who will be able to answer important questions pertaining to the audit

51

The Role of the Expert: Asking the Important Questions

Is the process used by the auditors properly designed?

Is this process suitable for use in this audit of my client?

Is the process, as applied to my client in this audit, proper?

Did it deviate from the norm? If so, how and why?

How was the sample size chosen? Is it adequate (large enough) for the confidence interval?

Did the auditors properly document the audit?

52

The Role of the Expert: Asking More Important Questions Is the software program certified or widely accepted for

this purpose?

Was adequate documentation for replication provided?

Was the software program used working as it was designed? ◦ When in doubt have an expert look at the source code

◦ Can’t get the source code? Argue violation of due process

53

The Role of the Expert: Asking Even More Important Questions

How were the random numbers selected? Were they generated by a valid method? ◦ Did the auditors perform appropriate tests for

randomness?

Can the results be replicated by the provider? ◦ If not, there is a serious problem ◦ Retaining the seed is an important factor ◦ Is the frame sorted as it was at the time the random

numbers were used to pick the sample?

What about the extrapolation calculations themselves?

54

Questions?

55

Faculty David R. Ross, Esq. Shareholder dross@oalaw.com (518) 462 5601 O’Connell and Aronowitz, P.C. 54 State Street Albany, NY 12207 www.oalaw.com

Pat Maykuth, Ph.D. President pm@researchdesignassociates.com (404) 373 4637 Research Design Associates, Inc. 721 E Ponce de Leon Decatur, GA 30030 www.researchdesignassociates.com

56