SDASA One-Day Conference Best Practices in Statistical Consulting

October 16, 2013Colleen Kelly, Ph.D.Principal Statistician

What are the attributes of the ideal consultant?

Condensed from the list prepared by the ASA Committee to examine the training needs of statisticians (1980):

• Well trained in theory and practice of statistics

• generalist rather than specialist

• Good oral and written communication skills

• Can work within the constraints of the real world

• Team player

What situations do we want to avoid?

Examples of Dissatisfied Clients: (from Derr 2000)

“The experience was unacceptable to me . . . [Consultant] was consistently late to meetings – sometimes not showing up at all. . .[recently] he did not return any email and dropped out of sight?!”

“The time frame in waiting . . . was longer than expected”

“I received poor advice. I did not know it at the time . . . but I ended up changing the model [after consulting with someone else].”

What situations do we want to avoid?

1. Not delivering results in a timely manner/not meeting an important deadline

2. Sending an unexpectedly large bill

3. Errors of the third kind in statistical consulting (Kimball 1957):

“Solving precisely the wrong problem”

Errors of the third kind

What the client wanted What the consultant understood

The solution conceived by the consultant

Errors of the third kind

How the client was billed The solution actuallydelivered

What the client really needed

Tips for Successful Consulting

1. Plan for good communication

2. Be a good communicator

3. Check that you have communicated what you wanted to.

Tips for a Successful First Meeting

1. Plan for good communication• Know your client• Be interested in their problem• Have a list of questions that you need answered

2. Be a good communicator• Listen• Ask questions

3. Check that you have communicated what you wanted to.• Describe to the client your understanding of the problem• Discuss what tasks you will do and what tasks they will do• Discuss timeline of project• A written statement of work (SOW) may be appropriate.

Tips for Study Design: look for ways to be efficient

Tips for Study Design: look for ways to be efficient• Parallel v. cross-over design

• Blocking, pairing, adjusting for covariates (what factors affect response?)

• More accurate measurements?

• Repeated measurements?

• Reducing drop-outs.

Tips for Study Design: Sample Size Rules of Thumb

Normally distributed response (van Belle 2002):

1. One arm study (pre- v. post- comparison):

(n= 32 for medium effect, n=13 for large effect)

2. Comparison of two groups (i.e. two-sample t-test):

(n=64 for medium effect, n=26 for large effect)

Tips for Study Design:Sample Size Rules of Thumb

Proportions (van Belle 2002): 1. The rule of threes: Given no observed events in n

trials, an upper 95% bound for the rate of occurrence is:

2. Comparison of two proportions:

(n=64 to compare 10% to 30% )

Tips for Study Design:Other Design Rules of Thumb

Regression (van Belle 2002): 1. Using continuous variables is generally preferable

(i.e. more powerful) than dichotomizing them.

2. Multiple logistic regression:

• You need at least 10 events per variable for stable regression coefficients.

Tips for Statistical Modeling• Educate yourself on the science involved (Box 1976 argues

that the best applied statisticians are more appropriately called scientists).

• Your solution should be as simple as possible, but not overly so.

• Choice of a statistical model:

“Since all models are wrong, the scientist must be alert to what is importantly wrong. It is inappropriate to be concerned about mice when there are tigers abroad.” - Box (1976)

Tips for Statistical ModelingThe mice: “In nature there never was a normal distribution, there never was a straight line, yet with normal and linear assumptions, known to be false, he can often derive results which match, to a useful approximation, those found in the real world.” - Box (1976)

The tigers: Violating assumptions of independence (see Hurlbert 1984), not adjusting for fishing expeditions (see Dmitrienko et al. 2010 and Wood et al. 2007) and improper analysis of missing data (Little and Rubin 2002) can result in more serious errors.

The Tigers:

1.Violating assumptions of independence:

Story A: To assess the precision of a new device, a client plotted the errors (new device - reference device) against the reference values. The errors were negatively correlated with the reference.

Story B: To assess the precision of a new device, 10 subjects were measured 15 times (every 2 minutes) with new and reference devices. A confidence interval for the mean error was constructed using the 150 measurements.

The Tigers:

2. Not adjusting for fishing expeditions:

A. In clinical trials, there are often multiple response variables of interest or there may be a number of doses to compare to placebo.

B. Data mining (searches for biomarkers or genetic signatures of disease).

The Tigers:3. Analysis of missing data:

High-Rise Syndrome: A 1987 article in JAVMA reported a study of 115 New York city cats that fell 2 to 32 floors. On average, they fell 5.5 floors. 90% survived. Mortality and injury rates decreased with increasingly large falls.

Somewhat embarrassingly, it wasn’t the scientific community that questioned the result, it was Ann Landers’ readers:

“Where in the world is the SPCA? Where are the police? Does anyone really care how a cat lands? Who goes around dropping

cats anyway?”

Deadly Statistical Sins

Tips for a Successful Presentation of the Solution1. Plan for good communication

• Write an organized report • Write an organized powerpoint presentation• Explain the results in the client’s language.

2. Be a good communicator• Listen• Ask questions• Avoid statistical jargon, or explain it

3. Check that you have communicated what you wanted to.

Other Tips

• Manage the client’s expectations with regard to time, cost and deliverables

• Be aware of the client’s timeline and budget

• Be aware of regulations for the analysis • developmental or pilot study • confirmatory or pivotal study• FDA submission?• Pharmaceutical v. device v. diagnostic

Conclusions:

1. Communicate, communicate, communicate!

2. These are goals to strive for – when we fall short of the “ideal consultant”, we can take it as a opportunity to learn how to do better.

3. The rewards of consulting are to continually learn new science and statistical methods and to make contributions to science.

References:Box G. 1976. Science and Statistics. JAMA 71: 791-789.

Cabrera J. and McDougall A. 2002. Statistical Consulting. Springer-Verlag, New York, NY.

Dmitrienko A., Tamhane A. and Bretz F. 2010. Multiple Testing Problems in Pharmaceutical Statistics. CRC Press, Boca Raton FL

Derr J. 2000. Statistical Consulting: A Guide to Effective Communication. Brooks Cole, Canada.

Hurlbert S. 1984. Pseudoreplication and the Design of Ecological Field Experiments. Ecological Monographs 54:187-211.

Kimball A.W. 1957. Errors of the Third Kind in Statistical Consulting. JAMA 52: 133-142.

Little R. and Rubin D. 2002. Statistical Analysis with Missing Data, Second Edition. John Wiley and Sons, Inc. Hoboken NJ

Van Belle G. 2002. Statistical Rules of Thumb. John Wiley, New York, NY.

For More Information

Contact:Colleen KellyKelly Statistical Consulting, Inc.760-846-6763

www.kellystatisticalconsulting.com

kstat.consulting@gmail.com

Types of Consulting

Academic Consulting

Industry Consulting

Expert Witness Testimony

Type of client More statistically sophisticated

Less statistically sophisticated

Least statistically sophisticated

Deadlines Longer and more flexible

Shorter and less flexible

Short and inflexible (but erratic)

Pay Low Good Very good