survey workshop slides final - aacc · 2018-11-27 · preconference workshop evalu‐ate.org 8 ate...

Preconference Workshop

Evalu‐ate.org 1

ATE PI Conference 2018

GOOD DATAstarts with

GOOD QUESTIONS

Survey Fundamentals

Lyssa Wilson Becho & Lori Wingate

ATE Principal Investigators Conference | 2018

Webinars | Newsletter | Resource Library | Blog

www.evalu-ate.org


Evalu‐ate.org 2


This material is based upon work supported by the National Science Foundation under Grant No. 1600992.

The content reflects the views of the authors andnot necessarily those of NSF.

INTRODUCTIONSIntroduceyourself to your tablemates

Find 5 things you have in common


Evalu‐ate.org 3


WHY SURVEYS?WHY SURVEYS?

WHY SURVEYS?


Evalu‐ate.org 4


SURVEY PROCESS

CLARIFY SURVEY PURPOSE

CREATE SURVEY

TEST SURVEY

DISSEMINATE SURVEY

CLEANDATA

ANALYZEDATA

COMMUNICATEFINDINGS


Evalu‐ate.org 5


WHY QUESTION QUALITY MATTERS

There is no such thing as a ____________ question.


Evalu‐ate.org 6


There are no right answers to wrong questions.

―Ursula K. LeGuin

SURVEY CRITIQUE

YOUR TURN:


Evalu‐ate.org 7


1. Complete survey as if you just finished this workshop.

2. Mark any problems you see with the questions and make notes.

3. Share results at your tables.

We hope that you found this workshop to be

worthwhile and educational and that you will be able

to apply what you learned in order to enhance your

evaluation work and the projects you evaluate. The

purpose of this survey is to obtain your feedback on

the quality of this workshop.


Evalu‐ate.org 8


We hope that you found this workshop to be

worthwhile and educational and that you will be able

to apply what you learned in order to enhance your

evaluation work and the projects you evaluate. The



The



Thank you for your participation in this session.


Evalu‐ate.org 9


For Questions 1-9 below, indicate the extent to which

you agree or disagree with each statement, using the

following scale: 1 = Strongly Disagree, 2= Disagree, 3 =

Neither Disagree nor Agree, 4 = Agree, 5 = Strongly

Agree, 6 = Not sure/Don’t know

For Questions 1-9 below, indicate the extent to which

you agree or disagree with each statement, using the

following scale: 1 = Strongly Disagree, 2= Disagree, 3 =

Neither Disagree nor Agree, 4 = Agree, 5 = Strongly

Agree, 6 = Not sure/Don’t know

Strongly

disagreeDisagree

Neither

Agree nor

Disagree

DisagreeStrongly

Agree

Don’t

Know/

Not sure


Evalu‐ate.org 10


1

SD

2

D

3

N

4

A

5

SA

6

DK

Question 1. The room in which

this workshop was conducted

was comfortable.

1

SD

2

D

3

N

4

A

5

SA

6

DK

Question 2. The instructional

design was appropriate in

relation to the workshop

objectives and knowledge

level of participants.


Evalu‐ate.org 11


1

SD

2

D

3

N

4

A

5

SA

6

DK

Question 3. This workshop met

my expectations.

1

SD

2

D

3

N

4

A

5

SA

6

DK

Question 4. The instructor was

engaging and knowledgeable.


Evalu‐ate.org 12


1

SD

2

D

3

N

4

A

5

SA

6

DK

Question 5. It will not be

difficult for me to apply what

I learned.

1

SD

2

D

3

N

4

A

5

SA

6

DK

Question 6. I am very

interested in learning more

on this topic.


Evalu‐ate.org 13


1

SD

2

D

3

N

4

A

5

SA

6

DK

Question 7. I definitely will use

what I learned in this

workshop.

1

SD

2

D

3

N

4

A

5

SA

6

DK

Question 8. I would strongly

recommend this workshop to

my peers.


Evalu‐ate.org 14


1

SD

2

D

3

N

4

A

5

SA

6

DK

Question 9. I would support

bringing this workshop to my

campus.

1

SD

2

D

3

N

4

A

5

SA

9

DK

The room in which this workshop was conducted

was comfortable.

The instructional design was appropriate in relation

to the workshop objectives and knowledge level of

participants.

This workshop met my expectations.

The instructor was engaging and knowledgeable.

It will not be difficult for me to apply what I learned.

I am very interested in learning more on this topic.

I definitely will use what I learned in this workshop.

I would strongly recommend this workshop to my

peers.

I would support bringing this workshop to my

campus.


Evalu‐ate.org 15


1

SD

2

D

3

N

4

A

5

SA

9

DK


was comfortable.



participants.







peers.


campus.

1

SD

2

D

3

N

4

A

5

SA

9

DK


was comfortable.



participants.







peers.


campus.

SATISFACTION


Evalu‐ate.org 16


How satisfied are you with this workshop?

4-point scaleFully anchoredUnipolarNo mid-point

Not at all Satisfied Somewhat Satisfied Mostly Satisfied Completely Satisfied

7-point scaleFully anchoredBipolarWith midpoint

Very Dissatisfied Moderately Dissatisfied Slightly Dissatisfied Neutral Slightly Satisfied Moderately Satisfied Very Satisfied



Evalu‐ate.org 17


10-point scalePartially anchoredBipolarNo midpoint

1 – Very Dissatisfied 2 3 4 5 6 7 8 9 10 - Very Satisfied


1

SD

2

D

3

N

4

A

5

SA

9

DK


was comfortable.



participants.







peers.


campus.

ACTION


Evalu‐ate.org 18


Will you __________________________?[behavior of interest]

Definitely not Probably not Possibly Probably Very Probably Definitely

Back to our survey…


Evalu‐ate.org 19


Question 10. What is your opinion of the overall quality

of this workshop?

Met my expectations

Good

Very good

Exceeded my expectations


of this workshop?

Met my expectations

Good

Very good

Exceeded my expectations Excellent

Fair


Evalu‐ate.org 20



of this workshop?

Poor

Met my expectations

Good

Very good

Exceeded my expectations Excellent

Fair

What were the most helpful topics addressed in this

workshop? Which aspects of the workshop were

especially good?


Evalu‐ate.org 21


What were the most helpful topics addressed in this

workshop? Which aspects of the workshop were

especially good?

Which aspects of the workshop were especially good?

Which aspects of the workshop need improvement?

or

What were the workshop’s strengths?

What were the workshop’s weaknesses?


Evalu‐ate.org 22


Question 12. In what disciplinary area do you conduct

evaluations?

Advanced Manufacturing

Agricultural and

Environmental Technologies

Bio and Chemical

Technologies

Engineering Technologies

Information and Securities

Technologies

Micro and

Nanotechnologies

Question 12. In what disciplinary area do you conduct

evaluations?

Advanced Manufacturing

Agricultural and

Environmental Technologies

Bio and Chemical

Technologies

Engineering Technologies

Information and Securities

Technologies

Micro and

Nanotechnologies

Not Applicable Other


Evalu‐ate.org 23


Question 13. What is your age?

Under 20

20-25

25-30

30-35

Over 30


Under 20 30

20-25 30-39

25-30 40-49

30-35 50-59

Over 30 60 or older


Evalu‐ate.org 24



Under 20 30

20-25 30-39

25-30 40-49

30-35 50-59

Over 30 60 or older

Question 14. How many surveys have you (a)

completed and (b) developed?


Evalu‐ate.org 25


Question 14. How many surveys have you (a)

completed and (b) developed?

How much experience do you have with survey

development?

?

?

?

?

?

BASIC PRINCIPLES OF GOOD SURVEY QUESTIONS


Evalu‐ate.org 26


ORDINARY SCALE EXAMPLES

ATE Principal Investigators Conference 2018

Ordinal scales – measurement scales for data that have a logical order, but no

standard distance between categories – are frequently used in surveys to measure the

extent of an attitude, belief, knowledge, or behavior. Below are some examples of

scales in varying lengths.

AGREEMENT

Strongly Agree Agree Disagree Strongly Disagree

Completely Agree Mostly Agree Slightly Agree Slightly Disagree Mostly Disagree Completely Disagree

Completely True Somewhat True Slightly True Neither True nor False Slightly False Somewhat False Completely False

FREQUENCY

Always Very Often Sometimes Rarely Never

A Great Deal Quite a Bit Somewhat Very Little Not at All

Always Very Frequently Occasionally Rarely Very Rarely Never IMPORTANCE

Highly Important Important Slightly Important Not Important

Extremely Important Important Moderately Important Somewhat Important Not Very Important

Very Important Important Somewhat Important Somewhat Unimportant Unimportant Very Unimportant

Lyssa Wilson Becho & Lori Wingate

ASKING ABOUT DEMOGRAPHICS


Evalu‐ate.org 27


ASKING ABOUT DEMOGRAPHICS

ASKING ABOUT AGE

How old are you?

In what year were you born?

What is your age range?

o 18 – 25o 26 – 34o 35 – 44o 45 – 54o 55 – 64o 65 or older


Evalu‐ate.org 28


ASKING ABOUT RACE & ETHNICITY

Which categories describe you?Select all that apply to you. American Indian or Alaskan Native Asian Black or African American Hispanic, Latino/Latina, or Spanish OriginMiddle Eastern or North Africa Native Hawaiian or Other Pacific IslanderWhite Some other race, please specify: _________ I prefer not to answer


American Community Survey, 2018


Evalu‐ate.org 29







Evalu‐ate.org 30



ATE Annual Survey, 2018

ASKING ABOUT GENDER & SEX

What is your gender identity?

o Femaleo Maleo Prefer to describe: __________o Prefer not to answer


Evalu‐ate.org 31


SUMMARY DEMOGRAPHICS

o Only ask what you need

o Be inclusive

o Be sensitive to respondent

o Consider utility and analysis

o Place demographics at the end

WRITING QUESTIONS

YOUR TURN:

DirectionsWrite 3 – 5 survey questions using the guidelines we discussed.


Evalu‐ate.org 32


SURVEY PRETESTING

Ways to pretest your survey:

Checklists & assessment tools

Expert review

Think-aloud

Focus group

Pilot testing

SURVEY PRETESTING


Evalu‐ate.org 33


DirectionsSwitch with a neighbor and conduct a think-aloud.Discuss how questions can be improved.

PRETEST YOUR QUESTIONS

YOUR TURN:

OTHER RESOURCES

Internet, Phone, Mail, and Mixed-Mode Surveys: The Tailored Design Method by Dillman, Smyth, and Christian (2014)


Evalu‐ate.org 34


OTHER RESOURCES

Designing Quality Survey Questions by Robinson & Leonard (2018)

OTHER RESOURCES

More online resources at:

bit.ly/ATEPI

survey workshop slides final - aacc · 2018-11-27 · preconference workshop evalu‐ate.org 8 ate...

Documents