developing a questionnaire for use in outcomes assessment kathleen a. martin, d.p.e. office of...
Post on 17-Dec-2015
221 Views
Preview:
TRANSCRIPT
DEVELOPING A QUESTIONNAIRE FOR
USE IN OUTCOMES ASSESSMENT
Kathleen A. Martin, D.P.E.Office of Institutional Research & Assessment
Bucknell University
POINT OF CLARIFICATION
Terms “survey” and “questionnaire” are often used interchangeably
“Survey” refers to a descriptive research technique or methodology (also includes interviews, and focus groups)“Questionnaire” refers to a paper and pencil instrument through which information is obtained
For the purposes of outcomes assessment, questionnaires are frequently developed to survey a population
QUESTIONNAIRES & ASSESSMENT
Useful for surveying groups of any sizeTypically designed to determine opinions, attitudes or present practicesLimitation: rely on self report information v. observed behavior
Challenges to validity become a concern
VALIDITY CHALLENGESValidity: the extent to which the test measures what it purports to measureInternal Validity: Relates to instrument (questionnaire) quality. Are the appropriate questions asked clearly and logically?External validity: can the results be generalized?
Measurement error: difference between the characteristics of your sample and the characteristics of the population
Non-response bias: answers of respondents do not mirror the characteristics of those who refused participation.
MAXIMIZING VALIDITYCreate a questionnaire that is clear and logical
Support content relevance and content representativenessContent Relevance: are the questions relevant to the purpose of the questionnaire? Unimportant content should be eliminated.Content Representativeness: Are the questions an adequate representation of the universe of possible questions?Solicit the input of experts in scale development and the content of the questionnaire for an objective evaluation
Select a large, representative random sample from the population of interest or census the finite populationIncrease response rate using formal follow up procedures
INITIAL CONSIDERATIONS
Is the development of a new questionnaire . . .
Necessary?Feasible?
Select mode of administrationWeb Based?Paper and pencil?Combination?
QUESTIONNAIRE PLANNING
1. Determine the ObjectiveWhat do you want to ask and of whom?Consider how the data will be analyzed. What information is needed?
2. Determine the Mode of AdministrationPaper and pencil
Cost effective, but low response rateWeb / E-mail based
Cost effective, higher response rate, some suspicion of the medium
CombinationCan be administratively cumbersome
QUESTIONNAIRE PLANNING
Determine the Sampling MethodologyTo whom will you send the questionnaire? Who can supply the information?Will everyone be surveyed?
ALL current students? ALL alumni?If yes, this is a census of the finite population
The selection of a representative sample of the population may be more effective based on time and cost of administration
Decision depends on the size of the finite population
CONSTRUCTING the QUESTIONNAIRE
Consider what objective each question measuresThree to five questions per objective is recommended
How many questions?Find a balance: long enough to obtain the necessary information but short enough so respondents won’t lose interest.Questionnaires that are too long won’t be completedGoal: 25-30 minutes MAX
DESIGNING THE QUESTIONNAIRE
Question Formats: Open Ended QuestionsAllow respondents to express feelings and expand on ideas. Question should be phrased to avoid a one word response.
“What aspects of the Sociology program best prepared you for your current position?”
Open ended questions at the beginning of the questionnaire may be discouraging to respondent – include one or two at the endAnalysis of responses can be challenging
QUESTIONNAIRE DEVELOPMENTQuestion Formats: Closed Questions
RankingForces the respondent to rank order responses based on a value judgment. Rank the following items with regard to how you spend time at Bucknell from 1 (spend the most time) to 5 (spend the least time).
_____ Studying_____ Socializing_____ Volunteering_____ Working at a part time job_____ Exercising
QUESTIONNAIRE DEVELOPMENTQuestion Formats: Closed Questions
ChecklistA number of possible answers are provided and the respondent is asked to choose one or choose all that apply. How did you finance your Bucknell University education? Choose all that apply.
_____ Parental contribution_____ Student Loan_____ Work Study_____ Off campus employment_____ Other (please specify): _____________
QUESTIONNAIRE DEVELOPMENTQuestion Formats: Closed Questions
Scaled itemsVery common
Indicate strength of agreement or disagreement with a statement.
Numbers are assigned to each response on the continuum
The English curriculum at Bucknell University adequately prepared me for my current position. 1 2 3 4 5
Strongly
Disagree Disagree
Neutral / No
Opinion AgreeStrongly
Agree
QUESTIONNAIRE DEVELOPMENTQuestion Formats: Closed Questions
Suggestions for writing scaled itemsUse positively worded or negatively worded items consistently
Use an odd number of responses along the continuum to allow for a neutral response
Anchors may be different based on focus of the item (Strongly Agree – Strongly Disagree; Frequently – Never; Very Good – Very Poor)
Keep items short
QUESTIONNAIRE DEVELOPMENT
Question Formats: CategoricalTypical demographic questions:
Yes/No/?, Male/Female, F/S/J/S/GAnswers fall into certain categories – respondents pick the one that applies to themAnalyzed using frequency, proportion, percentages2 considerations:
Questions on a continuum should stay on a continuum if they will be directly analyzed
Example: Age – allow respondent to write in their age rather than check off a box for their age range
Categories must be collectively exhaustive & mutually exclusive (next slide)
QUESTIONNAIRE DEVELOPMENT
Problematic:Number of hours spent studying per week:
18 – 21 21 – 2424 – 27
Categories are not mutually exclusive
Corrected:Number of hours spent studying per week:
18 – 2122 – 2526 – 29
IMPORTANT CONSIDERATIONS
APPEARANCE AND DESIGNHave clear directions that are age appropriate & easy to followFormat, size, reproduction should facilitate completion
No less than 12 ptTimes, Arial, Courier
PILOT STUDYGive the questionnaire to a few people to try it outHave them evaluate readability and clarity of questionsEstimate time required to complete the questionnaireSuggestions for revisions
SENDING IT OUTINITIAL CONTACT / COVER LETTER
Short & to the pointIdentify person by nameConvince respondent that participation is importantEnsure confidentialityDeadline for return of questionnaireSASE for return (mailed questionnaires only)
Institutional approvalInstitutional Review Board
Research involving human subjects must be approved by the Bucknell University IRBFor more information, visit the IRB website at http://www.departments.bucknell.edu/inst_research/IRB/irb.htm
Campus Survey CoordinationThe timing of all university-wide surveys must be coordinated through the Campus Survey Coordination Group.For more information, visit the CSCG website at http://www.departments.bucknell.edu/inst_research/cscg.htm
FINAL THOUGHTSFOLLOW UP PROCEDURES
Send to all participants to preserve confidentiality (“If you haven’t already returned the questionnaire…..”)May include a second copy if availableUsually done in 2 week intervals
RESPONSE RATE FOR MAILED QUESTIONNAIRES
Typically low - 30% is considered goodHigher response rate with web-based questionnairesGood follow-up increases response rate
SUMMARY OF STEPS1. Determine the Objective (consider analyses)2. Determine the Mode of Administration3. Determine the Sampling Methodology4. Construct the Questionnaire5. Institutional Approval6. Conduct the Pilot Study7. Write the Initial Communication 8. Send the Questionnaire9. Follow up10. Analyze the Results
top related