Analyzing data: Synthesis

What is merit determination? What is absolute merit? How to draw

rubric? What is relative merit? How to draw

rubric?

Review

S519

It is a tool to allow us to draw overall evaluative conclusions from multiple findings about a single evaluand.

Synthesis is the process of combining a set of ratings or performances on several components or dimensions into an overall rating.

Synthesis methodology

S519

Merit determination To develop the rubrics To use rubrics to summarize the multiple

findings Rubrics are one of the simplest methods to

blend data. But when data is a bit more complex, it is

difficult to use a rubric as the only tool Data are not equally important or reliable Multi dimensions or multi components Different nuances and combinations (such as

Table8.3)

Synthesis methodology

S519

It is not meta-analysis A special statistical technique to give a

weighted average of effect sizes across multiple studies – for quantitative studies

It is not literature review or a summary A judgment from a reviewer’s point of

view.

It is not

S519

Doing poorly on some minimal important criteria

Doing poorly on some crucial criteria Are very different!

Keep in mind

S519

Synthesis for „ranking“ If it is „ranking“ (relative) evaluation:

Consider each alternative and make explicit comparisons

Synthesis for „grading“ If it is „grading“ (absolute) evaluation:

Consider different context settings and provide better interpretation of merit

Evaluation

S519

Quantitative synthesis Using numerical weights

Qualitative synthesis Using qualitative labels

Qualitative or quantitative

S519

The primary evaluation question is for absolute quality or value How well did the evaluand perform on

this dimension or component? How effective, valuable, or meritorious

is the evaluand overall? Is this worth the resources put into it?

Synthesis for “grading”

S519

Case: Personnel evaluation in a small accounting firm 13 defined tasks (e.g., telephone, reception,

data entry, etc.) Each employee has responsibility for 4-6 tasks

Evaluation: Importance weighting (through the voting of

the selected stakeholders) In-depth discussion with business owners Derive the importance metric and bars

Quantitative weighting example with „bars“

S519

Evaluation Define the levels of importance: 3 to 5 levels work well in most case Do not go to too many levels (why? Is this useful?) For example

task 1. minor task (1) 2. normal-priority task (2) 3. high-priority task (3) 4. extremely high-priority task (4)

Quantitative weighting example with „bars“

S519

Evaluation Setting up rubrics for each 13 tasks Normally 4-6 level is sufficient

Example: Performance Rubric 1. Totally unacceptable performance (1) 2. Mediocre (substandard) performance (2) 3. Good performance (expected level) (3) 4. Performance that exceeded expectations (4) 5. All-around excellent performance (5)

Synthesis – draw the overall conclusion See Exhibit 9.2 (p158)

Quantitative weighting example with „bars“

S519

Personal evaluation in a small accouting firm

Exercise

Tasks Importance Score for Alice

Telephone 1 2

Data entry 2 3

Tax data management

4 1

Client support 4 5

Reporting 3 3

Communicating 1 3

How about Alice according to Exhibit 9.2?

S519

Personal evaluation in a small accouting firm

Exercise

Tasks Importance Score for John

Telephone 1 2

Data entry 2 3

Tax data management

4 2

Client support 4 5

Reporting 3 3

Communicating 1 3

How about John according to Exhibit 9.2?

S519

Perosnal evaluation in a small accouting firm

Exercise

Tasks Importance Score for Chris

Telephone 1 2

Data entry 2 3

Tax data management

4 4

Client support 4 5

Reporting 3 3

Communicating 1 3

How about Chris according to Exhibit 9.2?

S519

How about Chris Mean=

1*2+2*3+4*4+4*5+3*3+1*3/(1+2+4+4+3+1)=56/15=3.73

What is Chris‘ performance?

Exerice

S519

Case: a school-based health program evaluation It contains 9 different components: nutrition education,

mental health services, safer sex, legal service and others.

How to evaluate these systems in low-budget and short period of time whether they are meeting important needs of the students and their families

Evaluation: Interview Student surveys

Qualitative weighting example 1 (with no „bars“)

S519

Survey question design: Two quantitative questions

How useful was the program to you? (4-point response scale: not at all useful, somewhat useful, useful, very useful)

How satisfied were you with the program? One qualitative question (open-end)?

What other changes or events, good or bad, have happened to you or someone you know because of receiving the service?

School health system evaluation

S519

Survey result about nutrition system shows in Table 9.1

Look at table 9.1, think about: How can you draw a conclusion from

this result about the nutrition system? Is it good or bad?

School health system evaluation

S519

Setting the importance for these three questions (1-strongest data, 3=weakest data) 1. Ratings of usefulness (directly related

to needs) 2. Responses to the open-ended

question 3. Satisfaction ratings

Creating rubrics for each question Table 9.2 for question 1 and question 2 Table 9.3 for open-ended question

School health system evaluation

S519

How to grade the nutrition system based on the first two quantitative questions: Based on Table 9.1, come out with the

rubric as Table 9.2 Why 90% is select, 70%-90%.. How to draw Table 9.2 from Table 9.1 and

collected data?

School health system evaluation

S519

Table 9.3 Rubric for converting data from

qualitative evaluation - open-ended responses into merit ratings

Is that a good way to do this? Are you happy with this table? If not, how do you want to improve it?

School health system evaluation

S519

Synthesis to draw overall conclusion Step-by-step Start with the strongest data (question

1) Blend with open-ended comments Finally take the satisfaction ratings into

account

See table 9.4 for the whole process

School health system evaluation

S519

How to draw final conclusion?

School health system evaluation

Usefulness ratings

Satisfaction ratings

Open-ended comments

Final coclusion:

Merit of the nutrition program

See table 9.4Discuss how to apply this to your group project

Using quantitative ratings to draw the suggested results and using qualitative ratings to find the positive or negative facts to re-adjust the results

S519

Bar A minimum level of performance on a specific dimension Performance below this cannot be compensated for by

much better performance on other dimensions (see Exhibit 9.2)

Hard hurdle (also referred as global bars) Overall passing requirement for an evaluand as a whole

(see Exhibit 9.2) Soft hurdle

Overall requirement for entry into a high rating category Place a limit on the maximum rating (e.g., I want all As

for my classes)

Qualitative (nonnumerical) weighting example 2

S519

Case: Evaluation of the learning capacity of a small biotechnology start-up company „biosleep“.

Evaluation 27 subdimensions of organizational learning

capacity (see table 9.5) Data collection: survey and interview Rubric: similar as Table 8.2 Importance is built by using strategy 6 in Chapter 7

Using program theory and evidence of causal linkages (p118-125)

Qualitative (nonnumerical) weighting example 2

S519

Evaluation Synthesis

Pack the ratings on the subdimensions into 8 main dimensions

Combine the ratings on these 8 main dimensions to draw an overall conclusion

Biosleep

S519

Dimension by dimension Layer by layer

Biosleep

Sub-dimnention1

Sub-dimnention2

Sub-dimnention3

Sub-dimnention4

Dimnention1

Dimnention2

Overall rating

S519

Synthesis Subdimensions Dimensions

Using Table 9.6 to draw conclusions of dimentions based on subdimensions

Using Table 9.6 to judge Table 9.5 and come out the result as Exhibit 9.4

Dimensions overall evaluation Based on Table 9.7 (created based on

literature review, What is your conclusion for the evaluation of

Biosleep? And why?

Biosleep

S519

Form your group project Discuss on how are you going to

grade your evaluation? Which example you would like to follow? How to develop rubric for dimension and

overall?

Exericse

S519