Unités de recherche éducationnelle/Educational Research Units

L’adulte et l’apprentissage en milieu de travail/Adult and Workplace Learning

Mesure et évaluation des apprentissages/Measurement, Evaluation and Assessment

Séminaire d’après-midi/Afternoon Seminar30 janvier 2013/January 30th , 2013

De 16h à 17h15/4:00 to 5:15 p.m. LMX 388

La présentation sera en anglais/Presentation will be in English

Des rafraichissements seront servis/Refreshments will be served

RSVP avant le 28 janvier à :

RSVP before January 28th at:

claire.duchesne@uottawa.ca

Tips on how to develop measurement tools and collect data

Ivana Pavic, Maurice Taylor, Sait Atas and David Trumpower

Tips on how to develop measurement tools

• Involve your learners and instructors in the development (Ivana)

• Ensure the tool has a strong conceptual development (Maurice)

• Pilot test the tool to obtain evidence of reliability, validity, utility (Sait)

• Revise, as necessary, following psychometric analysis (David)

Let’s talk about the networks in our lives

Maurice Taylor & Ivana Pavic

• Talking about the social networks in our lives is very much related to a term called

SOCIAL CAPITAL

• Social capital relates to the connections we have with people

• …..connections to family

• …..connections to friends

• …..connections to the workplace

• …..connections to the neighbourhood

• …..connections to the local community

Why are these connections important?

1) they help us advance our interests in the society we live in

2) they help us co-operate within groups

3) they promote trust, goodwill and a desire to act in a supportive manner

Some examples

• The Ottawa East Community Association

• Immaculata Parent Council• The Art of Running Club• The Ottawa Newcomers Club • Sandy Hill Community Health

Centre Volunteer• One Parent Families Association • Jamaican Ottawa Community Association

What are the benefits of social capital?

• Glue- holds society together by facilitating co-operation within groups of people

• Oil- keeps the economic and social wheels of society rolling smoothly

Social capital is like:

• a resource that people can accumulate and use to their benefit and advantage

• having a bank account of all of the networks in our lives

• The lack of social capital also can hold back people preventing them from achieving their goals which can affect the quality of life

Social capital is an important part of the learning process for

1) individuals

2) the training outcomes of a program

3) the community that we live in

• How do we know if we’ve got social capital?

• How do we measure it?

Conceptual development of the tool

• How robust is the concept of social capital?

• OECD: 2000-2012 – framework for social outcomes of learning

• Australian Bureau of Statistics (2004) – indicators for a social capital framework

• A concerted effort to empirically understand adult literacy and social capital outcomes in Scotland, the UK, Australia and New Zealand

Conceptual development

• Interest on Canadian soil

• Canadian Institute for Policy Research Initiatives; National Centre for Literacy

• Is the timing right for developing a measure

• Do you have access to a target population?

Conceptual development

• 1st Subscale: Network Qualities (NQ): including sub-concepts of trust levels, efficacy, diversity and inclusiveness

• 2nd Subscale: Network Structure (NS): including sub-concepts of size, communication mode and power relationships

• 3rd Subscale: Network Transactions (NTS): including sub-concepts of sharing support and sharing knowledge

• 4th Subscale: Network Types (NTY): including sub-concepts of bonding, bridging, and linking

• What are the size of your networks?• How do you

communicate in your networks?

• How do you share support?• How do you share

knowledge?

• What are your levels of trust?• What is your level

of self-confidence?

• How do you bond, bridge, and link with your networks?

4. Network Types

1. Network Qualities

2. Network Structures

3. Network Transactions

Lets talk about the reliability, validity, and utility of the Social Capital Inventory

Sample Items from SCISub-

scalesItem Strongly

Disagree

DisagreeNeither agree nor

disagree

Agree Strongly Agree M SD

1.NQ I have the self-confidence to influence things in my work and community life. 23.64 5.13

2. NS When I am in a problem situation I have a network consisting of family, friends, and neighbours.

22.85 4.84

3. NTr I believe I am a valued member of my group because of the support that I give in class.

23.10 5.19

4. NTy I believe the program has provided opportunities for me to interact with new outside groups and services.

22.97 5.03

Reliability

…refers to the stability or consistency of scores (over time, across raters, across items).

When a test is reliable, it provides dependable and consistent results.

Methods for Estimating Reliability

• Involves administering the same test to the same group of participants on two different occasions and then correlating the two sets of scores.

Test-Retest Reliability

• Involves two raters independently scoring the same set of tests and then correlating the two rater’s sets of scores.

Inter-rater Reliability

• Involves administering the test once to a single group of participants and determining if items on the test are correlated with one another (as indicated by Cronbach's alpha).

Internal Consistency

Reliability

Let’s have a look at the Social Capital Inventory’s reliability

To test SCI ’ s reliability internal consistency reliability was determined.

*The SCI was found to have adequate reliability.

Subscales Number of items Cronbach's alpha (α)

1. Network Qualities 6 .86

2. Network Structures 6 .81

3. Network Transactions 6 .88

4. Network Types 6 .88

Total Scale 24 .96

Is reliability good enough for a test to be used

A reliability coefficient only indicates whether the attribute measured by the test— whatever it is—is being assessed in a consistent way.

Whether the test is actually assessing what it was designed to measure is addressed by an analysis of the test's validity.

Validity

The term validity refers to whether or not the test measures what it claims to measure.

On a test with high validity the items will be closely linked to the test’s intended focus.

SCI Validity

Three major types:

Content validity

Construct validity

Criterion-related validity

Involves systematic examination of the test content to determine whether it covers a representative sample of the domain to be measured

Typically conducted by panel of subject matter experts

SCI Validity

Three major types:

Criterion-related validity

Content validity

Construct validity

Involves demonstrating that a measurement tool varies systematically with another measure of the same construct

But, requires the existence of another valid measure…

SCI Validity

Three major types:

Construct validity

Criterion-related validity

Content validity

Involves empirical testing of hypotheses about the construct, using the test as a measure of the construct

e.g., if we know that one group should have higher levels of construct X than another group, then comparison of the mean level of construct X (as measured by the test) for the two groups should reveal a significant difference

Ongoing research

- N=115 adult learners seeking literacy and essential skills in formal, non-formal, and informal learning settings- 1 Adult high school (Ontario)- 6 Workplace programs (4 in Manitoba, 2 in

Nova Scotia

- Mixed methods study (SCI, SDLRS, interviews)

Ongoing research - results

- Relatively high SCI means are consistent with qualitative data obtained from instructors and learners

- Those with average to above average scores on the SCI have significantly higher scores on the SDLRS

- Disproportionately higher percentage of females than males had average to above average scores on the SCI, if they do not belong to any clubs

Take-home Tips

1. Start with theory

-Read, read, read!

Take-home Tips

2. Seek feedback from Subject Matter Experts

-Teachers

-Colleagues

-Committee members?!

Take-home Tips

3. Pilot test

-Small, representative sample

-Use multiple raters (if applicable)

-Evaluate descriptive statistics (frequency distributions, means, SDs)

-Evaluate reliability (Cronbach alpha, item- total correlations, inter-rater agreement)

Take-home Tips

4. Revise, if necessary

-Delete unreliable items

-Clarify wording

-Modify scale

-Consider difficulty/discriminability

Take-home Tips

5. Include validity checks

-Collect demographic data

-Include other measures related to construct

-Compare with prior studies/theory

To Summarize:

• Read

• Listen to your committee

• Pilot test

• Revise

• Include validity checks