question discussion · study the scenario carefully before you consider the discussion questions....

Study the scenario carefully before you consider the discussion questions. Always unpack an assessment scenario in terms of the following:

The assessment context The school community

The concern raised in the assessment context Illegal drug use amongst learners at the school

The owner of the concern The school board and parent association

The consultant The psychologist who is responsible to address the concern

The intention of the consultant To develop and tailor prevention programmes to accommodate the needs and circumstances of the learners

The test user The psychologist who uses the questionnaire (In this scenario the consultant and the test user is the same person, namely the psychologist)

The intention of the test user To identify and target learners who are at risk for illegal drug use

The group targeted for assessment The learners at the school

The questionnaire The Alcohol use Intention Questionnaire (AIQ)

The intention of the questionnaire To determine the level of an individual’s intention to use alcohol

Question 1 is your statement that it is your own work.

Question Discussion

2 Lindi: I presume a good place to begin this discussion is the intention of the AIQ

Banti: Well, the intention of the questionnaire clearly is to

determine the level of an individual’s intention to use alcohol Carl: No, the questionnaire’s intention is to identify and target

learners who are at risk for illegal drug use Jan: Actually the intention of the questionnaire is to assess

learners at the school You agree most with, second most with and least with: Banti Carl Jan

Banit’s response is correct. She indicates the intention of the questionnaire correctly. Carl identifies the intention of the test user (the psychologist) as the intention of the questionnaire. This is not correct. Although Carl refers to a reason for using the test, in this case the test user’s intention is not identical to the intention of the questionnaire. But Jan’s response is worse than Carl’s. Jan’s interpretation of the intention of the questionnaire does not refer to a reason for using the questionnaire. He refers to the target group of the assessment exercise.

3 Gcina: Do you think the questionnaire is fit for purpose in light of the intention of the test user?

Jan: No I do not think so, because the intention of the test user is

to identify and target learners who are at risk for illegal drug use whereas the intention of the AIQ is to determine the level of an individual’s intention to use alcohol

Lindi: No, because the intention of the questionnaire is to identify and target learners who are at risk for illegal drug use which falls short of the intention of the test user namely to tailor prevention programmes to accommodate the needs and circumstances of these learners

Carl: I think the questionnaire is fit for purpose because there is a relationship between the level of an individual’s intention to use alcohol, which is the intention of the AIQ, and the individual’s risk for illegal drug use, which is what the test user wants to find out

You agree most with, second most with and least with: Carl Jan Lindi

Carl’s response is correct. He identifies the intention of the questionnaire and the intention of the test user correctly, and he draws attention to the relationship between the intention of the questionnaire and how the test user wants to use the results. This relationship was reported in subject literature. Jan offers the correct information about the intention of the questionnaire and the intention of the test user, but because these are different he concludes that the test user cannot use questionnaire as he/she intends to do. In other words, Jan forgets (or does not know) about the subject literature that Carl refers to. Lindi’s response is incorrect. She not only confuses the intention of the questionnaire and the intention of the test user (i.e. what the psychologist wants to find out), she also confuses the intention of the test user and the intention of the consultant (i.e. how the psychologist wants to react to the concern that sparked the assessment).

4 Gcina Do you think the questionnaire can help to address the concern of the school board and parent association?

Carl: Yes, because we can use the AIQ to identify and target

learners who are at risk for illegal drug use Lindi: Yes, but then we have to know how to tailor prevention

programmes to accommodate the needs and circumstances of these learners after we identified and targeted learners who are at risk for illegal drug use

Banti: No I disagree, because the intention of the AIQ is to determine the level of an individual’s intention to use alcohol, which does not mean we can identify and target learners who are at risk for illegal drug use

You agree most with, second most with and least with: Lindi Carl Banti

Lindi’s response is most correct. She understands that the result of the questionnaire can be used to identify and target learners who are at risk of illegal drug use. But she also realises that this result does not automatically lead to prevention programmes. One has to know how to tailor such programmes to accommodate the needs and circumstances of the identified learners. Carl’s response is correct, but he does not consider the fact the assessment result does not automatically result in implemented. One has to know how to implement the result (as Lindi indicates). Banti’s realises that that there is a difference between the intention of the questionnaire and the intention of the test user, but she forgets (or does not know) about the relationship between the intention to use alcohol and drug abuse reported in subject literature. Thus she comes to an incorrect conclusion.

5

Gcina Do you think the AIQ is the correct type of questionnaire for our assessment purpose?

Jan: Yes, I think so. the AIQ is a survey type questionnaire and we

need a survey type questionnaire to gather information that allows us to identify and target learners who are at risk for illegal drug use

Lindi: No, the AIQ is not a survey type questionnaire, and it is not the type of questionnaire that is suitable to identify and target learners who are at risk for illegal drug use

Carl: Actually the AIQ is a survey type questionnaire, and it is suitable to assess learners at the school

You agree most with, second most with and least with: Jan Carl Lindi

The questionnaire is an example of a survey type questionnaire, and a survey type questionnaire is the correct type of questionnaire to use to determine how people perceive and feel about matters. Thus Jan’s response is most correct. Carl’s response is half correct and half incorrect. He is correct in identifying the AIQ as a survey type questionnaire, but he is incorrect to identify the assessment of learners at the school as the purpose of the assessment. He confuses the group to be assessed with the purpose of the assessment. Lindi’s response is incorrect on two accounts. She wrongly indicates that the AIQ is not a survey type questionnaire, and she is incorrect when she says that the AIQ cannot be used to identify learners who are at risk of illegal drug use.

6

Jan: Given the intention of the AIQ, what is its content domain? Carl: I would say the content domain of the AIQ is learners at the

school Banti: I think the content domain of the AIQ is attitude towards

using alcohol Lindi: No, I think the content domain of the AIQ is the psychological

factors associated with the intention to use alcohol You agree most with, second most with and least with: Lindi Banti Carl

The content domain is the area defined by the psychological construct that underlies the questionnaire. The intention of the AIQ is to determine the level of a person’s intention to use alcohol. Therefore Lindi’s response is most correct. The content domain of the AIQ (given its intention) is the psychological factors associated with the intention to use alcohol. Banti’s response is correct but incomplete. Attitude towards using alcohol is part (but not the whole story) of the intention to use alcohol. Carl’s response is incorrect. The group targeted for assessment is not the content domain of the questionnaire.

7 Banti: I am concerned about the content domain of the AIQ in light of the intention of the AIQ

Jan: I do not think there is need for concern. The questionnaire’s

content domain is theoretically determined as attitude towards using alcohol, subjective norm concerning the use of alcohol, and perceived control over using alcohol

Gcina: No I think there is need for concern. The questionnaire’s content domain is over determined. Perceived control over the use of alcohol should be deleted from the questionnaire’s content domain

The intention of the AIQ is to determine the intention to use alcohol. The intention to use alcohol is theoretically determined by the theory of planned behaviour. The intention to execute behaviour involves three factors as indicated in the scenario. Therefore there is no concern about the content domain of the AIQ. One could be concerned about the AIQ itself though because it only measures two of the three factors, namely attitude towards alcohol consumption and subjective norm concerning the use of alcohol. Biographical information and recreational behaviour are additional components of the AIQ. They may be included for other reasons, but they do not tap into the content domain associated with intention to use alcohol (which the questionnaire intends to measure). Therefore Jan’s response is correct. Gcina’s response is incorrect. He wants to limit the questionnaire’s content domain to the factors included in the questionnaire. This is not correct because the questionnaire is

Lindi: I agree, Gcina, but I want to add that perceived control over use of alcohol should be replace by biographical information and recreational behaviour

You agree most with, second most with and least with: Jan Gcina Lindi

determined by the content domain. The questionnaire does not determine the content domain. Lindi’s response is even worse because she wants to add factors to the content domain (biographical information and recreational behaviour) that do not concern the intention to use alcohol.

8 Carl: What is the scope of the AIQ? Banti: The scope of the questionnaire is the psychological factors

associated with the intention to use alcohol Jan: No, I think the scope is biographical information, attitude

towards alcohol consumption, subjective norm concerning the use of alcohol, recreational behaviour

Lindi: Actually the scope is attitude towards alcohol consumption and subjective norm concerning the use of alcohol

You agree most with, second most with and least with: Jan Lindi Banti

Jan’s response is most correct. The scope of the questionnaire refers to what the questionnaire measures, which are the factors he indicates in his response. Lindi’s response is partially correct, but incomplete. She limits the scope of the questionnaire to the content domain factors included in the questionnaire. The scope of the AIQ is broader than the factors indicated by Lindi. Banti’s response is incorrect. She identifies the content domain as the scope of the questionnaire. Note that a questionnaire should assess all the factors in its content domain, but the questionnaire is not necessarily limited to the factors in the content domain. It may include additional factors as for, example, biographical information.

9 Lindi: I am not convinced that the scope of the AIQ is sufficient given its intention.

Banti: I share your concern, Lindi. The items included in the AIQ are

not sufficient to determine the level of an individual’s intention to use alcohol

Jan: No, I am not concerned about the scope of the AIQ. It has sufficient scope to assess all learners at the school

Carl: Actually the scope of the AIQ is more than sufficient because it includes biographical information and recreational behaviour

You agree most with, second most with and least with: Banti Carl Jan

Banti’s response is correct. The items included in the AIQ are not sufficient because the AIQ does not include items to assess perceived control with regard to the use of alcohol, which is one of the factors of the intention to use alcohol. Carl’s response is incorrect because he misses the point that despite its broader scope (also including biographical information and recreational behaviour) the AIQ assesses only two of the three content domain factors. Jan’s response is really poor. The scope of the questionnaire is not defined by the group targeted for assessment.

10 Banti: What psychological construct is assessed by the AIQ? Gcina: learners at the school Lindi: intention to execute behaviour Carl: attitude towards using alcohol You agree most with, second most with and least with: Lindi Carl Gcina

Lindi’s response is most correct. The psychological construct is intention to execute behaviour. Carl’s response does not refer to the construct itself but to the application of the construct in the explanation of a particular behaviour, namely the intention to use of alcohol. Gcina’s response is absolutely incorrect. The group targeted for assessment certainly is not a psychological construct.

11 Carl: Which items of the AIQ do not contribute to the intention of the questionnaire?

Jan: All items concerned with biographical information and with

recreational behaviour Gcina: All items concerned with attitude towards alcohol

consumption and with recreational behaviour Lindi: All items concerned with attitude towards alcohol

consumption and with subjective norm concerning the use of alcohol

You agree most with, second most with and least with: Jan Gcina Lindi

Jan’s response is most correct. Biographical information and recreational behaviour are not content domain factors. Therefore items assessing these aspects do not contribute to assessment of the intention to use alcohol. (Note that this does not mean that these items may not be included in the AIQ. There may be other reasons for their inclusion). Gcina’s response is partially correct, but also contains an incorrect component. Items concerning recreational behaviour do not contribute to the assessment of intention to use alcohol. However, items concerning attitude towards alcohol consumption do contribute to the assessment of intention to use alcohol. Lindi’s response is incorrect. The two factors she refers to are factors that are part of the intention to use alcohol.

12 Gcina: People who complete the AIQ are informed that by answering the questions they indicate how they feel about the use of alcohol. Should this information be included in the questionnaire instructions?

Carl: It should be included because it refers to the reason for

completing the questionnaire Lindi: II think it should not be included because people who

complete the questionnaire should not be informed that by answering the questions they indicate how they feel about the use of alcohol

Jan: It can be included, but it is not necessary. The only information that should be included is to read all questions carefully and to select the answer that best describes the respondent’s thoughts and feelings

The instructions of a questionnaire should indicate to respondents why they complete the questionnaire. In this case it is to indicate how they feel about the use of alcohol. Thus Carl’s response is correct. Jan is correct in indicating that respondents should also be told how to complete the questionnaire, but he is incorrect when he suggests that it is optional to provide the reason for completing the questionnaire. Lindi offers a poor response. It is not correct to suggest that respondents should not be informed why they complete a questionnaire.

You agree most with, second most with and least with: Carl Jan Lindi

13 Lindi: Do you think the instructions of the AIQ encourage respondents to provide truthful answers?

Banti: Yes it does, because respondents are informed to read all

questions carefully and to select the answer that best describes the respondent’s thoughts and feelings

Gcina: No it does not, because they are informed that the school psychologist has access to the respondent’s answers

Carl: Yes it does, because they are informed that there are no right or wrong answers and that their opinion counts

You agree most with, second most with and least with: Carl Gcina Banti

Respondents are encouraged to provide truthful answers when they are told that there are no right or wrong answers and that their opinion counts. Carl’s response is most correct. Gcina’s response is incorrect. The fact that respondents are told that only the school psychologist has access to their responses should assure them that their responses will be treated confidentially, and therefore they should be more inclined to respond truthfully. Banti confuses truthfulness with accuracy. Telling respondents how to complete a questionnaire increases the accuracy because it minimises erroneous responses, but accuracy does not mean the respondents are truthful in their responses.

14 Gcina: Does the indication to the AIQ respondents that only the school psychologist has access to the respondent’s answers assure anonymity?

Lindi: No, it does not assure anonymity Banti: It is an assurance of confidentiality but not anonymity Carl: It is not an assurance of any kind. It merely encourages

respondents to complete the AIQ You agree most with, second most with and least with: Banti Lindi Carl

This part of the instruction is an indication that respondents’ information will not be available to anybody. It will only be seen by the psychologist. As such it is an assurance of confidentiality. But confidentiality should not be confused with anonymity. Confidentiality does not mean that the psychologist cannot identify the individual respondents. It simply means that he/she will not divulge the identities of the respondents. Both Banti and Lindi’s responses are correct, but Banti offers a better response than Lindi because she offers more information. She refers to confidentiality and anonymity, whereas Lindi only refers to anonymity. Carl’s response is incorrect. Contrary to his belief the instruction is an assurance rather than an encouragement to complete the AIQ.

15 Banti: We have to make sure the AIQ contains different types of questions. So I think we should identify some of the question types. For example, what kind of question is 2.8?

Carl: It is a rating scale item Gcina: It is a Likert scale item Lindi: It is semantic differential item

The item is a rating scale, and more precisely a Likert type rating scale. Gcina’s response is more correct than Carl’s because he indicates the exact nature of the item, whereas Carl merely indicates the more general nature of the item. Lindi’s response is incorrect. The item is not a semantic differential.

You agree most with, second most with and least with: Gcina Carl Lindi

16 Banti: What kind of question is 4.5? Carl: It is an open fill-in question Gcina: No, it is a closed fill-in question Lindi: I think it is a closed true/false question You agree most with, second most with and least with: Carl Gcina Lindi

The question is fill-in question type. In this regard Carl and Gcina are correct. But on the issue of it being an open or closed question, Carl is correct and Gcina incorrect. Lindi offers a poor response. The question is not a closed true/false question.

17 Banti: But what is the difference between questions 3.6 and 3.5? Carl: There is no difference. They both are checklist question types Gcina: No, I think 3.6 is a checklist question type and 3.5 is a MCQ

question type Lindi: Actually 3.6 is a MCQ question type, whereas 3.5 is an

inventory question type You agree most with, second most with and least with: Lindi Carl Gcina

Lindi is most correct. 3.6 is a MCQ type question and 3-5 is a checklist (also called an inventory) question type. A MCQ requires respondents to select one of the alternative responses. A checklist allows respondents to select one or more than one response. Carl’s response is partially correct because one of the questions is a checklist. However Gcina’s response is incorrect with regard to both items.

18

Banti: We need a semantic differential type question. Gcina: Then question 3.8 would be a good choice Lindi No, question 2.9 would be a better choice than 3.8 Carl: I disagree, Lindi. You cannot use question 2.9. it is Likert type

rating item, not a semantic differential type You agree most with, second most with and least with: Gcina Carl Lindi

Question 3.8 is a semantic differential question type. Thus Gcina’s response is most correct. Lindi’s response is incorrect. She confuses a Likert type rating item with a semantic differential question type. Carl’s response is correct. He correctly identifies question 2.9 as a Likert type question, but he does not suggest a semantic differential question type or indicate whether or not he agrees that 3.8 is a semantic differential question type.

19 Lindi: Table 4 shows 10 respondents’ responses to item 2.10. What do these responses tell us about the respondents’ attitudes towards alcohol consumption?

Gcina: The respondents’ responses are skewed towards the

disagree side of the scale

The responses to this item are mostly 1 or 2. Most respondents disagree or disagree strongly with the statement of the item, namely that one should not drink and drive. Thus most respondents think one can drink and drive, which reflects a positive attitude towards the use of alcohol. Both Jan and Gcina’s responses are correct, but Jan’s response is better than Gcina’s. Gcina simply indicates the obvious fact the responses are skewed towards the disagree side of the scale. Jan’s response indicates what the skewed responses mean in terms

Jan: The respondents express a positive attitude towards alcohol consumption

Banti: I think the respondents were indecisive about alcohol consumption

You agree most with, second most with and least with: Jan Gcina Banti

of what the item measures, namely attitude towards the use of alcohol. Banti’s response is incorrect. The group does not show indecisiveness because their responses are clustered towards the disagree side of the scale. If there responses were evenly spread across the scale one could say that the group’s response was indecisive.

So far our discussion was fairly general. But I think we should discuss the psychometric properties of the questionnaire and the questions in more detail. To begin with, how good is your knowledge about aspects such as validity and reliability?

20 Jan: So far our discussion was fairly general. But I think we should discuss the psychometric properties of the questionnaire and the questions in more detail. To begin with, how good is your knowledge about aspects such as validity and reliability? Gcina: A questionnaire is valid if it actually measures what it is

supposed to measure, and it is reliable if it provides consistent measures of whatever it measures

Lindi: I agree that a questionnaire is valid if it measures what it is supposed to measure, but reliability is not about consistency. A questionnaire is reliable if it is administered to the people it is supposed to be administered to

Carl: I do not agree with your understanding of validity. A questionnaire is valid if its questions are properly formulated. But I do agree that a questionnaire is reliable if it is administered to the correct people

You agree most with, second most with and least with: Gcina Lindi Carl

Gcina’s response is correct. He knows what the difference is between validity and reliability. Lindi knows what validity is, but she misunderstands reliability. Carl clearly has no idea of validity and reliability.

21 Jan: Ok, but I still not sure about validity and reliability when we talk about an actual questionnaire such as the AIQ

Lindi: The AIQ is valid if it really determines the level of an

individual’s intention to use alcohol, and it is reliable if it

Lindi’s offers the best response. She understands the meaning of validity and reliability and she is able to apply her knowledge. She identifies the intention of the questionnaire correctly, which is to determine the level of an individual’s intention to use alcohol. The AIQ is valid if it really measures an individual’s intention to use alcohol, and it is reliable if these measures are consistent. In his response Gcina refers to the intention of the test user, namely to identify

provides consistent measures of an individual’s intention to use alcohol

Gcina: The AIQ is valid if it identifies and targets learners who are at risk for illegal drug use, and it is reliable if it is used to measure all learners at the school

Banti: The AIQ is valid when there is a high correlation between its even items and its odd items, and it is reliable if it is used to assess all learners at the school

You agree most with, second most with and least with: Lindi Gcina Banti

and target learners who are at risk of illegal drug use. Although this is not a direct indication of the validity of the questionnaire the questionnaire has to be valid (i.e. provide a true measurement of the intention to use alcohol) in order for it to be used to identify and target learners who are at risk of illegal drug use. However, Gcina clearly does not know what the reliability of the AIQ refers to. Banti offers a similar response with regard to reliability, which means she is wrong in this regard. However, the first part of her response is also incorrect. Here she refers to the split-half reliability of the questionnaire, not to its validity.

22 Gcina: What is the difference between the information provided in Table 2 and in Table 3?

Carl: Table 2 shows the validity of the AIQ and Table 3 shows its

reliability Lindi: I think both tables provide information about the reliability

of the AIQ Banti: Actually Table 2 is about the AIQ’s internal inconsistency and

Table 3 is about its test-retest reliability You agree most with, second most with and least with: Banti Lindi Carl

Lindi is correct. Both tables are about reliability. The correlation between two halves of the questionnaire is a measure of internal consistency, which is a form of reliability, and the correlation between two applications of the questionnaire refers to its test-retest reliability. Banti’s response is better than Lindi’s because she correctly indicates specific information about reliability, which Lindi does not do. Carl’s offers the least correct response. He is correct about Table 3 showing reliability, but he is incorrect about Table 2 being about validity.

23 Carl: What does Table 2 in the manual tell us about the AIQ? Jan: The table shows that the AIQ was split into two halves, and it

indicates the total scores of five respondents for each half Lindi: The table shows a high correlation between the two halves

of the AIQ Banti: The table shows that the AIQ has a high degree of internal

consistency You agree most with, second most with and least with: Banti Lindi Jan

All three responses are correct, but they differ in terms of the value of the information they provide. Jan merely observes and describes the information provided in the table. Lindi adds information about the relationship between the two sets of values, namely that these values are positively correlated. Banti adds a further insight. She understands that the correlation means a high degree of internal consistency.

24 Gcina: I am unsure about correlations. How do you see whether there is a correlation between numbers?

Carl: Remember, correlation is between two groups of numbers.

There is a correlation if you can split the numbers into two groups

Lindi: You can see there is a correlation when high numbers in one group correspond to high numbers in the other group

Jan: And, there is no correlation when high numbers in one group correspond to low numbers in the other group

You agree most with, second most with and least with: Lindi Jan Carl

Carl is correct in suggesting that correlation is about the relationship between two groups of numbers, but he clearly does not understand what correlation really means. The fact that a set of numbers can be split into two groups does not mean that the two groups are related. Lindi offers the best response. She understands that the numbers in the two groups need to be paired. Correlation is about pairs of numbers. One member of each pair comes from one group and the other member of the pair comes from the other group. There is a correlation if the numbers in each pair has more or less similar values. Or, as Lindi puts it, if one member of a pair has a high value the other member’s value should also be high. There is an inverse correlation (i.e. a negative correlation) if in each pair of numbers one number is high and the other is low. This is what Jan refers to, but unfortunately Jan thinks this means no correlation. He is incorrect. What he refers to is a high but negative correlation. No correlation would be when there is no pattern in the values of the members of each pair of numbers. In other words, when there is no clear indication that high numbers in one group are paired with high numbers in the other group, or that high numbers are paired with low numbers.

25 Gcina: The other table in the manual I am unsure about is Table 3. What does the information in this table tell us about the AIQ?

Banti: The information in the table shows a poor correlation

between the two sets of scores, which means the AIQ did not demonstrate test-retest reliability

Jan: We should keep in mind that the poor correlation between the two sets of values refers to the criterion-related validity of the AIQ and not to the reliability of the AIQ

Carl: Actually the information in Table 2 does not reflect a poor correlation, which means the AIQ shows good criterion-related validity

You agree most with, second most with and least with: Banti Jan Carl

Table 3 shows a poor correlation between the two groups of scores. This correlation refers to two administrations of the questionnaire. As such it refers to the questionnaire’s test-retest reliability. A poor correlation means the test-retest reliability is low. Banti understands all of this. and offers the best response. Jan understands that the two groups of scores are poorly correlated, but he thinks the relationship between the two administrations of the questionnaire indicates criterion-related validity. This is incorrect. Carl really does not know what he is talking about. The correlation is poor, not good. And, it does not refer to criterion-related validity.

26 Gcina: So, if you compare the information from Table 2 with the information from Table 3, what do you think?

Carl: The AIQ has high test-retest reliability but its criterion

related validity is questionable Lindi: The items of the AIQ seem to measure the same thing, but

the questionnaire is vulnerable to external influences

Table 2 shows high internal consistency, and Table 3 shows low test-retest reliability. Lindi understand this, and she interprets the implication of this information correctly. Internal consistency refers to the items in the questionnaire measuring the same thing. This means the questionnaire is solid. Its items are bound together tightly. But when this solid questionnaire is administered on two separate occasions the scores correlate poorly. This means factors external to the questionnaire influence the scores. Banti and Carl do not demonstrate Lindi’s insight. Banti’s response holds some value because one of the two tables

Banti: The information obtained from these two tables supports the reliability of the AIQ

You agree most with, second most with and least with: Lindi Banti Carl

(but not both) supports the reliability of the questionnaire. Carl’s response is totally incorrect. The test-retest reliability is low, not high, and the none of the two tables provide information about the criterion-related validity of the questionnaire.

27 Jan: Will you keep item 2.3 in the AIQ or delete it? Banti: I will delete item 2.3 because it has low item-total correlation

and low item variance Lindi: I think the item shows high item variance, but one cannot

keep an item that has low item-total correlation Carl: I disagree. I will keep the item because it has low item

variance and high item-total correlation You agree most with, second most with and least with: Lindi Banti Carl

Item 2.3 has high item variance (the scores are distributed across the measurement scale), but low correlation (no clear pattern of the relationship between the item scores and the total scores). An item should have high variance and high correlation. Lindi’s response is correct. One cannot keep the item in the test because it has low item-total correlation. Banti’s is correct in suggesting that the item should be deleted from the questionnaire, but her reasons for doing so are only partially correct. The item does not show low item variance. Carl’s response is totally incorrect.

28 Lindi: What will you do about item 2.6 in the AIQ? Banti: I will keep the item in the AIQ because the negative

correlation indicates an inverse discrimination value Gcina: I will delete the item from the AIQ because the negative

correlation means a problem with the discrimination value of the item

Carl: I am not sure whether I will keep or delete the item. I will first check how the item was scored before I decide

You agree most with, second most with and least with: Carl Gcina Banti

Item 2.6 has high item variance and high negative item-total correlation. Gcina is correct in stating that the item should be deleted because the negative item-total correlation indicates a problem with the item’s discrimination value. Banti is incorrect because she should not keep the item. She also refers to an ‘inverse discrimination value’. Discrimination means differentiation between the respondents. An item can differentiate or not differentiate. There is no inverse differentiation. However, Carl provides the best response. He realises that the negative item-correlation may be because the item was scored incorrectly. An item that requires inversed scoring but is not scored as such may show an inverse relationship when the relationship should be positive. Therefore one should always check the scoring of an item with strong negative item-total correlation before one decides whether or not to delete the item. However, if its scoring is in order a negative correlating item should be deleted.

29 Carl: So, to summarise, which of the items in Table 4 we keep and which should we delete? Jan: We should keep item 2.8 and delete the others Lindi: We should keep item 2.8 and item 2.6 and delete the others Banti: We should keep 2.3 and delete the others

For the clever students: When answering this question simply consider the information provided in Table 4 and in the corresponding figures. Do not refer back to the original formulation of the item in the AIQ. Doing so would greatly complicate matters and may change the answers you have to provide. One would keep items that has positive item total correlation and high item variance. Only item 2.8 qualifies. The rest should be deleted. Therefore Jan’s response is most correct. Lindi is correct in identifying 2.8 as an item to be kept, but she is not correct in wanting to keep item 2.6. Banti’s response is totally incorrect.

You agree most with, second most with and least with: Jan Lindi Banti

30 Gcina: Given that we want to determine the level of an individual’s intention to use alcohol in order to identify and target learners who are at risk for illegal drug us, how should we think about the reliability of the AIQ?

Carl: It is important to make sure that the AIQ has high internal

consistency Lindi: I think test-retest reliability is more important than internal

consistency, given that we want to identify and target learners who are at risk for illegal drug use

Banti: I think criterion-related validity is the most important aspect of the AIQ’s reliability given our need to identify and target learners who are at risk for illegal drug use

You agree most with, second most with and least with: Carl Lindi Banti

In this scenario internal consistency is more important than test-retest reliability, because the questionnaire needs to be administered once only and one wants to be sure that the questionnaire provides a reliable result. Carl’s response is most correct. Lindi’s response is not correct. It looks like she associates the identification of learners at risk for illegal drug use as a second administration of the questionnaire, which it is not. Banti refers to validity and not to reliability. But Banti seems to be really confused. She indicates that validity is an aspect of reliability, which means she has no understanding of the difference between validity and reliability. Banti’s response is totally incorrect.

31 1. (0 point) Lindi: I presume the same holds for validity. Given how we want to

use the AIQ, how should we think about the validity of the questionnaire?

Banti: Because we want to determine the level of an individual’s

intention to use alcohol I think concurrent validity is the most important type of validity to consider

Carl: Given that we want to identify and target learners who are at risk for illegal drug use I think it is important to consider the AIQ’s criterion related validity

Jan: If we want to determine the level of an individual’s intention to use alcohol in order to identify and target learners who are at risk for illegal drug use I think we should consider the AIQ’s construct validity in addition to its criterion related validity

Jan offers the best response. He indicates that two kinds of validity, namely criterion related validity and construct validity are important given how the AIQ is going to be used in the present scenario. Subject literature suggests a relationship between the intention to use alcohol and future illegal drug use. Criterion related validity is important because the result of the questionnaire will be used to identify and target learners who may become illegal drug users. The AIQ If the result of the AIQ does not correlate with future illegal drug use the AIQ cannot be used to identify and target such learners. Construct validity is important because the AIQ needs to measure the construct that it is supposed to measure, namely the intention to use alcohol. If it does not measure this construct one cannot expect a relationship between the result of the AIQ and the future illegal drug use. Carl’s response is correct, but less valuable than Jan’s because he only refers to criterion related validity. Banti’s response is not correct because she sees the result of the AIQ (the level of intention to use alcohol) as a criterion score. Concurrent validity is a form of criterion related validity, and criterion related validity refers to the correlation between the result of a questionnaire and some measure of a criterion. Concurrent validity simply means the criterion scores coincide with the questionnaire scores.

You agree most with, second most with and least with: Jan Carl Bant

32 2. (0 point) Gcina: Considering the information provided in Table 5, what do

you think of the construct validity of the AIQ? Lindi: The high correlation between attitude towards alcohol

consumption and subjective norm concerning the use of alcohol supports the construct validity of the AIQ

Carl: The low correlation between the recreational behaviour and attitude towards alcohol consumption and subjective norm concerning the use of alcohol shows low construct validity of the AIQ

Jan: The information in Table 5 supports the construct validity of the AIQ , but the AIQ does not fully represent the content domain of the construct it purports to assess

You agree most with, second most with and least with: Jan Lindi Carl

Table 5 indicates the inter-correlations between the four factors of the AIQ. There is a high correlation (0.92) between the attitude factor and the norm factor, which means a strong relationship between these two factors. One would expect this to be the case as these two factors are part of the psychological construct measured by the AIQ, namely intention to execute behaviour. The biographical and recreation factors are not part of the psychological construct measured by the AIQ. They show low correlations with both the attitude and the norm constructs. Therefore the information in Table 5 supports the construct validity of the AIQ. But there is one quite serious problem with the construct validity of the AIQ namely that one of the factors of the intention to execute behaviour (the ‘perceived control’ factor) is not measured by the AIQ. In light of this discussion it is clear that Jan’s as well as Lindi’s responses are correct. But Jan’s response is better than Lindi’s because he realises that the AIQ does not include all factors of the content domain implicated by the ‘intention to execute behaviour’ construct. Carl’s response is incorrect. The low correlations between recreational behaviour and the two factors of the ‘intention to execute behaviour’ construct does not mean low construct validity.

33 3. (0 point) Gcina: Let us have a look at the convergent and discriminant validity

of the AIQ. What is the difference between these two kinds of validities?

Carl: Convergent and discriminant validity are two aspects of

construct validity Lindi: Convergent validity implies a high correlation between the

AIQ and another questionnaire, whereas discriminant validity means a low correlation between the AIQ and another questionnaire

Banti: I agree that convergent validity refers to the relationship between the AIQ and another questionnaire but discriminant validity is not a form of validity. It is a measure of reliability

Lindi’s offers the best response. Carl’s response is not incorrect, but note that he does not indicate how the two kinds of validities differ. He simply indicates that they are two aspects of construct validity. Banti’s response contains inaccuracies. Firstly, both convergent and discriminant validity (and not only convergent validity) refer to the relationship between the AIQ and other questionnaires, and secondly, it is not correct to state that discriminant validity is a measure of reliability.

You agree most with, second most with and least with: Lindi Carl Banti

34 4. (0 point) Carl: In Table 6, what does the correlation between the AIQ and

the AAQ mean? Lindi: It means there is a poor relationship between the AIQ and

the AAQ Gcina: It indicates a high degree of discrimination between the

measurements of the AIQ and the AAQ Banti: It means the AAQ should be grouped with the TSR rather

than with the AIQ You agree most with, second most with and least with: Gcina Lindi Banti

Both Gcina’s and Lindi’s responses are correct. However, whereas Lindi merely observes the poor correlation (0.04) between the two questionnaires, Gcina interprets the information correctly as an indication of discrimination between the two measures. Therefore Gcina’s offers a more valuable response than Lindi does. It is difficult to see what Banti is on about. It is not clear how the low correlation between the AAQ and the AIQ indicates that the AAQ should be grouped with the TSR, with which it has an equally poor correlation (0.08). It is also not clear what she means by ‘grouped with’. Therefore Banti’s response is really poor.

35 5. (0 point) Banti: I am wondering about the meaning of the correlation

between the TSR and the AIQ indicated in Table 6. Banti: It indicates a high correlation between TSR and the AIQ Gcina: It shows convergence between the measurements of the TSR

and the AIQ Lindi: I think it supports convergent as well as concurrent validity

for the AIQ You agree most with, second most with and least with: Lindi Gcina Banti

Although all three responses are correct they differ in quality. Banti observes the most basic fact, namely that the correlation is high (0.86). Gcina interprets the high correlation as an indication of convergence between the two measures. But the best response comes from Lindi. She realises that the TSR (a treatment suitability rating) is a criterion measure for the AIQ result. The AIQ provides a measure of the individual’s intention to use alcohol. If people in a treatment setting have high scores on the AIQ and they are rated suitable for treatment (the TSR result) it means the AIQ identifies those who end up in treatment. Note that the AIQ and the TSR results are obtained during the treatment process. Therefore one cannot consider the AIQ results as predictions of the TSR results. This is why Lindi indicates the criterion related validity as concurrent validity and not as predictive validity.

36 6. (0 point) Carl: To summarise: Does the information in Table 6 tell us

anything about the construct validity of the AIQ?

Gcina’s response is incorrect. The table does not provide information about the internal consistency of the AIQ. Banti’s response is only partly correct. The table indicates good convergent validity and good discriminant validity. Lindi offers the best response. The table

Gcina: The information in Table 6 shows good internal consistency

for the AIQ Banti: I think Table 6 indicates good convergent but poor

discriminant validity for the AIQ Lindi: Actually Table 6 shows good construct validity for the AIQ You agree most with, second most with and least with: Lindi Banti Gcina

indicates good construct validity for the AIQ.

This is a fruitful discussion, but I think we should move on now. We have to decide whether we can use the AIQ, which means we have to rate some aspects of the questionnaire and the manual. In each case we have to consider specific criteria to help us decide on a rating. We can express our rating on a scale of 1 to 4, where 1 reflects a bad rating and 4 is a good rating.

37 This is a fruitful discussion, but I think we should move on now. We have to decide whether we can use the AIQ, which means we have to rate some aspects of the questionnaire and the manual. In each case we have to consider specific criteria to help us decide on a rating. We can express our rating on a scale of 1 to 4, where 1 reflects a bad rating and 4 is a good rating. Rate the instructions of the AIQ The instructions of the AIQ are complete if: a the intention of the AIQ is indicated b the matter of confidentiality is addressed c instructions for completing the AIQ are indicated Rate 1 if: none of a, b, c Rate 2 if: a Rate 3 if: a + b Rate 4 if: a + b + c

Rate 4 Criterion a (the intention of the AIQ is indicated) is met: “By answering these questions you indicate how you feel about the use of alcohol” Criterion b (the matter of confidentiality is addressed) is met: “Only the school psychologist has access to your answers” Criterion c (instructions for completing the AIQ are indicated) is met: “Please read the following questions carefully and provide the answer that best describes your thoughts and feelings” and also: “There are no right or wrong answers. Simply give your opinion. Be as honest as possible.”

38

Rate the formulation of question 1.3 of the AIQ Question 1.3 is formulated correctly if: a it has content and construct relevance b it is clear and unambiguous c it is configured correctly d it is not leading e it is not likely to elicit a socially desirable response Rate 1 if: none of the following Rate 2 if: a Rate 3 if: a + b + c Rate 4 if: a + b + c + d + e

Rate 2 Criterion a is met: The purpose of the item is to get biographical information Criterion b is met: The item is straightforward to understand Criterion c is not met: There is a problem with the configuration of the item. It is a filter question but its filtering is set incorrectly. It requires individuals who should answer questions 1.4 and 1.5 to jump to Section 2. Criterion d is met: The question is not leading Criterion e is met: The question is not likely to elicit a socially desirable response Note: Because criterion c is not met one cannot assign a rating of 3 or 4. Thus the highest rating that can be assigned is 2.

39

Rate the suitability of question 2.4 in the AIQ Question 2.4 is suitable if: a respondents are likely to find the question’s level of

language appropriate b respondents are likely to find the question relevant in

their context c respondents are likely to possess the knowledge

required to respond correctly to the question Rate 1 if: none of the following Rate 2 if: a Rate 3 if: a + b Rate 4 if: a + b + c

Rate 2 Criterion a is met: The language level is appropriate Criterion b is not met: School level learners are most likely not married or living with a partner, and will not find the item relevant in their context Criterion c is not met: School level learners are most likely not married or living with a partner, and will not possess the knowledge required to respond to the question Note: Because criterion b is not met one cannot assign a rating of 3 or 4. Thus the highest rating that can be assigned is 2.

40

Rate the formulation of question 2.9 Question 2.9 is formulated correctly if: a it has content and construct relevance b it is clear and unambiguous

Rate 2 Criterion a is met: The item has relevant content and is appropriate with regard to the construct it aims to measure Criterion b is met: The item is clear and easy to understand

c it is configured correctly d it is not leading e it is not likely to elicit a socially desirable response Rate 1 if: none of the following Rate 2 if: a Rate 3 if: a + b + c Rate 4 if: a + b + c + d + e

Criterion c is not met: The item refers to multiple matters that may require different responses. Yet the item allows only one response. Criterion d is met: The item does not encourages the respondent to respond in a particular way Criterion e is met: The item does not involve a socially desirable matter

41





Rate 1 Criterion a is not met: The language level is too advanced. Only a professionally trained person knowing about psychodynamics and alcohol abuse is able to respond to this question. Criterion b is not met: School going children are not likely to understand the question and see how it relates to their experiences Criterion c is not met: It is unlikely that anyone who is not professionally trained will have the knowledge to respond correctly to this question. Note: Because criteria a, b and c are not met one cannot assign a rating of 2, 3 or 4. Thus the highest rating that can be assigned is 1.

42

Rate the formulation of question 3.2 Question 3.2 is formulated correctly if: a it has content and construct relevance b it is clear and unambiguous c it is configured correctly d it is not leading e it is not likely to elicit a socially desirable response

Rate 2 Criterion a is met: The item has relevant content and is appropriate with regard to the construct it aims to measure Criterion b is not met: It is unclear what the differences are between maybe, perhaps and sometimes. It is also not clear whether one may tick more than one response. Criterion c is not met: The item is not configured correctly because the labels used to distinguish the different categories of the item (e.g. maybe, perhaps) do not clearly separate the categories from each other.

Rate 1 if: none of the following Rate 2 if: a Rate 3 if: a + b + c Rate 4 if: a + b + c + d + e

Criterion d is met: The item does not encourages the respondent to respond in a particular way Criterion e is met: The item does not involve a socially desirable matter

43





Rate 1 Criterion a is not met: The average person in the street may not be able to assign the correct meaning to the terms used in the item Criterion b is not met: The question refers to symptoms of advanced alcoholism. School going children are not likely to understand the question and see how it relates to their experiences Criterion c is not met: It is unlikely that anyone who is not professionally trained will have the knowledge to respond correctly to this question.

44

Rate the functionality of the AIQ The AIQ is functional if: a the information gathered by the AIQ covers the

intended content domain b the AIQ produces information that is suitable for the

intended target population c the AIQ produces information that is practical to use Rate 1 if: none of the following Rate 2 if: a Rate 3 if: a + b Rate 4 if: a + b + c

Rate 1 Criterion a is not met: The AIQ intends to measure the intention to use alcohol. But the AIQ does not include all the factors associated with the intention to execute behaviour. Therefore the information gathered by the AIQ does not cover the intended content domain. Criterion b is met: The information produced by the AIQ is the level of the intention to use alcohol. According to the manual this information is translated into a risk index using norm tables. This information can be used with the intended target population (school going learners). Criterion c is met: The result of the AIQ is provided in the form of a risk index. The index is easy to apply in practice because it only needs to be considered in terms of how high or low it is. No complicated interpretations are required. Note: Because criterion a is not met one cannot assign a rating of 2, 3 or 4. This rating

may seem low in light of the fact that criteria b and c are met. However consider the implication of not meeting criterion a. The AIQ does not provide an adequate measure of the construct it intends to measure. Therefore, although, it functions well in terms of the other criteria it is not a good questionnaire to use.

45

Rate the manual’s indication of the purpose of the AIQ The purpose of the AIQ is indicated if: a the aim of the AIQ is stated b the AIQ’s target population is defined c the design of the AIQ is described Rate 1 if: none of the following Rate 2 if: a Rate 3 if: a + b Rate 4 if: a + b + c

Rate 3 Criterion a is met: The aim of the AIQ is described: “The Alcohol use Intention Questionnaire (AIQ) can be used to determine the level of a person’s intention to use alcohol.” Criterion b is met: The target population is indicated as: “The questionnaire is meant to be used in a preventive context – in other words, it should be used with individuals who have not been submitted to a treatment programme.” Criterion c is not met: The design (i.e. a brief indication of what the questionnaire looks like) is not described.

46

Rate the manual’s indication of the psychometric properties of the AIQ The psychometric properties of the AIQ are indicated if

information is provided concerning: a the sample used to test the AIQ b the procedures used to analyse and select items for

the AIQ c the reliability of the AIQ d the validity of the AIQ Rate 1 if: none of the following Rate 2 if: a Rate 3 if: a + b Rate 4 if: a + b + c or a + b + d Rate 5 if: a + b + c + d

Rate 2 Criterion a is met: The sample used for the development of the AIQ was 100 patients in a treatment clinic in Germany. Criterion b is not met: Although information is provided concerning item-total correlations and item variance the manual does not indicate what procedures were followed in selecting (or rejecting) items for the AIQ. Criterion c is not met: Although the manual includes information about reliability (split-half and test-retest) it does not provide and discuss reliability coefficients. Criterion d is not met: Although the manual includes information about validity (convergent and discriminant) it does not provide and discuss validity coefficients.

47

Rate the manual’s critical consideration of the psychometric properties of the AIQ

The psychometric properties of the AIQ are considered critically if:

a the sample used to pilot the AIQ is considered critically b item analysis and selection procedures used for the AIQ

are considered in terms of established practice c the reliability of the AIQ is considered critically d the validity of the AIQ is considered critically Rate 1 if: none of the following Rate 2 if: a Rate 3 if: a + b Rate 4 if: a + b + c or a + b + d Rate 5 if: a + b + c + d

Rate 1 Criterion a is not met: The sample is indicated but not discussed critically. Had they done so the test developers might have realised a fundamental issue about the sample. The sample consisted of people in treatment. But the manual indicates that the AIQ should not be used with people already in treatment. Because the AIQ was not developed using people who were not in treatment one wonders how valid and reliable it could actually be in assessing people who are in non-treatment situations (e.g. the learners in the school going context). Criterion b is not met: item analysis and selection procedures are not indicated and are not considered in terms of established practice. Criterion c is not met: Reliability coefficients are not indicated and there are no critical considerations of reliability. Criterion d is not met: Validity coefficients are not indicated and there are no critical consideration of validity.

48

Rate the manual’s description of the procedures for utilising the AIQ

The procedures for utilising the AIQ are described if: a instructions are provided for administering the AIQ b instruction are provided for scoring the AIQ c instructions are provided for interpreting the results Rate 1 if: none of the following Rate 2 if: a Rate 3 if: a + b Rate 4 if: a + b + c

Rate 4 Criterion a is met: (See Administration in Section 3 of the manual) Criterion b is met: (See Scoring in Section 3 of the manual) Criterion c is met: (See Interpretation in Section 3 of the manual)

49

Rate the structure of the manual The manual is structured properly if: a the text is clustered into topical sections b topics are presented in logical order c the information about each topic is relevant and

presented coherently

Rate 4 Criterion a is met: The manual is divided into clearly indicated sections that deal with particular topics Criterion b is met: The topics presented the manual is done so in a logical order Criterion c is met: The information provided in each section is relevant and presented coherently.

Rate 1 if: none of the following Rate 2 if: a Rate 3 if: a + b Rate 4 if: a + b + c

Note: This rating is only about the structure of the manual and how information is presented in it. The rating does not refer to the completeness or the quality of the information. The completeness and the quality of the information provided were already rated elsewhere.

50

Rate the text of the manual The manual is written properly if: a the text is factual and objective b the text is clear and to the point c the text is not inundated with spelling, grammar and

punctuation errors Rate 1 if: none of a, b, c Rate 2 if: one of a, b, c Rate 3 if: two of a, b, c Rate 4 if: a + b + c

Rate 4 Criterion a is met: The manual is written in factual and objective language. (As opposed to a text that is emotional and based on unfounded personal believes.) Criterion b is met: The text is clear and to the point Criterion c is met: The text is not inundated with spelling, grammar and punctuation errors. Note that this does not mean that no errors exist in the text. However such errors, if they exist, are minimal.

51

You: So, give me your final opinion. Do you think we can use the AIQ?

Jan: The AIQ is not perfect. But no questionnaire is ever going to

be absolutely perfect for the situation in which it is required. So I think we can use it

Lindi: I think we can use it if we correct the questions that are not formulated correctly

Carl: I think we cannot use the AIQ because there is a problem with its content validity, which is a fatal flaw

You agree most with, second most with and least with:

Carl Lindi Jan

Carl’s offers the best response. One cannot use a questionnaire if it does not adequately cover the content domain implicated by the questionnaire’s intention. Lindi’s suggestion is not without merit. Questions that are not formulated correctly have to be corrected. However, even if all the questions included in the AIQ are perfect one still cannot use the AIQ due to the flaw observed by Carl. Jan’s response is not in order. It does not reflect proper consideration of the AIQ in terms of its psychometric properties.

question discussion · study the scenario carefully before you consider the discussion questions....

Documents