q2008 - rome italy, july 20081 the mysterious 36% difference: studies on a measurement error anette...

The Mysterious 36% Difference: Studies on a Measurement Error

Anette Björnram, Boris Lorenc, Andreas Persson, Klas Wibell

Statistics Sweden

Q2008 - Rome Italy, July 2008 2

Agenda

• Context

• Two measurement methods

• Results

• Hypotheses about the difference

• A methodological study to understand the difference

• Tentative conclusions and further questions


Context

• A customer aimed to measure aspects of late payments of invoices issued by small & middle-sized enterprises (SME’s) in Sweden to other enterprises, e.g.– proportion of late payments– length of delay (in days)– proportion of late payments paid more than 10 days

and more than 30 days after the due-in date– etc.


Methods

• Study A: posing retrospective questions to person(s) responsible for bookkeeping in the sampled SME’s

• Study B: sampling invoices issued by the sampled SME’s within the reference period, recording relevant dates and events about these invoices, and calculating the desired information from these data


Methods (cont’d)

• Study A – example questions


Methods (cont’d)

• Study A – example questions (cont’d)


Methods (cont’d)

• Study A – Potential response issues– complex response process:

• determining the reference period• having an account of the number of issued invoices in

this period• having an account of the number of late paid invoices in

this period• having an account of the length of delay (in day) of the

late paid invoices in this period• building the desired averages• performing the above by customer type breakdown

– instead, the respondents might provide their impressions as answers (satisficing)


Methods (cont’d)

• Study B – the questionnaire


Methods (cont’d)

• Study B – Potential issues– possibly complex form– a balance between a complex sampling plan and a

sub-optimal sampling plan to sample invoices– sampling variation: small number of observations

(due to response burden considerations)– insufficient control of the data collection procedure

(a self-administered mode)• deviations regarding the sampling procedure• deviations caused by data unavailability


Results


Results (cont’d)

• No general reason to suspect the difference between the measurement methods A and B– instead of satisficing, the respondents by Method

A may have had “running tallies” directly available

• But, something happened with the item Proportion invoices paid late (a key variable for the customer)

• Given the issues associated with the two methods, several hypotheses may be considered

• Thus, a methodological follow-up performed


Hypotheses


Hypotheses (cont’d)

• Probed in a lengthy qualitative interview

• Those respondents chosen with the largest difference on Item 4 (proportion delayed payments) between Study A and Study B (range of 70-90% difference)

• 20 respondents chosen initially, data collection suspended after 12 respondents as a clear pattern emerged


Difference Explained

• In summary, H8(H1) strongly supportedH8: The respondents do not consider short delays as delays

H1: The question not comprehended correctly in Study A

• H2 and H4 got some supportH2: Access to data difficult or impossible in Study A

H4: There is a time lag in the availability of data in Study A

• The other hypotheses received weak or no support


Difference Explained (cont’d)

• The respondents do not consider short delays as delays (H8/H1)– there is a distinction between the implied definition

of ‘delay’ by the respondents and our customerThere is a buffer of maybe one week, so up to one week is the same as the due-in date

– in some cases related to some actWhen I feel that I must do something then at least one week must have passed, if it is shorter than that then I don’t careIf 4-5 days have passed and the money hasn’t arrived, then I’d call and ask if maybe the invoice had got lost on the way



– sometimes the question misinterpreted (H1/H3)I thought generally, how many invoices we issue in a year and how many reminders we send out

– technical delays (bank, weekends, post delivery)– there is a cost in sending out reminders– even routines do not always alarm promptly (H4)

As I don’t get [i.e. see] those [payments] that are between 1 and 4 days late, so of course that I have underestimated the proportion of late payments [one of the two SME’s in the study using an automated routine]

Thus, a “Gray Zone” exists



• But, Item 4 question difficult (H2)– especially customer size

Our system does not support provision of such data, so it was mostly feeling, impression.

– impression provided rather than data accessed Difficult, as I don’t know the size of the companies I’m selling to.



• A tentative model for gray zone (NB. Very approximate point estimates!)



• Further on ‘Gray Zone’ (based on data from Study B)– Suppose a ‘Gray Zone’ of length 8 days exists– What would the proportion of invoices paid late be

if the credit period is extended by the length of Gray Zone (i.e. 8 days) for all the 1550 invoices in Study B?



NB: the length of delay (given it is a delay) expected to be the same irrespective of whether there is no Gray Zone (as in Study B) or there is one (as in Study A), if the delays follow an exponential distribution

• memoryless property of the exponential distribution• thus, reasonable that there is no difference between Study A and

Study B on Item 7

Answer: 15 %• close to the

value in Study A (there: 14%)

0

10

20

30

40

50

60

4 8 12 16 20 24 28 Fler

Length of delay, in days (Study B)

Nu

mb

er o

f in

voic

es


Tentative conclusions

• Estimation not necessarily wrong: if information important, an approximate running tally may be kept– further empirical confirmation desirable, including

accounting of cognitive mechanisms that provide for this ability


Tentative conclusions (cont’d)

• But, the real research question (the length of delays that SME’s are exposed to by their enterprise customers) is difficult to measure by any method– context-specific interpretation of what constitutes a

delay– source data changed (e.g. if a customer demands

longer credit period)

• In that sense, it is a sensitive question in an establishment survey


Tentative conclusions (cont’d) – issues

• How to study such questions?– quantitatively: the numbers do not reflect the

intended phenomenon– quantitatively: the results cannot be expressed in

numbers

• How to recognize such questions?


Thank YouAnette Björnram

Boris Lorenc

Andreas Persson

Klas Wibell

[email protected]

q2008 - rome italy, july 20081 the mysterious 36% difference: studies on a measurement error anette...

Documents

2008resultsq2008 rome

dataq2008 rome italy

sampling invoices

methods contdstudy b

number of late paid

number of issued invoices

item proportion invoices

difference20 respondents