oecd/ec task force on harmonization of survey operations and technical design internet business...

OECD/EC Task Force on Harmonization of Survey Operations and Technical Design Internet Business Tendency Surveys

InternetBusiness Tendency Surveys

The author would like to thank all colleagues who have contributed to this research project:

Aggelos Tsakanikas, Alma Rutkauskiene, Aloisio Campelo, Bianca Maria Martelli, Braileanu Gheorghita and Nedelea Olga, Brigita Vrabic Kek, Casper Larsen, Dagmar Blahova, Dorthy Watson, Eliza Loucaidou, Evelin Ahermaa, Frank Hansen, George

Kershoff, Gerhard Schwarz, Hanna Sekowska, Illmars Vanags, Jan Henrik Wang, Jean-Paul Vonck and Isabelle De Greef, Jonathan Wood, Jose Manuel Almendros Ulibarri,

Kevin Caruana, Marie Hormannova, Paulo David, Penna Urrila, Philippe Scherrer, Raymund Petz, Richard Etter, Roger Knudsen, Sidney Vergouw, Soshihiro Masuda,

Teofana Genova, colleagues from the Ifo Business Surveys Dpt. and ECFIN

2/19


Previous ResearchTask Force on Harmonization of Survey Operations and Technical Design

PRACTICE OF INTERNET SURVEYS

Task Force Group on

Identification and Assessment of Recommended Practices for the Design of

Internet Surveys

Task Force MembersCurtin, R. (University of Michigan, United States), De Greef, I. (Bank of Belgium), Etter, R. (KOF / ETZ, Switzerland), Malgarini, M. (ISAE, Italy), Neubarth, W. (ZUMA, Germany), Nillson, Ronny (OECD), Scherrer, P. (INSEE, France), Stangl, A. (Ifo Institute, Germany), Wood, J. (CBI, United Kingdom), European Commission

Task Force on Harmonization of Survey Operations and Technical Design


Task Force Group on

Identification and Assessment of Recommended Practices for the Design of

Internet Surveys

Task Force MembersCurtin, R. (University of Michigan, United States), De Greef, I. (Bank of Belgium), Etter, R. (KOF / ETZ, Switzerland), Malgarini, M. (ISAE, Italy), Neubarth, W. (ZUMA, Germany), Nillson, Ronny (OECD), Scherrer, P. (INSEE, France), Stangl, A. (Ifo Institute, Germany), Wood, J. (CBI, United Kingdom), European Commission

3/19


1. Internet Adoption of the BTS Institutes Web Surveys E-mail Surveys

2. Practical Issues: Formats and Spam

3. The Experience with the Internet as Survey Instrument

4. Summary and Research Outlook

Overview

32 institutes (25 EU countries and 7 non-EU countries)

4/19


Internet Adoption of the BTS Institutes

Countries

Web is used for the following reasons: EU / non EU

Website of the institute 20 / 7

Display of survey results to the public 18 / 7

Display of individualized survey results for participants 4 / 1

Web questionnaire 10 / 4

None of the 14 institutes has abandoned the traditional survey mode yet.

5/19


Web SurveysImplemented in the year Manufacturing Retail Services Construction Investment

1998 Estonia Estonia Estonia Estonia

2000Sweden,

SwitzerlandSweden,

SwitzerlandSweden,

SwitzerlandSweden

2001 Finland, BrazilFinland,Germany

Finland

2002 LuxembourgGermany,

LuxembourgFinland

2003 Malta

2004Hungary, Germany,

Austria, NorwayHungary

Austria,Hungary

Austria,Hungary

2005 France Austria

2006Portugal,

South AfricaPortugal,

South AfricaFrance, Portugal,

South AfricaPortugal,

South AfricaPortugal

Existing Web Surveys 14 6 10 9 3

6/19


49%

95%85%

45% 45%35%

20%

5%

60%

45%

55%

10%

49%

50%

5% 5%

55% 55%65%

80%

95%

40%

55%45%

85%

50%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Paper Interview Web

EU Countries non-EU CountriesEU Average

Response Rates: Manufacturing Sector

7/19


51%

90%

47%

15%

70%

49%

10%

53%

85%

30%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Paper Interview W eb

EU Countries non-EU CountryEU Average

Response Rates: Retail Sector

8/19


39%

85%

40%

25%

40% 40%

5%

60%

56%

15%

30%

75%

60% 60%

95%

40%

5%

30%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Paper Interview W eb Email


Response Rates: Service Sector

9/19


62%

98%87%

70% 70%

35%

1%

97%

60%

2%3%

30% 30%

65% 65%

99%

3%

40%

5%

37%

5%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%



Response Rates: Construction Sector

10/19


E-mail Surveys

Implemented in the year

Manufacturing Retail Services Construction Investment

2000Slovakia,

NetherlandsSlovakia, Slovakia

2002Austria

(abandoned)Slovakia, Germany

Austria(abandoned)

2003 Italy, BelgiumItaly,

Belgium

Netherlands,Italy,

BelgiumBelgium

2004 Italy

2005Czech Rep. Romania

Czech Rep. Romania

Czech Rep. Romania

Czech Rep. Romania

Czech Rep. Romania

2006 Greece Greece Greece

Existing E-mail surveys

6 6 8 6 2

11/19


53%

80%

15%

30%20%

75%

99% 95%

33%

20%

85%

25%

1% 5%

14% 80%70%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%



Response Rates: Manufacturing Sector

12/19


52%

75%

20%

45%

20%

60%

95% 98%

35%25%

80%

55%

5%

40%

3% 2%

13% 75%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%



Response Rates: Service Sector

13/19


86% 85%

55%

100%92%

99% 97%

14% 15%

45%

8%1% 3%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%



Response Rates: Retail Sector

14/19


74%85%

50%

90%

45%

99% 95%

26%15%

50%

10%1% 5%

55%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%



Response Rates: Construction Sector

15/19


Practical Issues: Formats and Spam Filter

E-mail format: - 5 institutes send their E-mail in the HTML format- 17 institutes adhere to the plain-text format

Web questionnaire format: - 9 of 10 institutes in the EU have chosen the one-page design- 2 of 4 institutes outside the EU have multiple-page Web forms

The majority of institutes reported an overall bounce rate below 5% According to the institutes’ estimations, of those E-mails that do not

reach the recipients, about 20% are due to Spam-filter.

16/19


Countries For Participants For the Surveyor Paper questionnaire Email questionnaire Web questionnaire

1 J JJ J J2 JJ JJ K L JJ3 LL L J K L4 JJ JJ K JJ5 J JJ JJ JJ6 J J J LL L7 J JJ K JJ8 JJ L JJ J K9 JJ JJ J K J10 J J J11 JJ JJ JJ JJ12 J JJ K JJ13 J JJ J JJ14 L LL J J15 JJ J J J J16 L L JJ JJ17 J19 JJ21 JJ

26 L J JJ LL27 JJ JJ L L28 JJ JJ J JJ29 J J J J30 JJ JJ K JJ31 J J J32 L L JJ L

Practicability of the Internet as a survey instrument

Experience with the different questionnaire modes

Non-EU Countries

EU Countries

17/19


Practicability of the Internet as Survey Instrument for Participants and the Surveyor

Practicability of the Internet for surveying

Very practical

PracticalSomewhat problematic

Problematic Total

For Participants 8 9 4 1 22

For the Surveyor 12 6 4 1 23

18/19


Experience with the Different Survey Modes

Experience with the different survey modes

Very good

Good SatisfactoryRather bad

/ badTotal

Paper questionnaire 9 12 4 0 25

Email questionnaire 4 3 2 4 13

Web questionnaire 6 4 1 4 15

19/19


19/

OECD/EC Task Force on Harmonization of Survey Operations and Technical Design


Reliabilty

Internet respondents tend to read more quickly

Internet environment may lead to a higher inaccuracy

Respondent has to learn the structure of the hypertext

Internet questionnaires affords higher cognitive effort

Higher risk of measurement errors

Prevention and Control

19/



Reliabilty

Internet respondents tend to read more quickly

Internet environment may lead to a higher inaccuracy

Respondent has to learn the structure of the hypertext

Internet questionnaires affords higher cognitive effort

Higher risk of measurement errors

Prevention and Control

23/



Validity – Differences in Responses Findings vary greatly

Selection bias may influence responses

-40

-30

-20

-10

0

10

20

30

40

50

60

70

Q1/20

00

Q2/20

00

Q3/20

00

Q4/20

00

Q1/20

01

Q2/20

01

Q3/20

01

Q4/20

01

Q1/20

02

Q2/20

02

Q3/20

02

Q4/20

02

Q1/20

03

Q2/20

03

Q3/20

03

Q4/20

03

Q1/20

04

Q2/20

04

Total Panel Web Respondents

Balances of the assessments of the present economic situation (IT companies)

23/



Validity – Differences in Responses Findings vary greatly

Selection bias may influence responses

-40

-30

-20

-10

0

10

20

30

40

50

60

70

Q1/20

00

Q2/20

00

Q3/20

00

Q4/20

00

Q1/20

01

Q2/20

01

Q3/20

01

Q4/20

01

Q1/20

02

Q2/20

02

Q3/20

02

Q4/20

02

Q1/20

03

Q2/20

03

Q3/20

03

Q4/20

03

Q1/20

04

Q2/20

04

Total Panel Web Respondents

Balances of the assessments of the present economic situation (IT companies)

20/



Item Non-response

Lower item non-response for Web surveys

On-line participants tend to be less thorough

Internet is perceived to be anonymous

Lower perception of responsibility

On-line readers are less patient

Internet usage is associated with costs

20/



Item Non-response

Lower item non-response for Web surveys

On-line participants tend to be less thorough

Internet is perceived to be anonymous

Lower perception of responsibility

On-line readers are less patient

Internet usage is associated with costs

3/18



Coverage Problems

50

60

70

80

90

100

Fin

lan

d

Sw

ede

n

Den

mark

Can

ad

a

Au

stralia

Czech

Rep

.

Au

stria

Japa

n

New

Zeala

nd

Germ

any

Ireland

Sp

ain

No

rwa

y

Neth

erland

s

Lu

xem

bo

urg

Sw

itzerland

Italy

Po

rtug

al

Greece

UK

Mexico

%10-49 50-249 250 and more

Percentage of Businesses Using the Internet by Number of Employees

Source: OECD 2001 or latest available year 3/18



Coverage Problems

50

60

70

80

90

100

Fin

lan

d

Sw

ede

n

Den

mark

Can

ad

a

Au

stralia

Czech

Rep

.

Au

stria

Japa

n

New

Zeala

nd

Germ

any

Ireland

Sp

ain

No

rwa

y

Neth

erland

s

Lu

xem

bo

urg

Sw

itzerland

Italy

Po

rtug

al

Greece

UK

Mexico

%10-49 50-249 250 and more

Percentage of Businesses Using the Internet by Number of Employees

Source: OECD 2001 or latest available year

20/19


Research Outlook

Methodological Manual

Coverage Response patterns Reliability Validity

Creation of a Sample Internet Questionnaire

Technical Manual


Email: Anna Stangl [email protected]

Ronny Nilsson [email protected]

oecd/ec task force on harmonization of survey operations and technical design internet business...

Documents