The impact of weight adjustment on the

accuracy of business tendency surveys

An assessment of the manufacturing survey of South Africa

George Kershoff, BER

What are business tendency surveys?

Use the same questionnaire

to collect qualitative information

from the same selection (panel) of firms.

Use weights to calculate the survey results

Sample weights

Seize weights

Firm weights

Sector weights

Firm weights

No. of

employees

Size class Firm

weight

Small

1 – 19 1 1

20 – 49 2 4

50 – 99 3 10

Large

100 – 199 4 19

200 – 299 5 34

300 – 399 6 48

400 – 499 7 62

500 – 999 8 94

1 000+ 9 286

Firm weights

No. of

employees

Size class Firm

weight

Small

1 – 19 1 1

20 – 49 2 4

50 – 99 3 10

Large

100 – 199 4 19

200 – 299 5 34

300 – 399 6 48

400 – 499 7 62

500 – 999 8 94

1 000+ 9 286

Sector weightsSector Weight %

Food 15.1

Beverages 4.7

Textiles 2.4

Clothing 2.3

Wood 2.3

Paper 4.6

Printing 4.0

Chemicals 8.7

Plastic 3.3

Non-metal minerals 3.3

Basic metals 7.6

Metal products 5.4

Machinery 6.0

Electrical machinery 4.5

Transport equipment 18.1

Furniture & other 5.3

Not published 2.2

Total excl petroleum 100.0

Calculating the results in practiceSize

class

Firm

weight

Sector

weight

Total

weight

Up Same Down Total

M

e

a

t

A. Small 2 4 0.051 0.204 0.204

B. Small 3 10 0.051 0.510 0.510

C. Large 6 48 0.051 2.448 2.448

D.

Large8 94 0.051 4.794 4.794

Sum 0.204 2.958 4.794 7.956

% 3 37 60 100

V

e

h

i

c

l

e

s

E. Small 3 10 0.106 1.060 1.060

F. Large 8 94 0.106 9.964 9.964

G. Large 8 94 0.106 9.964 9.964

Sum 11.024 0.000 9.964 20.988

% 53 0 47 100

T

o

t

Sum 11.228 2.958 14.758 28.944

% 39 10 51 100

Calculating the results in practiceSize

class

Firm

weight

Sector

weight

Total

weight

Up Same Down Total

M

e

a

t

A. Small 2 4 0.051 0.204 0.204

B. Small 3 10 0.051 0.510 0.510

C. Large 6 48 0.051 2.448 2.448

D.

Large8 94 0.051 4.794 4.794

Sum 0.204 2.958 4.794 7.956

% 3 37 60 100

V

e

h

i

c

l

e

s

E. Small 3 10 0.106 1.060 1.060

F. Large 8 94 0.106 9.964 9.964

G. Large 8 94 0.106 9.964 9.964

Sum 11.024 0.000 9.964 20.988

% 53 0 47 100

T

o

t

Sum 11.228 2.958 14.758 28.944

% 39 10 51 100

Calculating the results in practiceSize

class

Firm

weight

Sector

weight

Total

weight

Up Same Down Total

M

e

a

t

A. Small 2 4 0.051 0.204 0.204

B. Small 3 10 0.051 0.510 0.510

C. Large 6 48 0.051 2.448 2.448

D.

Large8 94 0.051 4.794 4.794

Sum 0.204 2.958 4.794 7.956

% 3 37 60 100

V

e

h

i

c

l

e

s

E. Small 3 10 0.106 1.060 1.060

F. Large 8 94 0.106 9.964 9.964

G. Large 8 94 0.106 9.964 9.964

Sum 11.024 0.000 9.964 20.988

% 53 0 47 100

T

o

t

Sum 11.228 2.958 14.758 28.944

% 39 10 51 100

Calculating the results in practiceSize

class

Firm

weight

Sector

weight

Total

weight

Up Same Down Total

M

e

a

t

A. Small 2 4 0.051 0.204 0.204

B. Small 3 10 0.051 0.510 0.510

C. Large 6 48 0.051 2.448 2.448

D.

Large8 94 0.051 4.794 4.794

Sum 0.204 2.958 4.794 7.956

% 3 37 60 100

V

e

h

i

c

l

e

s

E. Small 3 10 0.106 1.060 1.060

F. Large 8 94 0.106 9.964 9.964

G. Large 8 94 0.106 9.964 9.964

Sum 11.024 0.000 9.964 20.988

% 53 0 47 100

T

o

t

Sum 11.228 2.958 14.758 28.944

% 39 10 51 100

Calculating the results in practiceSize

class

Firm

weight

Sector

weight

Total

weight

Up Same Down Total

M

e

a

t

A. Small 2 4 0.051 0.204 0.204

B. Small 3 10 0.051 0.510 0.510

C. Large 6 48 0.051 2.448 2.448

D.

Large8 94 0.051 4.794 4.794

Sum 0.204 2.958 4.794 7.956

% 3 37 60 100

V

e

h

i

c

l

e

s

E. Small 3 10 0.106 1.060 1.060

F. Large 8 94 0.106 9.964 9.964

G. Large 8 94 0.106 9.964 9.964

Sum 11.024 0.000 9.964 20.988

% 53 0 47 100

T

o

t

Sum 11.228 2.958 14.758 28.944

% 39 10 51 100

Average number of response per quarter

0

50

100

150

200

250

300

350

400

2001 2002 2003 2004 2005 2006 2007 2008 2009

Sector Total - Small (<100) Sector Total - Large

Impact of non-responsesSize

class

Firm

weight

Sector

weight

Total

weight

Up Same Down Total

M

e

a

t

A. Small 2 4 0.051 0.204 0.204

B. Small 3 10 0.051 0.510 0.510

C. Large 6 48 0.051 2.448 2.448

D.

Large8 94 0.051 4.794 4.794

Sum 0.204 2.958 4.794 7.956

% 3 37 60 100

V

e

h

i

c

l

e

s

E. Small 3 10 0.106 1.060 1.060

F. Large 8 94 0.106 9.964 9.964

G. Large 8 94 0.106 9.964 9.964

Sum 11.024 0.000 9.964 20.988

% 53 0 47 100

T

o

t

Sum 11.228 2.958 14.758 28.944

% 39 10 51 100

Impact of non-responsesSize

class

Firm

weight

Sector

weight

Total

weight

Up Same Down Total

M

e

a

t

A. Small 2 4 0.051 0.204 0.204

B. Small 3 10 0.051 0.510 0.510

C. Large 6 48 0.051 2.448 2.448

D.

Large8 94 0.051 4.794 4.794

Sum 0.204 2.958 4.794 7.956

% 3 37 60 100

V

e

h

i

c

l

e

s

E. Small 3 10 0.106 1.060 1.060

F. Large 8 94 0.106 9.964 9.964

G. Large 8 94 0.106 9.964 9.964

Sum 11.024 0.000 9.964 20.988

% 53 0 47 100

T

o

t

Sum 11.228 2.958 14.758 28.944

% 39 10 51 100

Impact of non-responsesSize

class

Firm

weight

Sector

weight

Total

weight

Up Same Down Total

M

e

a

t

A. Small 2 4 0.051 0.204 0.204

B. Small 3 10 0.051 0.510 0.510

C. Large 6 48 0.051 2.448 2.448

D.

Large8 94 0.051 4.794 4.794

Sum 0.204 2.958 4.794 7.956

% 3 37 60 100

V

e

h

i

c

l

e

s

E. Small 3 10 0.106 1.060 1.060

F. Large 8 94 0.106 9.964 9.964

G. Large 8 94 0.106 9.964 9.964

Sum 11.024 0.000 9.964 20.988

% 53 0 47 100

T

o

t

Sum 11.228 2.958 14.758 28.944

% 39 10 51 100

3.162

24.150

Composition of the survey results vis-à-vis

the benchmark

2001 2005 2009 Average

Transport equipment

% of total

Benchmark

(domestic sales)15.3 18.1 15.3 17.5

Survey (sum of

responses)9.9 8.6 4.7 8.1

Sector composition of survey results using

fixed weights

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%0

1Q

20

1Q

30

1Q

40

2Q

10

2Q

20

2Q

30

2Q

40

3Q

10

3Q

20

3Q

30

3Q

40

4Q

10

4Q

20

4q3

04

Q4

05

Q1

05

Q2

05

Q3

06

Q1

06

Q2

06

Q3

06

Q4

07

Q1

07

Q2

07

Q3

07

Q4

08

Q1

08

q2

08

Q3

08

Q4

09

Q1

09

Q2

09

Q3

09

Q4

Not publ -

Transp equip -

Elec machin -

Machinery -

Metal products -

Basic metals -

Non-metal min -

Chemicals -

Wood -

Clothing -

Textiles -

Beverages -

Food -

Plastics -

Printing -

Paper -

Furn & other -

Weight adjustment

Adjust individual weights ex post to align the survey’s

sector composition with that of the reference series

Provide for

Over and under representation

Non-responses

Weight adjustment in practiceSize

class

Firm

weight

Sector

weight

Total

weight

Up Same Down Total

M

e

a

t

A. Small 2 4 0.051 0.204 0.204

B. Small 3 10 0.051 0.510 0.510

C. Large 6 48 0.051 2.448 2.448

D.

Large8 94 0.051 4.794 4.794

Sum 0.204 2.958 4.794 7.956

% 3 37 60 100

V

e

h

i

c

l

e

s

E. Small 3 10 0.106 1.060 1.060

F. Large 8 94 0.106 9.964 9.964

G. Large 8 94 0.106 9.964 9.964

Sum 11.024 0.000 9.964 20.988

% 53 0 47 100

T

o

t

Sum 11.228 2.958 14.758 28.944

% 39 10 51 100

3.162

24.150

Sector composition after weight adjustment

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%0

1Q

20

1Q

30

1Q

40

2Q

10

2Q

20

2Q

30

2Q

40

3Q

10

3Q

20

3Q

30

3Q

40

4Q

10

4Q

20

4q3

04

Q4

05

Q1

05

Q2

05

Q3

06

Q1

06

Q2

06

Q3

06

Q4

07

Q1

07

Q2

07

Q3

07

Q4

08

Q1

08

q2

08

Q3

08

Q4

09

Q1

09

Q2

09

Q3

09

Q4

Not publ -

Transp equip -

Elec machin -

Machinery -

Metal prod -

Basic metals -

Non-metal min -

Chemicals -

Wood -

Clothing -

Textiles -

Beverages -

Food -

Plastics -

Printing -

Paper -

Furn & other -

Survey results with and without weight

adjustment

-80

-60

-40

-20

0

20

40

60

80

100

Ma

r-01

Sep-0

1

Ma

r-02

Sep-0

2

Ma

r-03

Sep-0

3

Ma

r-04

Sep-0

4

Ma

r-05

Sep-0

5

Ma

r-06

Sep-0

6

Ma

r-07

Sep-0

7

Ma

r-08

Sep-0

8

Mar-

09

Sep-0

9

Ne

t %

= %

up

le

ss

% d

ow

n

Without weight adjustment (lhs) With weight adjustment (lhs)

Adjusted survey results vs. reference series

-30

-20

-10

0

10

20

30

-100

-80

-60

-40

-20

0

20

40

60

80

100

Ma

r-01

Sep-0

1

Ma

r-02

Sep-0

2

Ma

r-03

Sep-0

3

Ma

r-04

Sep-0

4

Ma

r-05

Sep-0

5

Ma

r-06

Sep-0

6

Ma

r-07

Sep-0

7

Ma

r-08

Sep-0

8

Ma

r-09

Sep-0

9

Ye

ar

on

ye

ar

% c

ha

ng

e

Ne

t %

= %

up

le

ss

% d

ow

n

With weight adjustment (lhs) Reference series (rhs)

Conclusion

Weight adjustment did not improve accuracy in the

case of the manufacturing sector between 2001 and

2009

Accuracy = closeness between survey results and

reference series

Survey results are therefore robust