analysis of users and usage for uk citizens online democracy · 2014-03-27 · this report was...

1 of 63

Analysis of users and usage for UK Citizens Online Democracy

May 2011

Tobias Escher1

http://people.oii.ox.ac.uk/escher/ :: [email protected]

1 Tobias Escher is currently a Doctoral Student at the Oxford Internet Institute of the University of Oxford.

http://people.oii.ox.ac.uk/escher/

mailto:[email protected]

2 of 63

About this report

This report was commissioned by UK Citizens Online Democracy as part of an evaluation of the major

mySociety websites.

UK Citizens Online Democracy

483 Green Lanes

LONDON

N13 4BS

United Kingdom

Acknowledgements

The author would like to thank the Oxford Internet Institute for making available the datasets of the

Oxford Internet Surveys 2007 & 2009. Furthermore this research has benefited from discussions with

Steffen Albrecht and Matthias Trénel. Not least, various members of mySociety have provided valuable

help. To all of you: Thank you very much!

Copyright

This report is licensed under a Creative Commons Attribution-Non-Commercial-Share Alike 2.0 UK: England

& Wales Licence. This allows you to copy, distribute, display, and perform the work and to make

derivative works for non-commercial purposes as long as you give the original author credit and share

any derivative works under the same license. For more information, see

http://creativecommons.org/licenses/by-nc-sa/2.0/uk/

For any use not covered by this license please contact [email protected]

http://creativecommons.org/licenses/by-nc-sa/2.0/uk/


3 of 63

Contents

Executive Summary ................................................................................................................................................ 4

1 Site Description .............................................................................................................................................. 7

1.1 History ...................................................................................................................................................... 9

1.2 Awards & Media ................................................................................................................................... 11

1.3 Size & Cost ............................................................................................................................................ 11

2 Site Usage ...................................................................................................................................................... 13

2.1 General Usage: Visitor Statistics ......................................................................................................... 13

2.2 Specific Usage: Popular Content and Feature Usage ...................................................................... 20

2.3 User Satisfaction ................................................................................................................................... 27

3 Site Users ...................................................................................................................................................... 31

3.1 Political Profile of TheyWorkForYou Users .................................................................................... 32

3.2 Demographics of TheyWorkForYou Users ..................................................................................... 34

4 Ecosystem ..................................................................................................................................................... 40

4.1 Political Engagement in Britain .......................................................................................................... 40

4.2 Similar Sites ............................................................................................................................................ 41

5 Anecdata........................................................................................................................................................ 44

References .............................................................................................................................................................. 46

Appendices ............................................................................................................................................................ 47

A. List of Figures .............................................................................................................................................. 47

B. List of Tables ................................................................................................................................................ 49

C. Methodology ................................................................................................................................................ 50

C.1. Survey ..................................................................................................................................................... 50

C.2. Survey Questionnaire ........................................................................................................................... 53

C.3. Sample Bias / Representativeness ...................................................................................................... 56

D. Detailed Survey Results and Comparative Data ..................................................................................... 59

E. mySociety – a short summary .................................................................................................................... 61

4 of 63

Executive Summary

About

TheyWorkForYou is a website, launched in 2004, that provides detailed information on members of

parliament (including their voting behaviour and expenses) as well as parliamentary proceedings such as

debates. It covers the Westminster and the Scottish parliaments as well as the Northern Ireland

Assembly. In 2006 it officially became a mySociety project.

Cost

The website cost only a few thousand pounds to set up, as a result of the major contributions to the

site made by volunteers, but with annual maintenance costs of more than £20,000 it is currently the

most expensive of mySociety‟s projects.

Usage

TheyWorkForYou is by far the most heavily used mySociety website, receiving on average between

200,000 and 300,000 visits every month. This is about twice as much as traffic to WhatDoTheyKnow

which is currently the second most popular mySociety site. Noticeable usage peaks occurred in May

2009 during the height of the MP expenses scandal when many people visited the site to see the

expenses records of their MPs (450,000 visits within one month), as well as in May 2010 when the site

provided an election quiz to help voters decide on how to cast their vote in the General Election

(280,000 visits in a week). The web statistics and data from the user survey suggest that about half of

the overall audience of TheyWorkForYou are regular users that come to the site repeatedly, with about

one in five using the site at least once a month. A considerable share of usage is work-related. Not only

do 30% of respondents to the user survey state this explicitly but usage is also concentrated during

working hours and working days.

The 2% of visits that can be directly attributed to the UK parliament are clearly motivated by

professional interest, as well as the additional 2-3% from other governmental sources. However, two

thirds of traffic is received from search engines with about 2% contributed by a Google Adwords

campaign. About 10% of searches are explicitly for “TheyWorkForYou” but the majority are for names

of specific representatives. The remaining 30% of traffic is split equally between direct entry and

referring websites such as Wikipedia and the mySociety site WriteToThem which both contributed 1-

2% of overall traffic.

By far the most popular content on TheyWorkForYou is the section with information about the House

of Commons, which receives two thirds of all traffic. Within this section the profiles of individual MPs

are of most interest while debates in the House receive only 15% of traffic. The option to receive email

alerts for appearances of certain representatives or keywords in debates also proved popular, with

almost 110,000 registered email alerts and a total of 2.8m alerts sent out in 2009 alone.

5 of 63

User satisfaction

Overall satisfaction with the site as measured by the Net Promoter score is 18% which is considerably

lower than that reported for WriteToThem (44%). Satisfaction levels are mainly related to whether

users found the information they were looking for (Net Promoter score of 30%) or not (Net Promoter

score of -30%). The only slight issue with the site seems to be with design, but almost all users think

the site is very navigable and well structured.

User description

Data on users is derived from a user survey conducted between October 2009 and March 2010. Due to

the nature of the site it was not possible to collect information from a representative sample of users.

The data therefore can only be taken as indicative of broad trends (see section C.3 on Sample Bias /

Representativeness).

In terms of prior political activity three out of five users (60%) have never looked up information on

what their representatives were doing before they came across TheyWorkForYou. While there are

clearly users who are already politically active, not only in relation to checking information on their

representatives, and who are organized in groups, one in five users (21%) of the site has not been

politically active within the last year, was not involved with a political or community group and has with

the help of TheyWorkForYou looked up information on representatives for the first time.

In terms of demographics there is a strong male bias and a strong overrepresentation of people with a

university degree that also translates into strong participation from high income groups. Few of these

deviations can be explained by already existing biases in political participation as the observed bias is

much stronger than the one reported for those who are politically active (for reference data see section

4.1 Political Engagement in Britain). Also in the TheyWorkForYou audience people above the age of

54 tend to be over-represented, while those younger than 45 are under-represented in comparison to

the Internet population. The share of retired people is twice as high as the Internet average and also

sick or disabled people are over-represented.

TheyWorkForYou‟s success in achieving its own goals

Goal 1: to provide unbiased, non-partisan information

Almost all users believe that TheyWorkForYou provides unbiased and non-partisan

information.

Goal 2: to bridge the growing democratic disconnect (e.g. most people don‟t know the name of their

MP, nor their constituency, let alone what their MP does or says in their name)

The majority of people who come to site already have basic political knowledge: four out of five

state that they already knew the name of their MP. At the same time three out of five users have

never before looked up information on what their MP was doing in parliament. About 90% of

users say the website has improved their knowledge about their representatives. As one user

commented: “It is a way of getting parliament to the people”

6 of 63

Goal 3: to make it easy for people to keep tabs on their elected and unelected representatives in

Parliament and other assemblies, provide information in a better way than official sites

Most users argue the site is easily navigable and well structured. As one user said: “it gives every

voter the chance to participate more in our democracy and ensure that their MP is representing the constituency”

The comments also repeatedly explicitly mentioned that TheyWorkForYou is much better than

the official parliament site. However, in the teaser to the survey about 40% of users indicated

that they did not find what they were looking for. While this number is in part related to the

timing of the teaser question which might have come too early for some visitors, there is some

significant concern here for a systematic bias as in particular women, older people, those

without a university degree and those with some form of disability had more difficulty of

finding the information they were looking for.

Goal 4: to allow fact checking (e.g. give access to source evidence) and make MPs feel accountable; to

reward truthful MPs, to allow fair judgement of MPs on basis of what they do

The watchdog function is to some degree on the users' minds, given the 30% who responded

they were checking a fact and keeping an eye on what their representatives do. The user

comments clearly indicate that this is important to users, as expressed for example by this one:

“I can make a judgment on an MP by looking at his/her voting record”. For about half of all users this

has even led to an improved opinion about their representatives.

General goals of mySociety:

Goal 5: to build websites that give people simple, tangible benefits in the civic and community aspects

of their lives as well as to teach the public and voluntary sectors, through demonstration, how to use

the internet most efficiently to improve lives.

mySociety has succeeded in building a site that a large number of people is using, that seems to

offer some benefit to them as almost half of them use the site repeatedly and that users say is

easy to use.

Goal 6: to reach a representative share of the (Internet) population and activate people who would

otherwise not get engaged

The data on the politicisation of TheyWorkForYou users highlighted above shows that this goal

has been achieved to some degree, e.g. as three out of five users (60%) have never looked up

information on what their representatives were doing before they came across

TheyWorkForYou. However, the user demographics exhibit a very strong bias towards male

users as well as the educated and high income groups of society, suggesting that engagement

does indeed happen but mainly from social groups that are already more likely to be politically

engaged.

7 of 63

1 Site Description

TheyWorkForYou.com is a website that offers rich information on both the members and the

proceedings of all parliaments and assemblies in the UK with the exception of Wales but including the

House of Lords. The information available includes records of voting behaviour, speeches and

expenses by members of the respective parliament or assembly as well as the written proceedings such

as debates or written statements. The information available varies depending on the respective

parliament or assembly (see Table 1 below) with most detailed information being provided for the

Westminster parliament for which debates are not only accessible in text form but also as video

coverage. Users have the opportunity to annotate written proceedings or create customised newsfeeds

and email alerts for the latest appearances of an individual members as well as email alerts mentioning

certain keywords. Last but not least TheyWorkForYou provides a short and uniform URL to every

section of parliamentary proceedings to allow easy and precise linking.

Figure 1: Screenshot of TheyWorkForYou homepage (March 2011)

The website obtains its data via scraping and parsing the official parliamentary records published on the

Web. For example the data for the UK parliament is extracted from the online version of Hansard with

8 of 63

the help of code developed by the Parliament Parser project2, run by many of the same people as is

TheyWorkForYou. The following table outlines in detail which information is available on

TheyWorkForYou for the various parliaments and assemblies.

Table 1: Coverage of TheyWorkForYou

parliament / assembly members proceedings

House of Commons all Members of Parliament

(MPs) back until 1806

House of Commons Hansard:

all debates since General

Election November 1935; all

written answers and written

ministerial statements since

General Election of June 2001

House of Lords all Lords back to the House of

Lords Act 1999

full House of Lords Hansard

(except Grand Committees)

since November 1999

Scottish Parliament all Members of the Scottish

Parliament (MSPs)

full Scottish Parliament Official

Report (since May 1999)

Northern Ireland Assembly all Members of the Legislative

Assembly (MLAs)

full Northern Ireland Assembly

Hansard (since July 1998)

Welsh Assembly no coverage no coverage

Goals of Site

The main goal of TheyWorkForYou is to provide unbiased information on what members of

parliament have been doing, saying and voting on. As it says on the website3, it is motivated by the

perception of a growing democratic disconnect that manifests itself in the fact that many people do not

know the name of their MP or what their MP does or says in their name. Out of this main goal come

some specific aims related to the provision of this information. These became clear during a number of

informal interviews with those people at mySociety involved in running the site. Roughly these goals

can be put into three categories: value, transparency and engagement:

Value

To provide better information than official sites, in particular make access to information more

logical for people who are interested

To allow easy navigation of all the data related to a specific MP (e.g. make it easy to find out

how MPs vote)

To allow tracking of legislative bills

2 http://ukparse.kforge.net/parlparse/ [26.04.2010]. This project is also supplying the data for the website

www.publicwhip.org.uk

3 TheyWorkForYou – About Us. http://www.theyworkforyou.com/about/ [15.03.2011]

http://ukparse.kforge.net/parlparse/

http://www.theyworkforyou.com/about/

9 of 63

Transparency

To allow fact-checking (e.g. give access to source evidence)

To allow fair judgement of MPs on the basis of what they do and to make MPs feel

accountable; also to reward truthful MPs

To allow citizens to act as watchdogs

Engagement

To make citizens better informed

To engage people in politics (engagement being any form of action concerned with politics)

To reduce costs of lobbying on issues people care about

Underlying these goals are mySociety‟s two main missions. The first is to be a charitable project which

builds websites that give people simple, tangible benefits in the civic and community aspects of their

lives. The second is to teach the public and voluntary sectors, through demonstration, how to use the

internet most efficiently to improve lives. These official goals are again supplemented by goals that are

not necessarily explicitly stated but that seem to be inherent in many of the discussions among the

mySociety community. These centre around activating people who would otherwise not get engaged

and more broadly in reaching a representative share of the (internet) population.

1.1 History

TheyWorkForYou was officially launched on 6 June 2004 at the NotCon04 (now Open Tech)

conference. Work on the website had begun about a year earlier by the group of people who had

already built FaxYourMP and Public Whip.

Immediately after its launch it was named “the most amazing, subversive piece of political webware” by Web guru

Cory Doctorow on the widely read blog boingboing.net�. In early 2006 it officially became a mySociety project

and mySociety has been responsible for keeping it up and running ever since. It started out covering

the debates in the House of Commons.

10 of 63

Table 2: Timeline of major steps in TheyWorkForYou development (2004 – 2010)

2004

6 June 2004 Launch of TheyWorkForYou.com at NotCon04 (now Open Tech) conference

2005

April 2005 Now possible to see how MPs voted on key issues in parliament

2006

early 2006 TheyWorkForYou officially becomes a mySociety project

June 2006 The proceedings and members of the House of Lords are added

December 2006 Addition of the Northern Ireland Assembly

2007

September 2007 Even the Queen has her own page on TheyWorkForYou4

2008

May 2008 Coverage of the Scottish Parliament is introduced thanks to the volunteer

contribution of Mark Longair

June 2008 Video coverage of the House of Commons debates is added to

TheyWorkForYou, helped by the efforts of users in timestamping the data

2009

June 2009 Site design underwent a major revamp

July 2009 Extension of TheyWorkForYou coverage to include House of Commons

debates back until 1935

2010

May 2010 As part of a major volunteer effort TheyWorkForYou offers an election quiz

in which voters can match their opinions on national as well as local issues

with the preferences of the candidates running in their constituency

Currently Matthew Somerville of mySociety takes the lead on maintenance of the site, assisted by the

other mySociety paid developers, but the creation was a major volunteer effort to which many people

contributed including Richard Allan, Martin Belam, James Crabtree, James Cronin, Louise Crow,

Stephen Dunn, Yoz Grahame, Phil Gyford, David Heath, Francis Irving, Joe Lanman, Ben Laurie,

Mark Longair, Tom Loosemore, Stefan Magdalinski, Dorian McFarland, Anno Mitchell, Danny

O'Brien, Duncan Parkes, Etienne Pollard, Richard Pope, Sam Smith, Matthew Somerville, Tom

Steinberg, Stuart Tily, Julian Todd and Denise Wilton.

Section 1.3 “Size & Cost” outlines the efforts involved in developing and maintaining the site.

4 http://www.theyworkforyou.com/royal/elizabeth_the_second [17.03.2011]

http://www.theyworkforyou.com/royal/elizabeth_the_second

11 of 63

1.2 Awards & Media

Early on the site achieved recognition of its important contribution to democracy when it received the

2005 New Statesman Media Award in the category Contribution to Civic Society.5 The website is also

regularly mentioned in newspapers with an average of slightly more than one article a week. An

exception is the year 2009 when coverage more than doubled with 141 news articles, mainly related to

the MPs expenses scandal when newspapers were regularly referring their readers to the website in

order to get details on their representatives‟ expense figures.

Figure 2: Articles in English language newspapers mentioning ‘theyworkforyou’ (2004 – 2010)

year articles

2004 (from June) 64

2005 52

2006 66

2007 67

2008 58

2009 141

2010 71

Source: Nexis UK, search for terms „theyworkforyou‟

Unfortunately information on the number of blog posts mentioning „theyworkforyou‟ is not reliably

available because Google Blogsearch, the only blog search engine that covers past years, counts some

content on TheyWorkForYou as blog posts and includes these in its search. However, coverage in the

blogosphere is extensive with Google Blogsearch reporting more than 14,000 posts mentioning

“theyworkforyou” in 2010 alone.

1.3 Size & Cost

Based on code size, TheyWorkForYou used to be the third biggest of the main mySociety projects after

WhatDoTheyKnow and WriteToThem but with the inclusion of an elaborate testing suite its code base

is now the biggest, as the table below highlights. While it cost very little to develop, reflecting the major

contribution to the site made by volunteers, it is by some distance the most expensive project to

maintain.

5 http://www.newstatesman.com/nma/nma2005/nma2005winners.php [17.03.2011]

http://www.newstatesman.com/nma/nma2005/nma2005winners.php

12 of 63

Table 3: Development and maintenance cost of major mySociety projects

WriteTo

Them

TheyWork

ForYou

Pledge

bank ePetitions66

FixMy

Street

WhatDo

TheyKnow

start of

development 01.08.2004 01.06.2003 01.01.2005 24.04.2006 01.09.2006 01.08.2007

official launch 14.02.2005 06.06.2004 13.06.2005 14.11.2006 07.03.2007 22.02.2008

development

man hours 1,288 1,500 833 651 315 760

development cost £24,080 £5,000 £16,280 £18,764 £6,660 £23,750

lines of code (incl.

markup)a) 85,404 298,078 19,359 31,440 15,670 210,939

maintenance (annually)

server &

bandwidth £1,000 £1,000 £1,000 £1,000 £1,000 £1,000

developer support

(days) 8 48 8 8 8 20

labour cost £2,400 £14,400 £2,400 £2,400 £2,400 £6,000

backups and other

support cost b) £5,000 £5,000 £5,000 £5,000 £5,000 £5,000

total annual

maintenance cost £8,400 £20,400 £8,400 £8,4000 £8,400 £12,000

Source: mySociety

Notes: Numbers reflect state of affairs in 2010. a)

as of February 2011, does not reflect a number of libraries shared between projects; as with

all mySociety projects the source code is open source, in the case of TheyWorkForYou under a BSD-style license, b)

rather than just static state

backups, mySociety works through a regular programme of testing, including recreating sites and services totally from the back-ups in place to make

sure they are robust and appropriate

Before TheyWorkForYou officially became a mySociety project one of the key developers was paid for

a month by a small grant from UK Citizens Online Democracy. In summer 2006, the Ministry of

Justice (then the Department for Constitutional Affairs) provided £6,000 out of its Innovation Fund to

develop the TheyWorkForYou API. In summer 2008 the same fund again gave money to allow

extending the TheyWorkForYou coverage to include historical debates back until 1935. In the past, as

with all mySociety sites, the bandwidth costs were donated by Easynet, but this has ceased.

TheyWorkForYou also gets some support through a Google grant for free Google Adwords campaigns

which amounted to about $60,000 in 2010 (see section 2.1.4 Advertising).

6 Online petitions of the UK Prime Minster, available at http://petitions.number10.gov.uk/. Since May 2010 it is no longer possible for

members of the public to create and sign new petitions, but mySociety still maintains the searchable archive of former petitions available

from the Prime Minister‟s website.

http://petitions.number10.gov.uk/

13 of 63

2 Site Usage

2.1 General Usage: Visitor Statistics

The following figure illustrates the development of visits to the website7. The numbers show a growth

in the first two years of the existence of the site. Since 2007 on average visitor numbers have remained

fairly stable between 200,000 and 300,000 visits a month with noticeable drops during the summer

months (which corresponds to the summer recess of the Westminster parliament) and between

Christmas and New Year. A visitor record was achieved in the run-up to the 2010 General Election

when TheyWorkForYou received more than 230,000 visits in election week alone, equalling the

number it would usually receive in a full month. For more details see section 2.2.3 General Election

Campaign 2010. Another significant usage spike occurred in May and June 2009 when in relation to the

MP expenses scandal8 many people were using the site to check on their MPs‟ expenses.

0

50,000

100,000

150,000

200,000

250,000

300,000

350,000

400,000

450,000

500,000

550,000

Jan

05

Mar 05

May 0

5

Jul 0

5

Sep

05

No

v 0

5

Jan

06

Mar 06

May 0

6

Jul 0

6

Sep

06

No

v 0

6

Jan

07

Mar 07

May 0

7

Jul 0

7

Sep

07

No

v 0

7

Jan

08

Mar 08

May 0

8

Jul 0

8

Sep

08

No

v 0

8

Jan

09

Mar 09

May 0

9

Jul 0

9

Sep

09

No

v 0

9

Jan

10

Mar 10

May 1

0

Jul 1

0

Sep

10

No

v 1

0

nu

mb

er

per

mo

nth

TheyWorkForYou.com visitor statistics

visits

unique visitors

Google Analytics PiwikAWStats

Figure 3: Visits and unique visitors to TheyWorkForYou (November 2004 - December 2010)

Source: mySociety AWStats, mySociety Google Analytics, mySociety Piwik web analytics

Note: Underreporting for data in April 2010 as no statistics were collected from 26 April 2010 until around midday Friday 30 April 2010

due to an update error.

7Detailed web stats are available from Google Analytics since September 2006 and from Piwik since September 2008, both of which

employ a page-tagging technology. Before this, logfile data (dating back to November 2004) was analysed using AWStats.

8 For a useful summary see the Wikipedia article: http://en.wikipedia.org/wiki/United_Kingdom_Parliamentary_expenses_scandal

http://en.wikipedia.org/wiki/United_Kingdom_Parliamentary_expenses_scandal

14 of 63

In terms of traffic TheyWorkForYou has been consistently mySociety‟s most popular site. It receives

about twice as many visits as the second most popular site WhatDoTheyKnow where people can view

and submit Freedom of Information requests.

0

100,000

200,000

300,000

400,000

500,000

Sep

08

Oct 0

8

No

v 0

8

Dec 0

8

Jan

09

Feb 0

9

Mar 09

Ap

r 09

May 0

9

Jun

09

Jul 0

9

Aug

09

Sep

09

Oct 0

9

No

v 0

9

Dec 0

9

Jan

10

Feb 1

0

Mar 10

Ap

r 10

May 1

0

Jun

10

Jul 1

0

Aug

10

Sep

10

Oct 1

0

No

v 1

0

Dec 1

0

vis

its

pe

r m

on

th

visitor statistics of major mySociety sites

Pledgebank WriteToThem WhatDoTheyKnow TheyWorkForYou FixMyStreet

Figure 4: Visitor statistics of major mySociety sites (September 2008 - December 2010)

Source: mySociety Piwik web analytics

Note: Underreporting for data in April 2010 as no statistics were collected from 26 April 2010 until around midday Friday 30 April 2010

due to an update error

2.1.1 Usage Patterns

During its more than six years of existence the site has exhibited a number of periodic usage patterns.

As can be seen in Figure 3, each year in August it receives considerably less traffic, owning to the

summer recess of Westminster parliament and most likely that of other parliaments and assemblies as

well. A similar low-usage period time is the time around Christmas and New Year when apparently few

people are motivated to check relevant information.

General elections typically result in an increased interest in the information on TheyWorkForYou. The

unprecedented usage peak during the General Election campaign in 2010 is covered in detail in section

2.2.3, and the General Election campaign in May 2005 also resulted in an increase in traffic. Apart from

the General Election campaign in 2010 the most popular time in terms of traffic was the MP expenses

scandal, particularly in May 2009. Weekly traffic to the site was more than twice as high as average with

for example 150,000 visits in the week from 11-17 May. Major news sites such as the website of the

Telegraph and the Guardian would refer their users to TheyWorkforYou.

15 of 63

Days with particularly high numbers of visits usually reflect significant events British politics. In 2007

the most popular day (18,402 visits) was 28 June 2007 when Jacqui Smith was appointed Home

Secretary. On that day, 7% of all searches mentioned her name and 160 visits came via her Wikipedia

page. In 12 June 2008 when David Davis resigned as MP in order to force a by-election traffic hit

another year high (17,731 visits) with significant traffic coming from search engine searches for David

Davis (14% of all visits) and order-order.com, otherwise known as the Guido Fawkes blog (5% of all

visits). In 2009 the most popular day was 28 April (36,960 visits). 58% of visitors came through direct

entry of the URL, about four times as many as average (see section 2.1.3 on referrers), and 16% of

search engine searches were specifically for www.theyworkforyou.com. This unusual pattern resulted

from an explicit mention of the website in The Mirror. The newspaper urged their readers to visit this

website and then to email their MPs in a campaign to get full citizenship rights for Gurkhas who fought

for Britain9. In 2010 the busiest day on the site was pre-election day with 84,253 visits to

TheyWorkForYou (including the election quiz).

Most visits occur on working days during office hours (9am – 5pm), when traffic is about twice as high

as on weekends. This indicates that a considerable share of traffic could be work related, and indeed

about 30% of people answering our user survey say that their use of the site is somehow related to their

work (see also section on user groups below). Nevertheless, considerable usage also takes place after

5pm as the figure below shows for the year 2010.

Figure 5: Time of day for visits to TheyWorkForYou (2010)

Source: mySociety Piwik web analytics

Figure 6: Work-related usage of

TheyWorkForYou (from survey)

Source: mySociety/UK Citizens Online Democracy

user survey 2009/10

9 http://www.mirror.co.uk/celebs/news/2009/04/28/treat-gurkhas-like-the-heroes-they-are-115875-21314272/ [17.03.2011]

http://www.theyworkforyou.com/

http://www.mirror.co.uk/celebs/news/2009/04/28/treat-gurkhas-like-the-heroes-they-are-115875-21314272/

16 of 63

2.1.2 Frequent Users

In 2010 about one third of visits were repeat visits. This number has been very stable over the last few

years, and it can be thought of as a lower bound for the share of actual people using TheyWorkForYou

repeatedly10. Data from the user survey suggests that about 60% of users have used TheyWorkForYou

more than once within the last twelve months, as the figure below shows, which seems not

unreasonable given the web statistics. According to this data (however, see notes on how far one may

generalise survey results in section 3 Site Users) about 20% of users access the site very frequently, at

least once a month or more. This is roughly supported by data from Google Analytics showing that in

2010 at least 12% of users (again, a lower bound for the reasons cited above) visited the site nine times

or more.

Figure 7: Use frequency of TheyWorkForYou (based on responses to user survey)

Source: mySociety/UK Citizens Online Democracy user survey 2009/10

10 The number of first time visits is bound to be inflated given that people use different computers and browsers or might delete their

cookies, all resulting in their visit being counted as a first time visit.

17 of 63

2.1.3 Referrers

There are three main sources of traffic to a website: visitors arriving from search engines after searching

for a term for which the site was in the relevant results list, links from other websites, and people who

access it directly by typing the address in the browser. The figure below illustrates the share of these

three sources for TheyWorkForYou.

0%

10%

20%

30%

40%

50%

60%

70%

80%

2006(since

September)

2007 2008 2009 2010

sh

are

of

tota

l vis

its t

o s

ite

TheyWorkForYou.com referrers

search engines websites (incl. WTT)

direct entry WriteToThem.com

Figure 8: Where visits to TheyWorkForYou are coming from (2006 - 2010)

Source: mySociety Google Analytics

It shows a consistently high share of around 70% of traffic arriving from search engines. In nine out of

ten cases traffic from search engines comes from Google, of which, in 2010, around 2% was traffic

generated by a Google Adwords campaign. The most popular search terms are variants of “they work for

you” which in 2010 accounted for almost one out of ten searches. Other popular searches that lead

people to TheyWorkForYou are searches for variants of “who is my mp”. However, the majority of

searches are for names of particular representatives. Traffic from search engines declined slightly in

2010 as a result of an increase in traffic from other websites (in particular Wikipedia) and from users

accessing the site directly. This change is likely to be related to the election campaign.

Search engine referrals often include traffic from people who already know TheyWorkForYou and

might use it to find the site faster, illustrated by the already highlighted fact that in 2010 around 10% of

all keyword searches included the words “they work for you” in some variation. However, the most

important role for search engines is to let people know about the site in the first place as data from the

18 of 63

user survey underscores: about half of those responding to the question say they found out about

TheyWorkForYou with the help of a search engine.

Figure 9: How users of TheyWorkForYou find out about the site


Personal recommendations and campaigns as well as media coverage are also important ways for

people to find out about TheyWorkForYou, together accounting for about 30%.

About 4% in the user survey said they found out about TheyWorkForYou from other mySociety

websites. Indeed, as Table 4 documents, for a long time WriteToThem.com (a mySociety website

where citizens can send messages to their elected representatives) was the website from which

TheyWorkForYou received most visits, usually around 2% of all visits from 2006 to 2008. In recent

years this share has dropped, most notably in 2010. This seems part of a general decline in referrals

from WriteToThem, which might have been accelerated by the General Election: there were no MPs

for a period of about three weeks between the dissolution of Parliament and the election, so

WriteToThem did not allow writing to MPs during this time. Currently, the number one referring

website is Wikipedia which often carries links to TheyWorkForYou from its pages about MPs and in

2010 was responsible for 2% of overall traffic.

The following table outlines the top five referring websites for the last four years. Apart from the

dominance of WriteToThem and Wikipedia, in 2007 and 2008 traffic from the official website of the

UK Prime Minister was an important driver of traffic to the site because it linked to the transcript of

Prime Minister's Questions on TheyWorkForYou11. 2010 also saw Twitter rise among the top five

referrals, particularly on 17 November 2010 when more than 1,000 visitors came to the site in relation

to a written answer about the telephone tapping of the Metropolitan Police Service, as well as through

tweets relating to the General Election on the 29 April, 6 May and 12 May. The General Election also

11 http://webarchive.nationalarchives.gov.uk/+/http://www.number10.gov.uk/Page306

19 of 63

marked the start of a sustained increase in traffic from Facebook, which now also features in the top 5

referrers.

Table 4: Top 5 referring websites to TheyWorkForYou (2007 – 2010)

rank 2007 2008 2009 2010

1 writetothem.com

(58,710)

writetothem.com

(59,176)

en.wikipedia.org

(64,246)

en.wikipedia.org

(75,184)

2 en.wikipedia.org

(51,855)

en.wikipedia.org

(55,501)

writetothem.com

(43,499)

writetothem.com

(28,840)

3 number-10.gov.uk,

number10.gov.uk ,

pm.gov.uk

(24,579)

number-10.gov.uk ,

number10.gov.uk ,

pm.gov.uk

(25,156)

telegraph.co.uk

(17,781)

ukpollingreport.co.uk

(24,412)

4 hearfromyourmp.com

(17,360)

google.co.uk

(10,479)

google.co.uk

(12,695)

facebook.com

(22,461)

5 publicwhip.org.uk

(11,352)

publicwhip.org.uk

(10,470)

publicwhip.org.uk

(10,566)

twitter.com

(20,492)

Total traffic

from websites

419,868 462,081 498,554 624,027


Note: Numbers in brackets denote visits from this site to theyworkforyou.com.

2.1.4 Advertising & Marketing

In general mySociety does not heavily market its sites, mainly due to financial constraints. The only real

form of advertising happens in the form of Google Adwords sponsored via a Google grant. In 2010

about 43,000 visits (2% of overall traffic) were generated by sponsored ads that showed up next to

people‟s Google searches for words such as “member of parliament” or names of particular MPs. In 2010

an ad was displayed for about 4m searches which resulted in almost 150,000 clicks.

Apart from that, TheyWorkForYou is linked from other mySociety sites, in particular WriteToThem

where users are for example encouraged to sign up to TheyWorkForYou e-mail alerts for their

constituency MP after they send a message to their representative. As a result WriteToThem is one of

the most important website referrers for TheyWorkForYou, in 2010 sending about 30,000 visits to the

site and accounting for about 1% of total traffic. Otherwise publicity efforts are usually related to

specific mySociety campaigns and announcements, and make use of the mySociety blog, Facebook and

not least the site itself. For example the blog was used for publicising the election quiz prior to the

General Election 2010 when a blog post on mysociety.org told readers “How to get TheyWorkForYou Into

https://www.google.com/analytics/reporting/referring_sources?id=1095626&pdr=20090101-20091231&cmp=average&trows=50&gdfmt=nth_day



https://www.google.com/analytics/reporting/referring_sources?id=1095626&pdr=20100101-20100520&cmp=average&trows=500&gdfmt=nth_week&pli=1












https://www.google.com/analytics/reporting/referring_sources?id=1095626&pdr=20100101-20100520&cmp=average&trows=500&gdfmt=nth_week









20 of 63

Your Local Paper/Radio Station in 5 minutes”12 and in 2009 the Facebook group had some success in

marshalling support against a Commons vote which was aimed at concealing MPs‟ expenses13. These

campaigns also generated some media coverage (see also section 1.2 Awards & Media).

2.1.5 Special User Groups

Among the main site users are a number of organisations of particular interest. Every week about 1,500

visits come from the Houses of Parliament (as identified by source IP addresses resolving to

parliament.uk) which account for about 2% of all visits made to TheyWorkForYou. The figure below

illustrates this usage for 2010 – notably, it only takes place during working days.

Figure 10: Visits from UK parliament to TheyWorkForYou (2010)


An additional 2.5% of visits come from governmental sources (.gov.uk and mod.uk). Apart from

government, the site sees substantial usage by the BBC, which accounted for almost 0.5% of all visits in

2010, or by political parties. The Conservative Party Central Office accounted for 0.26% of traffic up

until the General Election, after which usage substantially decreased. Comments from the user survey

also underscore that parts of the civil service are relying on TheyWorkForYou:

“I find it very useful at work (civil servant).”

“Great research tool - everything is clearer than the PIMS [Parliamentary Information Management

Service] database we have on the intranet”

“I've just been told by an MP's chief of staff that it is better than Hansard!”

2.2 Specific Usage: Popular Content and Feature Usage

In general the bounce rate of visits to TheyWorkForYou is around 60%. That is, three out of five

visitors to the site access just one single page before leaving. What is more, two thirds of visits take less

than 30 seconds, although this is not a reliable measure for people accessing one page only. This does

not necessarily signal that users did not find relevant content. It might also mean that people very

quickly find what they are looking for, in particular as all the information about an individual MP is

12 http://www.mysociety.org/2010/05/04/how-to-get-theyworkforyou-into-your-local-paperradio-station-in-5-minutes/ [17.03.2011]

13 http://www.mysociety.org/2009/01/21/blimey-it-looks-like-the-internets-won/ [17.03.2011]

http://www.mysociety.org/2010/05/04/how-to-get-theyworkforyou-into-your-local-paperradio-station-in-5-minutes/

http://www.mysociety.org/2009/01/21/blimey-it-looks-like-the-internets-won/

21 of 63

summarized on a single page. This is supported by the fact that even among returning visitors, those

that apparently have found a real value on the site, more than half of visits bounce, suggesting that the

single page they visited did indeed give them what they were after. Also around 15% of visits take five

minutes or longer, suggesting a real interaction with the content.

2.2.1 Popular Content

Judging by traffic the most popular content on TheyWorkForYou is the information relating to the

House of Commons, which accounted for two thirds of page impressions in 2010. The second most

popular section is search, which is often also for content related to proceedings of the House of

Commons. 6% of traffic goes to the House of Lords while the remaining two assemblies covered on

TheyWorkForYou (Scottish Parliament, Northern Ireland Assembly) account for comparatively little

traffic totalling 3% of all page impressions. Last but not least around 8% of the traffic is related to

miscellaneous features of TheyWorkForYou such as the API, email alerts, debate annotations and the

lookup for the constituency boundaries in the Scottish and Northern Irish elections. The overall picture

was the same as in 2009, hence indicating a rather stable interest pattern.

House of Commons

65%

Lords6%

Norther Ireland Assembly

1%

Other0%

Scottish Parliament

2%

Search18%

TheyWorkForYou features

8%

Figure 11: Distribution of page impressions to content sections on TheyWorkForYou (2010)


Note: Based on a total of 9,022,537 page impressions for whole year of 2010; the same picture emerges for analysis of unique page impressions,

i.e. counting repeated impression within one session of the same user only once.

The figure below breaks down the traffic to the House of Commons content in order to show what

kind of information is accessed in this most popular section. The majority of traffic goes to content

about MPs (profiles and postcode lookup), followed by some margin by written answers and

statements as well as Commons debates. Again, the popularity of the individual content sections has

changed little since 2009.

22 of 63

MP information60%

Written answers and statements

18%

House of Commons

debates15%

Westminster Hall debates

3%

written ministerial statements

2%

video timestamping

1%

Public Bill Committees

1%

Figure 12: Traffic to House of Commons related content (2010)


Note: Based on a total of 5,886,650 page impressions for whole year of 2010.

The table below reports the top three most popular pages in 2009 for a number of content sections.

One fact to highlight is that debates can be popular even if they are already dated, the famous speech

by Winston Churchill on Britain‟s war efforts being a case in point which is linked from Wikipedia

article(s). Similarly, a written answer to a question by Sarah Teather MP in 2006 about the minimum

classroom temperatures in the UK is still very frequently found through Google searches as well as a

particular answer on Yahoo Answers.

Table 5: Most popular pages in selected content sections (2010)

rank MP

profile pages

Commons debates Written Answers

and Statements

Lords

profile pages

Lords debates

1 David Cameron

(29,022)

12.07.2010 debate

on NHS White

Paper

(5,440)

03.11.2009 James

Brokenshire to

Ministry of Justice

on offences relating

to Regulation of

Investigatory

Powers Act

(5,305)

Lord James of

Blackheath

(4,293)

01.11.2010 debate

on Comprehensive

Spending Review

(8,987)

2 Gordon Brown

(25,761)

04.06.1940 Winston

Churchill: “We shall

fight on the beaches

…”

(4,687)

25.07.2006 Sarah

Teather to

Education and Skills

on classroom

temperatures

(4,493)

Lord Mandelson

(3,340)

21.10.2010 debate

on economic and

cultural impacts of

immigration in the

UK

(6,416)

23 of 63

3 Diane Abbott

(24,181)

06.04.2010 debate

on Digital Economy

Bill

(2,297)

16.11.2010 Tom

Watson to Attorney

General on

telephone tapping of

the Metropolitan

Police Service

(3,596)

Lord Henley

(2,745)

15.06.2010 debate

UN paper “Treating

Drug Dependence

through Healthcare,

not Punishment”

(1,367)


Note: Based on page impressions (provided in brackets).

Video coverage of Commons debates was added in June 2008. It dates back to June 2007 and is up to

date with the exception of recent months, due to some technical difficulties which still need to be fixed.

Access figures of these videos also point to popular topics in Commons debates as illustrated in Table

6 below. It is important to note that these access numbers do not necessarily translate into debates

having been watched, but usually signal that a user has accessed the text of the debate for which the

video will automatically be loaded. Still it is a rather eclectic collection. The Christmas Recess debate for

2008, in which traditionally mainly backbenchers discuss a wide range of issues, is apparently the all

time favourite. Otherwise Prime Minister‟s Questions during the economic crisis in late 2008 have

proved popular with users as has for example the Academies Bill.

Table 6: Top 3 video of debates most often requested from TheyWorkForYou (2008 – 2010)

rank 2008 2009 2010

1 18 December 2008

Christmas Recess Debate (mainly

backbenchers on a range of issues)

(19,933)

19 November 2008

Prime Minister‟s Questions

(see rank 2 for 2008)

(36,884)

8 December 2009

questions in the House of Commons

to Communities and Local

Government Secretary

(18,322)

2 19 November 2008

Prime Minister‟s Questions (mainly

on economic crisis and recovery)

(17,876)

18 December 2008

Christmas Recess Debate


(25,323)

18. December 2008

Christmas Recess Debate


(17,096)

3 11 June 2008

debate on Counter-Terrorism bill

(6,924)

15 January 2009

debate about report on collapse of

Equitable Life as well as Gaza

(21,579)

22 July 2010

Debate on Academies Bill

(15,220)

Source: mySociety TheyWorkForYou AWStats

Note: Numbers in brackets denote hits on respective video file.

24 of 63

The search queries of users are another source of information about content of interest. The largest

chunk of external search traffic albeit with only 10% of the total search traffic is for variants of “they

work for you”, apparently from people who know the site or have heard about it and want to find it via

the search engine. The remaining 90% of searches are predominantly for names of individual

representatives. The internal search phrases also reveal a strong interest for individual persons, led by

searches for the Prime Minister.

Users were also questioned directly about their usage motivations. These do not always translate

directly into specific content sections and, with hindsight, the provided categories were not ideal which

is probably also the reason why they are nearly evenly distributed. Nevertheless they do highlight that

there is a significant number of people who tend to just browse content on TheyWorkForYou with no

specific use intention as well as that the watchdog function is to some degree on the users mind, given

the 30% of users who responded they were checking a fact or keeping an eye on their representatives.

As one user commented:

“Because a) it's useful and provides info in simple terms and b) it's necessary to keep track of

MPs.”

Figure 13: Usage motivations of TheyWorkForYou users


2.2.2 Popular Features

The website provides the opportunity to create email alerts notifying users of appearances of a particular

representative or mentioning of certain keywords. As of March 2011 there were almost 110,000

different email alerts registered, sent to about 65,000 different email addresses. A substantial amount of

email is generated in this way. For example about 2.8 million emails were sent in 2009 alone in relation

to registered email alerts. After the General Election in May 2010 the number of email alerts has

25 of 63

substantially increased as can be seen in Figure 14. A noticeable spike in May, June and July 2010 is

likely to indicate users creating alerts for newly elected MPs and those who have moved to the front

benches. Even towards the end of the year several thousand new alerts were still being created each

month. As of March 2011 the most popular email alerts for MPs were Gordon Brown (684 alerts), Meg

Hillier (587) and Diane Abbott (553), however, none of them can currently rival the subscription

numbers reached by George Galloway (1040) when he was still an MP.

There is also the opportunity to annotate debates. There are now more than 26,000 comments on debates

and each month between 200 and 300 new annotations are added. The drop in annotations during

summer recess (see figure below) indicates that users seem mainly to annotate current debates.

Users can also register on the site if they want to post annotations to debates or manage their email alerts

more easily. As of March 2011 there are more than 20,500 people (i.e. email addresses) registered and

every month this number increases by 200 to 300 with a marked increase in the month following the

election (see figure below).

As of March 2011 almost 11,000 people have also subscribed to receive the TheyWorkForYou newsletter

which is supposed to provide information on TheyWorkForYou stories and relevant content on a

monthly basis.

0

500

1,000

1,500

2,000

2,500

3,000

3,500

4,000

Jan 09 Mar 09 May 09 Jul 09 Sep 09 Nov 09 Jan 10 Mar 10 May 10 Jul 10 Sep 10 Nov 10

TheyWorkForYou feature take-up

users registering on TheyWorkForYou debate annotations email alerts created&confirmed

Figure 14: Signup rates to TheyWorkForYou site and email alerts as well as debate annotations (2009 - 2010)

Source: mySociety TheyWorkForYou database records

Note: In May 2010 a total of 8,481 email alerts were created and confirmed, in June 13,612 and in July 5,420.

In June 2008 video coverage of Commons debate was added to the site. Now when reading a debate

the video of the particular speech can be watched too while the corresponding Hansard text is

26 of 63

highlighted next to it (see section 2.2.1 on Popular Content for the most often watched videos). The

video is obtained from BBC Parliament but was manually marked up to tag the beginning of each

individual speech by a group of more than 350 volunteers. Coverage of Commons debates goes back to

June 2007 and is up to date with the exception of the most recent few months, due to some technical

difficulties. The volunteers not only timestamped the initial full year's backlog of video coverage,

amounting to more than 40,000 speeches, in about twelve weeks,14 but they continue to mark up new

video as it is added, as the figure below shows. The total lack of activity around parliamentary summer

recess in August and September in all three years is testament to the speed of the volunteer efforts as

there is never a real backlog of videos to timestamp.

0

2,500

5,000

7,500

10,000

12,500

15,000

17,500

20,000

22,500

25,000

Jun 08 Sep 08 Dec 08 Mar 09 Jun 09 Sep 09 Dec 09 Mar 10 Jun 10 Sep 10

nu

mb

er

of

sp

ee

ch

es

tim

es

tam

pe

d

TheyWorkForYou video timestamping efforts

Figure 15: Development of volunteer video timestamping on TheyWorkForYou (June 2008 – September 2010)

Source: mySociety TheyWorkForYou database records

Note: There is no data for the last three months of 2010 as during this time import of video was broken.

2.2.3 General Election Campaign 201015

The election campaign for the General Election started formally with the dissolution of parliament on

12 April 2010 and ended with the close of polls at 10pm on 6 May. For this election a quiz was

provided under the subdomain election.theyworkforyou.com that allowed voters to compare their views on

local and national issues with those of the candidates running for election in their constituency. It was

set up in a major effort by mySociety and volunteers (read more about it in section 5 Anecdata) and had

a sizeable impact on traffic to TheyWorkForYou. In 2009 TheyWorkForYou received on average

around 65,000 visits every week. Usage of TheyWorkForYou had already increased slightly but notably

in March with about 70,000 weekly visits, and further in April with about 90,000 weekly visits, despite

the fact that there were no debates taking place in the House of Commons nor were there any MPs

14 http://www.mysociety.org/2008/08/28/amazing-volunteers-do-entire-year-of-theyworkforyou-video-clip-timestamping-in-weeks/

[17.03.2011] 15

All data reported in this section obtained from Google Analytics because due to an update problem no data was collected with Piwik

from 26 April 2010 until around midday Friday 30 April 2010.

http://www.mysociety.org/2008/08/28/amazing-volunteers-do-entire-year-of-theyworkforyou-video-clip-timestamping-in-weeks/

27 of 63

because Parliament had been dissolved. In the week of the election TheyWorkForYou experienced a

record usage peak: with more than 230,000 visits to TheyWorkForYou (excluding the election quiz)

normal traffic nearly quadrupled. Usage was particularly high on the day before the election when the

site received in a single day the same number of visits as it would otherwise get in a full week. Traffic to

TheyWorkForYou has clearly benefited from the election quiz: on election day and the day before,

about 25,000 visitors to TheyWorkForYou went on to access the election quiz. The election quiz alone

received about 100,000 visits in election week of which traffic from TheyWorkForYou accounted for

about 55%. Effectively TheyWorkForYou and the election quiz received a total of 280,000 visits in

election week (2 – 8 May), 84,000 on pre-election day alone. The figure below visualises visits to both

TheyWorkForYou (in green) and on top of that visits to the election quiz

(election.theyworkforyou.com) that came not via TheyWorkForYou but from other sources (in red).

daily traffic to TheyWorkForYou during election campaign

0

10,000

20,000

30,000

40,000

50,000

60,000

70,000

80,000

90,000

1

Mar

6

Mar

11

Mar

16

Mar

21

Mar

26

Mar

31

Mar

5

Apr

10

Apr

15

Apr

20

Apr

25

Apr

30

Apr

5

May

10

May

15

May

vis

its

theyworkforyou.com election.theyworkforyou.com (without traffic from TWFY)

Figure 16: Traffic to TheyWorkForYou and election quiz during election campaign


Note: Traffic to theyworkforyou.com excludes traffic from election.theyworkforyou.com and vice versa so numbers add up to total visits to both

sites.

2.3 User Satisfaction

The following section presents user assessments of the site that are mainly derived from the survey of

users of TheyWorkForYou, which is described in more detail in section 3 Site Users. This sample is not

representative of the entire TheyWorkForYou audience. In general it can be assumed that this is less of

a problem for questions concerning the general assessment of the site. However, user satisfaction is

28 of 63

clearly related to whether use of the site was successful, here defined as the user having been able to

find the information that was being searched for. From the statistics generated by the teaser that asked

this particular question it can be seen that of all 12,333 users who responded to the teaser by 20 April

2010, 59% found what they were looking for. In the survey sample this is substantially more, with 80%

of users saying that they had found the relevant information. So people who found the information

they were looking for on the site were more likely to answer the survey and clearly their assessment of

the site is more positive than that of people who did not successfully use the site to find a particular

piece of information. Therefore where appropriate, the answers of these two groups will be analysed

separately. Cause for concern should be the differences between those users who were able to

successfully use the site and those who failed to find the information they were seeking. There are no

significant differences between these two user groups in relation to political knowledge and

involvement but there are significant demographic differences. Users who fail to find information on

TheyWorkForYou are more often women (50% more than successful users), older people (in particular

55-64 years old), are less likely to have a university degree (54% vs 66%) and twice as likely to have a

disability than successful users. What is more, this is not just a question of becoming familiar with the

site, because half of those unsuccessful users are actually repeat users who still fail to find information

despite their prior experience with the site as Figure 17 shows.

Figure 17: Comparing usage frequency of successful and unsuccessful users


On general measure of satisfaction is asking users how likely they are to recommend the site to friends

or colleagues. Based on this a measure called the Net Promoter score is calculated. By asking users

about the likelihood of recommendation of a service the Net Promoter score aims to be a simple

measure of both customer satisfaction and user loyalty. It calculates the difference between the group

of promoters of a site (those indicating a likelihood of recommendation of 9 or 10) and the group of

29 of 63

detractors (scoring 0 - 6) – users indicating 7 or 8 are considered to be passive and are not taken into

account (for more details see Reichheld, 2003).

According to this calculation, 44% of TheyWorkForYou users are Promoters while 26% are

Detractors, resulting in a Net Promoter Score of 18%. While in general this might be called a

conservative estimate of satisfaction because it does not count many that are “likely” to recommend the

site (i.e. those scoring 7 or 8), it is a rather low score, in particular compared to the Net Promoter score

of 44% reported for WriteToThem (Escher, forthcoming).

Figure 18: Net Promoter score, i.e. likelihood of users to recommend TheyWorkForYou


Unsurprisingly the Net Promoter score of those who did not find the information they were looking

for is much lower. In fact, they are diametrically opposed: those who found the information have a

score of 30% while those not finding the information scored -30%.

While it comes as no surprise that TheyWorkForYou will be mainly judged by whether or not it

satisfies a particular need for information, it is useful to identify the issues that users like and don‟t like

about the site that might be responsible for the comparatively low overall Net Promoter score. Figure

19 reports the assessment of a number of site characteristics by respondents to the user survey.

TheyWorkForYou achieves very good scores for navigability and the structuring of information.

Almost all respondents found the site easy to use and well set up. Comments by users in the survey

very often highlighted that TheyWorkForYou is much more usable than the official sources:

“It is a hundred times easier to search than Hansard itself. I've spent two hours on Hansard before

coming here and have just found what I want in under a minute.”

“More functional than the Parliament website”

30 of 63

“In general keep up the good work. Far better than the Parliament website.”

Concern only derives from the comparatively poor assessment of the site design as about 30% of users

judged the site not pretty to look at. Crucially, virtually all users believe that TheyWorkForYou provides

information in an unbiased and non-partisan way which is also expressed in some of the comments in

the user survey:

“It is good to see what was actually said rather than heresay from the media.”

“you can access factual information rather than rely on the media`s interpretation”

“impressive in it's neutrality”

This information has helped to improve most users‟ knowledge about their representative as expressed

in this comment:

“Excellent repository of information. Important democratic resource.”

This improved knowledge has improved the opinion about their representatives for at least half of the

surveyed users. On all these items unsuccessful users were predictably more negative than those users

who found the information they were looking for.

Figure 19: User assessment of TheyWorkForYou usability and use effects


Users made a number of suggestions for features that would improve the site. Regularly these included

the contact details of MPs and their surgery times, but also there were requests for an improved print

layout, and lists of MPs who did (not) attend a certain debate.

31 of 63

3 Site Users

Most of the information in this section is derived from an online survey to which users were randomly

invited when visiting the site (for detailed information see section C.1 Survey). This form of sampling

via an intercept survey design is notoriously difficult for obtaining samples that are representative of the

wider website audience. Therefore, what the results present is a detailed picture of characteristics and

opinions of 903 people who used the site between 22 September 2009 and 11 April 2010. While it

cannot confidently claim any formal representativeness for the wider TheyWorkForYou audience, we

believe that in general terms it is indicative of that wider audience's characteristics and opinions. For a

detailed discussion on the representativeness of the sample and possible biases please refer to section

C.3 Sample Bias / Representativeness.

One of mySociety‟s key aims is to build sites that are useful to and used by citizens from all walks of life

without exception. Therefore, this research will analyse how inclusive the audience of

TheyWorkForYou is by comparing it with the British population. However, obviously the site cannot

reach those who lack Internet access. For that reason, the data collected on the demographics of

TheyWorkForYou users will be compared to the demographics of British Internet users, in other

words this comparison will focus on how well British Internet users are represented on TheyWorkForYou.

At the same time, it is important to note that political participation in democracies has never been

inclusive. Instead, it has always been biased towards resource-rich citizens, that is those with better

education, higher income and larger social networks as section 4.1 “Political Engagement in Britain”

will outline. A simple comparison of the audience of TheyWorkForYou with British Internet users in

general fails to take these circumstances into account and is destined to simply show that

TheyWorkForYou users are not representative of the Internet population. Instead, a fair comparison

would be to focus only on Internet users who have looked up political information online, in other

words, people who have engaged in the form of political participation for which TheyWorkForYou can

be relevant. Therefore, another comparison will focus on those Internet users only, who have looked up

information online on a MP, local councillor, political party or candidate. As will be shown below, this group is

very much biased towards resource-rich parts of the population. Their demographics will be compared

with those of users of TheyWorkForYou in order to assess the potential of TheyWorkForYou to

change these persistent patterns of participation.

The data for these comparisons is provided by the Oxford Internet Survey 2009 (Dutton et al., 2009)16.

Based on a random sample of more than 2,000 people it aims to give a representative picture of all

people in Britain aged 14 years and older17. For a comprehensive comparative overview refer to the

table in section D Detailed Survey Results and Comparative Data.

16 See website of OxIS: http://www.oii.ox.ac.uk/microsites/oxis/ [17.03.2011]

17 This does exclude Northern Ireland but its population is comparatively small and there is little reason to believe the results would be

much different. Given that the basic comparator is British Internet users and the sample size of for these is 1,401, in the worst case (ie.

http://www.oii.ox.ac.uk/microsites/oxis/

32 of 63

3.1 Political Profile of TheyWorkForYou Users

Based on data from the online user survey of visitors to TheyWorkForYou, this section mainly enquires

into the question of whether users of TheyWorkForYou are already politically engaged, or whether

their use of the site is their first experience of contact with politics that may or may not constitute a

step towards more engagement.

Knowledge & information about political representatives

According to recent data by the Hansard Society (2010: 24,101:24,101) not even half the population can

correctly name their MP (44%) which is only slightly better for higher social grades (52%). Compared

to these findings the users of TheyWorkForYou are very knowledgeable: four out of five users claim

they knew the name of their MP before they used TheyWorkForYou. While this suggests a better

knowledge of basic parliamentary politics, it has not translated into a real interest for most of them as

three out of five users (60%) have never looked up information on what their representatives were

doing before they came across TheyWorkForYou. This is a large number given that - according to

OxIS 2009 - 15% of Internet users have looked up this kind of information within the last year (see

section 4.1 Political Engagement in Britain), and presumably more have done this in the past.

Figure 20: Knowledge of TheyWorkForYou users about their MP and whether with the help of TheyWorkForYou

they looked up information about representative for the first time


50% share) there will be a confidence interval of +/-2.6% (confidence level 95%) for measures derived from this data. For most

measurements the confidence interval will be smaller but for comparison to politically active people or people who have looked up

information on politicians or parties sample sizes are smaller (687 and 204 respectively) hence confidence intervals can be larger (up to 3.7

and 6.9 respectively). However, the statistical test of choice is a χ2 – test to analyse whether or not two distributions are significantly

different from each other and which will take the sample size numbers into account.

33 of 63

Involvement with politics and community

TheyWorkForYou users are more politically engaged than the average Internet user, and their political

participation18 takes place particularly often online though by no means exclusively. However, around

two out of five users have not been engaged in any other political activity apart from using the

TheyWorkForYou website.

Figure 21: Political activities online and offline of TheyWorkForYou users

Source: mySociety/UK Citizens Online Democracy user survey 2009/10; comparative data from Oxford Internet Survey (OxIS) 2009

Despite the bias from the population the site nevertheless reaches out to a large share of otherwise

politically inactive people. This is also supported by findings regarding the participation in political

and/or community groups. More than half of TheyWorkForYou users have not been involved in

organized groups. As we can see from OxIS data, this is a substantially lower degree of organization

than exhibited by the British Internet population and in particular by those Internet users who looked

for political information in the last year. However, TheyWorkForYou users show a stronger degree of

organization in political groups (such as parties, unions or civic organisation e.g. for human rights) than

in community groups. Their lower propensity to be active in community groups (such as charities,

initiatives, churches, sports clubs or volunteer organisations) as compared to the OxIS data might in

part be related to a slightly more inclusive definition of community group by the OxIS data (which also

named neighbourhood and school organizations).

18 This includes, for example, taking part in demonstrations, signing a petition, contacting a politician, boycotting a product, donating

money or displaying a campaign badge.

34 of 63

Figure 22: Membership of TheyWorkForYou users in political and community groups


In summary, while there are clearly TheyWorkForYou users who are already politically active and who

are also organized in groups, the online survey shows that one in five users (21%) has neither been

politically active (online and/or offline) within the last year nor been a member of any group and

importantly, has got information for the first time on what his or her representatives are doing through

the use of TheyWorkForYou.

3.2 Demographics of TheyWorkForYou Users

The following section reports the findings in relation to demographic characteristics of

TheyWorkForYou users. For detailed numbers please refer to section D Detailed Survey Results and

Comparative Data.

3.2.1 Gender

These days there is no gender gap for Internet use and political participation in general. Women are just

as likely to use the Internet or engage in politics as men. However, when it comes to looking for

information about politicians, men are still overrepresented. This male overrepresentation is mirrored

in the TheyWorkForYou audience where two thirds of users are male.

35 of 63

Figure 23: Gender of TheyWorkForYou users


3.2.2 Education

It is a well-established finding that better educated people are more likely to participate in politics, and

this is in particular the case for researching information about politicians: Internet users with a higher

education background are five times as likely to have looked for this type of information in the last year

(26%) as people with only a basic level of education (5.5%). TheyWorkForYou users exhibit an even

stronger bias with almost two thirds (64%) having completed a higher education degree as compared to

a quarter (27%) of Internet users and 47% of those Internet users who were checking information on

politicians in the past year.

Figure 24: Educational attainment of TheyWorkForYou users


36 of 63

3.2.3 Income

Income is problematic to measure, as many people do not want to share this information. In this survey

the income question was the one most often left unanswered with about a quarter of participants

skipping it. Nevertheless, income is an important indicator as political participation is very much

resource driven. According to OxIS, 54% of people in the highest income range (here defined as

having an annual household income of more than £40,000) have been politically active within the last

year. They are more than twice as likely to be politically active as people from the lowest income group

(up to £12,500) where only 24% have participated.

This overrepresentation of high-income groups is also apparent on TheyWorkForYou. While

intermediate income groups are underrepresented, low-income groups are fairly represented on

TheyWorkForYou.

Figure 25: Income of TheyWorkForYou users


37 of 63

3.2.4 Age

The age cohorts that are most politically active are those aged 35 to 74 where between 35% and 40% in

each group have been engaged within the last year. There are signs of increased engagement also for

those below 18 years of age (34%). In the TheyWorkForYou audience, people above the age of 54 tend

to be overrepresented while those younger than 45 are underrepresented in comparison to the British

Internet population. The findings are similar for a comparison with only those Internet users that have

checked this type of information in the last year.

Figure 26: Age of TheyWorkForYou users


38 of 63

3.2.5 Occupation

Political participation is also related to occupation, even if not very strongly. For example, the data

from OxIS shows that in the British population, only 17% of the unemployed have engaged in political

activities within the last year, while people in employment (38%), retirement (34%) or students (40%)

were more than twice as likely to do so. However, for the particular activity of checking information on

politicians, these differences are less pronounced: only sick or disabled people and home carers are less

likely to get involved. Overall, in this regard, their profile is not significantly different from British

Internet users in general.

What is noticeable on TheyWorkForYou is the high share of retired users which is twice as high as the

Internet average, and also sick or disabled people are over-represented. At the same time, home

caretakers and students are under-represented.

Figure 27: Occupation of TheyWorkForYou users


3.2.6 Ethnicity

Questions regarding the ethnic background are difficult because they measure a sensitive issue and the

usually small numbers of respondents with minority backgrounds make analysis problematic. Due to

this there is no reliable data available from OxIS on the ethnicity of people in relation to particular

political activities. The available data suggests that white people are overrepresented on

TheyWorkForYou with about 95% of the audience compared to 92% in the Internet population.

39 of 63

Figure 28: Ethnic background of TheyWorkForYou users


3.2.7 Disability

According to the OxIS data, citizens with disabilities are not more likely to participate politically, but

they are still under-represented online. While three quarters of people without a disability are online,

this is only the case for 41% of people with disabilities. There are no differences between British

Internet users in general and those Internet users who have looked up information on politicians in the

past year. In contrast, on TheyWorkForYou there is a strong over-representation of disabled people

who account for about one in five users on TheyWorkForYou, compared to less than one in ten for

the British Internet population.

Figure 29: TheyWorkForYou users with health problems and disabilities


40 of 63

4 Ecosystem

4.1 Political Engagement in Britain

The most general measure of political participation is voter turnout. This has been declining in the UK

since the early 1990s: while in 1992 almost 80% of those eligible to vote turned out, the most recent

General Election in May 2010 had only 65% of those eligible actually casting their vote (Tetteh, 2008:

18:18; BBC News, 2010). Looking beyond turnout to more specific acts of political participation, the

numbers are even smaller. According to the Oxford Internet Survey in 2009 about two thirds of the

population had not been politically active19 at all within the previous year (see detailed analysis in

section D Detailed Survey Results and Comparative Data). This leaves a third of the population that is

active to some degree and for those, signing petitions (69%), deliberately buying certain products for

political, ethical or environmental reasons (41%) and contacting politicians or government officials

(35%) are the most popular forms of participation. Relevant for the particular form of participation

that TheyWorkForYou focuses on: more than half of the population (56%) would use the Internet to

find the name of their local MP if they did not know it (Dutton et al., 2009). For Internet users, the

Internet is by far the most popular choice in this situation with three out of four Internet users (77%)

would go online to find the information. Moving on from the hypothetical situation to the real action,

within the last twelve months in total 15% of Internet users20 have looked for information about an

MP, local councillor, political party or candidate and indeed the vast majority (87%) used the Internet

to do so.

Not only are the numbers of people participating in political activities rather small, those people that do

also constitute a particular group of the population. Research into political participation has

consistently established that people who are politically engaged are heavily biased towards resource-rich

parts of the population (Milbrath 1965; Verba and Nie 1972; Parry, Moyser et al. 1992; Deth 2006). In

effect politically active people in Britain differ from the general population in a variety of important

aspects. They are better educated (50% more people with a higher degree), have a higher income (50%

more people have a household income before tax of more than £40,000 a year) on the expense of low

income groups (less than £12,500 a year) and are clearly more often organised in political groups (23%

have participated in the activities of a political group within the last year vs 11% in population). They

also tend to be in their forties or fifties (45 to 64 years old) and are focused around the employed

citizens.

The profile of participants varies in relation to the specific activity carried out. British Internet users

who have looked up information on a politician or party in the last year exhibit stronger deviations

19 Defined as signing a petition, taking part in a demonstration, deliberately buying certain products, contacting a politician, donating

money to a political or civic organisation or group, contacting a political party or joining a civic organisation or association or a political

party.

20 There is unfortunately no data available on the population as a whole.

41 of 63

from the average British Internet user than the differences found between politically active Britons and

the general population. This is in particular the case for education (47% have a higher degree compared

to 27% in Internet population), income (32% with household income above £40,000 a year vs 22% for

Internet users) and gender (male bias with 61% vs 49%) as well as the degree of political organization

(31% participated in a political group within the last year compared to 14% in the Internet population).

The potential of information and communication technologies to overcome this participation bias is

severely limited by the digital divide, as both access to the Internet and the skills to use it are not equally

distributed. It is in particular those groups that are already excluded from traditional participation that

are also more likely to be excluded from the Internet and lack the appropriate skill base (Helsper, 2008).

The combination of biased political participation and the digital divide is expected to produce even

more uneven results in terms of online political participation. As a result, several authors (Gibson,

Lusoli et al. 2005; di Gennaro and Dutton 2006; Hindman 2008) have found that online participation is

not more but may in fact be less inclusive than offline participation.

4.2 Similar Sites

It is a claim to TheyWorkForYou‟s fame that it has inspired a large number of similar sites in other

countries. In a number of cases the source code of TheyWorkForYou has been the basis of these

projects.

Table 7: Websites similar to TheyWorkForYou

Address country description

http://www.openaustralia.org/ Australia An Australian version of

TheyWorkForYou, based on the same

source code and started in November

2007.

http://openparliament.ca/ Canada A site with the same mission and

similar functionality as

TheyWorkForYou but built on Python

and Django

http://folketsting.dk/ Denmark Records of votes and speeches.

http://www.abgeordnetenwatch.de/ Germany Reports mainly on MPs and the voting

behaviour of their parties as well as

adding a feature to ask public

questions. No written record of

parliamentary proceedings.

42 of 63

http://www.nosdeputes.fr/ France Offering information on the members

of the French parliament.

http://www.kildarestreet.com/ Ireland Set up by John Handelaar with the

source code of TheyWorkForYou.

http://parlamento.openpolis.it/ Italy An ambitious effort to make accessible

the complicated workings of Italy‟s

government.

http://atviras-seimas.info/ Lithuania provides statistics for MPs

http://www.politikercheck.lu/ Luxembourg A partner website of the German site

Abgeordnetenwatch which is focused

on public questioning of MPs but also

documents some parliamentary votes.

http://www.politix.nl/ Netherlands Tracks the voting behaviour of Dutch

MPs.

http://www.theyworkforyou.co.nz/ New Zealand Effort by Rob McKinnon, who is also

a mySociety volunteer. As the name

suggests, similar to the UK version but

with some additional features, e.g. news

coverage. Set up in November 2006 but

suspended in January 2010.

http://www.alesiivoteaza.ro/ Romania A site by the Institute for Public Policy

(IPP), a Romanian non-governmental

organization, which is focused on MPs

and their activities in parliament

http://www.undemocracy.com/ United Nations Effort by Julian Todd, who is also a

mySociety volunteer, to make

accessible the written proceedings of

the United Nations Security Council

and General Assembly.

43 of 63

http://www.parliament.uk United

Kingdom

Main website of UK parliament.

Redesign in 2008 totalled £3.5m and

annual running costs of more than

£200,00021

http://www.govtrack.us/ United States Offers very detailed analysis of voting,

speeches and donations of Members of

the US congress.

A more extensive list is provided by the Wikipedia page dedicated to “Parliamentary informatics”22.

21 http://www.guardian.co.uk/politics/2008/jul/11/houseofcommons.lords1 [28.05.2010]

22 http://en.wikipedia.org/wiki/Parliamentary_informatics [28.05.1981]

http://www.guardian.co.uk/politics/2008/jul/11/houseofcommons.lords1

http://en.wikipedia.org/wiki/Parliamentary_informatics

44 of 63

5 Anecdata

Without any claim to being comprehensive, the following list provides a brief summary about some

noticeable issues and events around TheyWorkForYou:

A 2006 article by The Times23 argued that the statistics on TheyWorkForYou about the number

of times an MP has spoken in debates would result in an increase of unnecessary interventions

during debates. The reporting of these statistics was subsequently changed to indicate rough

trends (e.g. above average, etc) but still in 2008 this issue was raised by Chris Bryant,

Parliamentary Private Secretary to the Leader of the House of Commons, in a Westminster Hall

debate on 23 October 200824.

In 2007 Anne Snelgrove MP used a public parliamentary speech to remind her husband of their

approaching 30th wedding anniversary, using the fact that he is subscribed to a

TheyWorkForYou email alert on her speeches25.

In summer 2008 the volunteer effort to match up BBC video coverage of parliament with the

written transcript was a sweeping success, timestamping more than 40,000 video clips in just

twelve weeks with several volunteers contributing thousands of timestamps.

In 2008 The Daily Telegraph newspaper rated TheyWorkForYou as 41st in a list of 101 most

useful websites as “A powerful way of keeping tabs on MPs and peers: attendance records, voting patterns,

recent statements and more.”26.

In January 2008, a TheyWorkForYou widget was added to the personalised homepage of Google

(iGoogle) which was also included in http://www.google.co.uk/politics. TheyWorkForYou also

featured on the UK Google site for the General Election in 201027.

In July 2009 a story emerged in newspapers about a civil servant that had been sacked after

making comments to Communities minister Hazel Blears, suggesting she had used

TheyWorkForYou to post the comment, which caused concern about mySociety‟s handling of

user data. In fact she never posted anything on the site and the newspaper later printed a

correction28.

23 http://www.timesonline.co.uk/tol/news/politics/article735429.ece [17.03.2011]

24 http://www.theyworkforyou.com/whall/?id=2008-10-23b.139.0#g151.0

25 http://www.theyworkforyou.com/debate/?id=2007-07-11b.1548.0

26 http://www.telegraph.co.uk/technology/6554835/The-101-most-useful-websites-2008.html [17.03.2011]

27 http://www.google.co.uk/intl/en/landing/elections2010/mps/

28 http://www.mysociety.org/2009/07/06/theyworkforyou-nothing-to-do-with-this-sacked-civil-servant-story/ and

http://www.flickr.com/photos/tomskitomski/3707069444/ [17.03.2011]

http://www.google.co.uk/politics/

http://www.timesonline.co.uk/tol/news/politics/article735429.ece

http://www.theyworkforyou.com/whall/?id=2008-10-23b.139.0#g151.0

http://www.theyworkforyou.com/debate/?id=2007-07-11b.1548.0

http://www.telegraph.co.uk/technology/6554835/The-101-most-useful-websites-2008.html

http://www.google.co.uk/intl/en/landing/elections2010/mps/

http://www.mysociety.org/2009/07/06/theyworkforyou-nothing-to-do-with-this-sacked-civil-servant-story/

http://www.flickr.com/photos/tomskitomski/3707069444/

45 of 63

TheyWorkForYou produces tables of expenses together with their overall ranking among all MPs

which have been used by a number of MPs on their personal websites, for example by Harriet

Harman and Mark Hoban29.

Since March 2010 the website of the Conservative Party has used the API of TheyWorkForYou

to provide a postcode-to-constituency look-up facility.

In order to provide an elaborate election quiz on TheyWorkForYou for the general election of

2010, mySociety initiated a major volunteer effort that was instrumentally put into action by

volunteers Seb Bacon, Tim Green and Edmund von der Burg. It involved the creation of

Democracy Club30, a network of about 6,000 people volunteering to collect details about

candidates and opinions in their constituency for YourNextMP31, a database of all candidates

standing for election. The volunteers devised questions regarding local issues in their

constituency that were put to all candidates in the relevant constituency together with a number

of question on national issues devised by an independent panel of judges. Of a total of 4,156

candidates running in the election, 1,669 (40%) answered the survey questions.

In May 2010 a woman tried to stab Stephen Timms MP because of his support for the war in

Iraq. In her police interview she claimed to have used TheyWorkForYou32 to obtain information

on MPs who had supported the war.

29 http://www.harrietharman.org/my-allowances & http://www.markhoban.com/type3.asp?id=20&type=3 [17.03.2011]

30 http://democracyclub.org.uk/ [17.03.2011]

31 http://yournextmp.com/ [17.03.2011]

32 http://www.guardian.co.uk/uk/2010/nov/03/roshonara-choudhry-police-interview [17.03.2011]

http://www.harrietharman.org/my-allowances

http://www.markhoban.com/type3.asp?id=20&type=3

http://democracyclub.org.uk/

http://yournextmp.com/

http://www.guardian.co.uk/uk/2010/nov/03/roshonara-choudhry-police-interview

46 of 63

References

Alvarez RM, Sherman RP and VanBeselaere C. (2003) Subject Acquisition for Web-Based Surveys. Political Analysis 11(1): 23-43.

BBC News. (2010) Election 2010 National Results. Available at: http://news.bbc.co.uk/2/shared/election2010/results/.

Couper MP. (2000) Review: Web Surveys: A Review of Issues and Approaches. Public Opin Q 64(4): 464-494.

Dutton WH, Helsper EJ and Gerber M. (2009) Oxford Internet Survey 2009 Report: The Internet in Britain. Oxford Internet Surveys. Oxford: Oxford Internet Institute.

Escher T. (forthcoming) WriteToThem.com - Analysis of Users and Usage. London: mySociety/UK Citizens Online Democracy.

Hansard Society. (2010) Audit of Political Engagement 7. London: Hansard Society. Helsper EJ. (2008) Digital Inclusion: An Analysis of Social Disadvantage and the Information Society. London:

Department for Communities and Local Government. Reichheld FF. (2003) The One Number You Need to Grow. Harvard Business Review 81(12): 46-54. Tetteh E. (2008) Election Statistics: UK 1918-2007. In: Library HoC (ed) Library Research Paper 08/12.

London: House of Commons.

http://news.bbc.co.uk/2/shared/election2010/results/

47 of 63

Appendices

A. List of Figures

Figure 1: Screenshot of TheyWorkForYou homepage (March 2011) .................................................................................... 7

Figure 2: Articles in English language newspapers mentioning „theyworkforyou‟ (2004 – 2010) .................................... 11

Figure 3: Visits and unique visitors to TheyWorkForYou (November 2004 - December 2010) ..................................... 13

Figure 4: Visitor statistics of major mySociety sites (September 2008 - December 2010) ................................................ 14

Figure 5: Time of day for visits to TheyWorkForYou (2010) ................................................................................................ 15

Figure 6: Work-related usage of TheyWorkForYou (from survey) ....................................................................................... 15

Figure 7: Use frequency of TheyWorkForYou (based on responses to user survey) ......................................................... 16

Figure 8: Where visits to TheyWorkForYou are coming from (2006 - 2010) ..................................................................... 17

Figure 9: How users of TheyWorkForYou find out about the site ....................................................................................... 18

Figure 10: Visits from UK parliament to TheyWorkForYou (2010) .................................................................................... 20

Figure 11: Distribution of page impressions to content sections on TheyWorkForYou (2010) ...................................... 21

Figure 12: Traffic to House of Commons related content (2010) ......................................................................................... 22

Figure 13: Usage motivations of TheyWorkForYou users ..................................................................................................... 24

Figure 14: Signup rates to TheyWorkForYou site and email alerts as well as debate annotations (2009 - 2010) .......... 25

Figure 15: Development of volunteer video timestamping on TheyWorkForYou (June 2008 – September 2010) ..... 26

Figure 16: Traffic to TheyWorkForYou and election quiz during election campaign ....................................................... 27

Figure 17: Comparing usage frequency of successful and unsuccessful users ..................................................................... 28

Figure 18: Net Promoter score, i.e. likelihood of users to recommend TheyWorkForYou ............................................. 29

Figure 19: User assessment of TheyWorkForYou usability and use effects ........................................................................ 30

Figure 20: Knowledge of TheyWorkForYou users about their MP and whether with the help of TheyWorkForYou

they looked up information about representative for the first time ...................................................................................... 32

Figure 21: Political activities online and offline of TheyWorkForYou users ....................................................................... 33

Figure 22: Membership of TheyWorkForYou users in political and community groups .................................................. 34

Figure 23: Gender of TheyWorkForYou users ......................................................................................................................... 35

Figure 24: Educational attainment of TheyWorkForYou users ............................................................................................. 35

Figure 25: Income of TheyWorkForYou users ........................................................................................................................ 36

Figure 26: Age of TheyWorkForYou users ............................................................................................................................... 37

Figure 27: Occupation of TheyWorkForYou users ................................................................................................................. 38

Figure 28: Ethnic background of TheyWorkForYou users .................................................................................................... 39

48 of 63

Figure 29: TheyWorkForYou users with health problems and disabilities .......................................................................... 39

Figure 30: Screenshot of teaser inviting visitors to TheyWorkForYou to participate in online survey .......................... 50

Figure 31: Start of questionnaire ................................................................................................................................................. 51

Figure 32: Responses to survey questionnaire over time ........................................................................................................ 56

49 of 63

B. List of Tables

Table 1: Coverage of TheyWorkForYou ..................................................................................................................................... 8

Table 2: Timeline of major steps in TheyWorkForYou development (2004 – 2010) ........................................................ 10

Table 3: Development and maintenance cost of major mySociety projects ........................................................................ 12

Table 4: Top 5 referring websites to TheyWorkForYou (2007 – 2010) ............................................................................... 19

Table 5: Most popular pages in selected content sections (2010) .......................................................................................... 22

Table 6: Top 3 video of debates most often requested from TheyWorkForYou (2008 – 2010) ..................................... 23

Table 7: Websites similar to TheyWorkForYou ....................................................................................................................... 41

Table 8: Response rate for user survey as of 20 April 2010 .................................................................................................... 52

Table 9: Patterns of political participation in Britain compared to TheyWorkForYou users ........................................... 59

50 of 63

C. Methodology

C.1. Survey

The user survey on TheyWorkForYou was started on 22 September 2009 and employed an intercept

survey design (Couper, 2000: 485p:485p). A special procedure would randomly invite about 20%33 of

visitors to the survey by a teaser asking “Did you find what you were looking for?” (shown in Figure 30 below).

In order to keep the survey as unobtrusive as possible it was decided not to force a reaction to the teaser so

users could continue navigating the site whilst the teaser was displayed. The selection procedure ensured

that after either answering the questionnaire once or declining to participate, there would be no further

invitations (based on a cookie).

Figure 30: Screenshot of teaser inviting visitors to TheyWorkForYou to participate in online survey

If the invited visitors to the site selected either Yes or No to the teaser question (rather than ignoring it)

they would be forwarded to a single web page containing the questionnaire, of which a part is shown in

Figure 31 below (for the complete questionnaire see C.2 Survey Questionnaire). Users were not required to

fill in the survey, nor did they have to answer all questions if they chose to participate.

33 From 22 September until 20 October 2009 the sampling rate was 10% which was then increased to 20% of all visitors in order to collect

more responses.

51 of 63

Figure 31: Start of questionnaire

The user survey was designed in order to obtain a more detailed picture on the demography and political

engagement of TheyWorkForYou users and to gather information about their experience using the site.

Data was collected from 22 September 2009 until 11 April 2010. The survey was then suspended on 12

April when the Westminster parliament was dissolved for the General Election, in order to separate the

data from the election campaign.

52 of 63

Table 8: Response rate for user survey as of 20 April 2010

teaser shown on site (visits to site, i.e. does not equal unique

visitors; only visitors who have not responded to teaser earlier)

235,149

followed invitation (responded to teaser) 12,333 (~5%)

cooperation rate (submitted questionnaire) 1,130 (~9%)

completion rate (defined as having missed not more than four out

of eleven essential variables34)

1024 (~91%)

response rate I

(completed surveys from total number of times teaser was

shown)

response rate II

(completed surveys from total number of people who

responded to teaser)

>0.4%

(lower bound as repeat

visitors will have been

shown teaser several times)

8%

Number of responses used in analysis (only those completed before 12

April 2010, i.e. dissolution of parliament for General Election)

903

The questions that people most often refused to answer were35:

1. income (26%)

2. opinion on representative (23%)

3. opinion on design of site (17%)

4. opinion on objectivity of site (14%)

5. opinion on change in knowledge about representative (13%)

6. location (13%)

7. usefrequency (10%)

8. ethnicity (10%)

9. opinion on structure of site (9%)

10. opinion on navigability of site (8%)

11. education (8%)

12. gender (7%)

13. disability (6%)

34 Essential variables concerned netpromoter, referrer, groups, activity, age, gender, education_any, lifestage, income, ethnicity, disability.

35 Percentage based on people that completed the questionnaire (ie. not more than 4 missing variables)

53 of 63

This result is somewhat interesting if compared to the findings on WriteToThem (Escher, forthcoming).

While not surprisingly most users are hesitant to answer the income question, the reluctance to share

details on frequency of use and the opinion about several aspects of the site are hard to explain. It is

possible that the question on opinions about the site posed some usability challenges. In general, the rather

large number of missing variables points to the fact that users were less motivated to answer the

questionnaire.

C.2. Survey Questionnaire

The complete text of the survey follows below (for a screenshot see Figure 31). The design was very

simple, all the questions were displayed on a single webpage and users were free to leave questions

unanswered.

Within the last twelve months: How often have you used TheyWorkForYou?

never, this is the first time in the last 12 months

about once within the last 12 months

2 - 5 times within the last 12 months

6 - 10 times within the last 12 months

roughly every month within the last 12 months

roughly every week within the last 12 months

roughly every day within the last 12 months

don't want to answer

Before you used this site, did you know who your Member of Parliament in the House of Commons was?

Yes

No


Before you used TheyWorkForYou did you ever look up information on what your representatives were doing?

Yes

No


How would you describe your latest use of this site?

just generally browsing out of interest/curiosity

obtaining information on my representative

obtaining information on a particular debate

checking a particular fact

keeping an eye on what (my) representatives do

other (please specify)


Is your use of this site in any way related to your work?

Yes

No


Are you a registered user of TheyWorkForYou? (This means do you have a log-in that allows you to add annotations or contribute to the glossary?)

Yes

No


How likely is it that you would recommend this site to a friend or colleague? (assuming they would be interested in such a service)

from 0 (not at all likely) to 10 (extremely likely)

feel free to tell us why


54 of 63

How did you find out about this site? from another mySociety site

from media such as newspapers, etc

from a search engine (e.g. Google or Yahoo)

recommendation by friends or colleagues

from a campaigning website

used this site before

other (please specify)

can't remember


Is there any feature that you would like to see on the site for which you would be willing to donate some money so that it can be developed?

free text

How much do you agree with the following statements?

possible options: strongly disagree, disagree, agree, strongly agree, don‟t want to answer

TheyWorkForYou is easy to navigate

TheyWorkForYou is well structured

TheyWorkForYou provides information in an unbiased and unpartisan way

TheyWorkForYou is pretty to look at

TheyWorkForYou has improved my knowledge about my representative

TheyWorkForYou has improved my opinion about my representative

In the last twelve months have you been involved with a political or a community group, e.g. by being a formal member or by volunteering?

a political group (e.g. a party, an union, a civic organisation e.g. for human rights)

a community group (e.g.a charity, an initiative, a church, a sports club, a volunteer organisation)

both community as well as political group(s)

none of the above


Apart from your use of this website: Within the last twelve months have you taken part in any broadly political activity? (This includes for example demonstrations, signing a petition, contacting a politician, boycotting a product, donating money or displaying a campaign badge)

yes, online

yes, offline

yes both online as well as offline

none of the above


How old are you? less than 18 years old

18-24 years old

25-29 years old

30-34 years old

35-39 years old

40-44 years old

45-49 years old

50-54 years old

55-59 years old

55-64 years old

60-64 years old

65-69 years old

70-74 years old

75 years and older


55 of 63

Could you please indicate your gender? female

male


What is the last type of educational institution (e.g. school, college or university) that you have attended or which type of educational institution are you attending now?

Primary school or equivalent

Secondary school or equivalent

Special school or equivalent

Sixth form college or equivalent

Technical college or equivalent

Further Education College

Adult Community College

University or equivalent

other


Which of these descriptions best describes your current situation?

working full time (30 hours a week or more)

working part time (8-29 hours a week)

retired

unemployed

permanently sick or disabled

in community or military service

undergraduate student

postgraduate student

in full time education (not degree or higher)

in part time education (not degree or higher)

doing housework, looking after children or other persons

none of the above


The incomes of households differ a lot in Britain today. Which figures best represents the total income of your household before tax?

up to £12,500

£12,501 to £25,000

£25,001 to £37,500

£37,501 to £50,000

£50,001 to £75,000

£75,001 to £100,000

more than £100,000


To which one of these ethnic groups do you consider you belong?

White o British o English o Welsh o Scottish o Irish o Other white

Mixed o White and Black Caribbean o White and Black African o White and Asian o Other Mixed

Asian or Asian British o Indian o Pakistani o Bangladeshi o Other Asian

56 of 63

Black or Black British o Caribbean o African o Other Black

Chinese

any other ethnic group

don't know


Do you have a health problem or disability which prevents you from doing every day tasks at home, work or school or which limits the kind or amount of work you can do?

Yes

No


Do you have any other comments (e.g. on the survey, on your usage, etc)?

C.3. Sample Bias / Representativeness

This survey relies on the cooperation of TheyWorkForYou users. As there was no other means available of

contacting the whole audience of TheyWorkForYou, the chosen method to invite users to participate in

this survey was via a teaser shown on the site. This is always the least preferable option as many people will

simply overlook it, others will be annoyed even though it was designed to minimise disruption, and

frequent users of the site have a higher chance of being invited to the survey. Last but not least, even

though it was recorded how often the teaser was shown, there is no way to determine exactly how many

people (as opposed to visits to the site) have been presented with the choice to participate in the survey.

Recruiting of participants is even more difficult because as the analysis shows most users come to

TheyWorkForYou only for a short time, looking for a particular fact. They leave the site very quickly and

can hardly be expected to be motivated to spend up to five minutes doing a survey.

Figure 32: Responses to survey questionnaire over time


57 of 63

The overall response rate is at least 0.4%. This value is a lower bound as the teaser is likely to have been

shown multiple times to the same person visiting the site repeatedly. What is more, it is fair to assume that

many people simply did not notice the teaser as it was designed to disrupt browsing as little as possible. Of

those that actually got to the questionnaire because they answered the question in the teaser (Did you find

what you were looking for?) about 8% completed the questionnaire. There is little comparative data available

but an experimental study of Alvarez et al (2003: 31:31) which used a website banner to recruit participants

to a survey reported an overall response rate of 0.017% (compared to 0.4% in this survey), a click rate of

0.3% (compared to 5% in this survey) and a conversion rate from clicking the banner to participating of

5.6% (compared to 8% in this survey). The figures reported for the TheyWorkForYou survey compare

favourably to these numbers though it has to be kept in mind that the Alvarez study recruited subjects on

third party sites and the impressions probably did not account for automated queries.

The response rate itself is just an imperfect proxy measure for how representative the sample is for the

whole TheyWorkForYou user population. The goal of every survey is to obtain a sample representative of

the population (in this case of the users of TheyWorkForYou). As long as all people are equally likely to

participate and the surveyed time frame is long enough to account for variations throughout the survey

period the actual response rate is not really a problem. It is, however, possible that certain types of users

are more (or less) likely to respond to the survey which would introduce a bias in the sample and hence

into the results. The representativeness of the sample is in particular relevant for demographic questions, in

order to see how much TheyWorkForYou users differ from the British Internet population according to

the nationally representative sample from OxIS 2009. For the assessment of the site demographic

characteristics such as age and gender can still play a role but probably less so.

There is no data collected on the website about the users, hence there is a lack of reliable indices that might

allow some assessment of the representativeness of the sample. However, there are a number of weak

estimators. For example until 20 April 2010 of all 12,333 users who responded to the teaser 59% found

what they were looking for which is a more or less consistent rate throughout the sampling period36. By

contrast, in the survey37 this is substantially more with 80% of users finding the relevant information. It

should perhaps not surprise that those “successful” users are more inclined to answer the survey.

Subsequently it can be expected that those users answering the survey are more satisfied than the general

user. What is more, the analysis has shown that those successfully using the site differ significantly in their

demographic characteristics from those who fail to find the relevant information on the site.

Another estimator is the amount of frequent usage. According to the web stats, in any given year around

two thirds of visits to TheyWorkForYou are first time visits. This is an upper limit for the actual share of

people using TheyWorkForYou for the first time because the number of first time visits is bound to be

inflated given that people use different computers and browsers or might delete their cookies, all resulting

in their visit being counted as a first time visit. It seems not unreasonable to expect that around 50% of

users of TheyWorkForYou are regulars who have used the site at least twice within the last year. In the

36 Data is available for 27 November and 20 October 2009 when the rate was 56% each time.

37 That only takes into account responses up to 11 April 2010 but as highlighted the rate seems to be quite consistent.

58 of 63

sample four out of ten respondents said they were visiting the site for the first time. This is not a huge

deviation from the web stats, in particular as the sampling method is bound to produce samples that

include more frequent users of the site as these have a higher chance of being invited to the survey. Taken

together, this signals that, at least on this measure, the sample seems to have reached a broad share of the

TheyWorkForYou audience and not just a selective share. Also there is a rather constant share of about

two thirds of all visits arriving via search engines on TheyWorkForYou. Many of those people arriving via

search engines do already know about the site, as around 10% of searches use TheyWorkForYou or a

combination of these words in their search. This means the number of people who genuinely find out

about TheyWorkForYou for the first time by using a search engine is lower. On this assumption, the share

of 50% of users who indicated in the survey that they found out about the site from a search engine is not

too far off.

Overall it cannot be assumed with any confidence that the sample is representative of the entire audience

of TheyWorkForYou. With confidence the results can be used to assess the characteristics and opinions of

a share of the TheyWorkForYou audience only but it can be assumed that based on the few imperfect

proxy measures above the bias of the sample might not be too extreme.

59 of 63

D. Detailed Survey Results and Comparative Data

Based on the Oxford Internet Survey 2009 (Dutton et al., 2009) the following table summarises the main

socioeconomic indicators of a) the British population, b) British Internet users, c) those British people who

said they have carried out at least one political activity within the last year and d) those Internet users who

have looked up information (online and/or offline) about an MP, local councillor, political party or

candidate38. The table also reports the respective results from the survey of TheyWorkForYou users.

Table 9: Patterns of political participation in Britain compared to TheyWorkForYou users

British population

Internet users politically active

citizens (in last year) a)

Internet users who looked up information on politician/party (in last year)b)

TheyWork ForYou users

Source OxIS 2009 OxIS 2009 OxIS 2009 OxIS 2009

mySociety online user survey

Number of cases c) 2,013 1,401 687 204 903

all values as percentage of respective population

Gender male

48

49

50

++ 61

**/++ 66

Education d)

higher education

20 ** 27

** 30

++ 47

**/++ 64

Monthly gross income household ≤ £12,500 £12,501 - £40,000 > £40,000

26 58 16

** 14 64 22

** 18 58 24

++ 14 55 32

**/++

16 42 (- £37,500) 42 (>£37,500)

Age < 18 years 18-24 years 25-34 years 35-44 years 45-54 years 55-64 years 65-74 years ≥ 75 years

6 10 18 18 16 12 12 8

** 8 12 22 22 18 10 7 2

* 6 9 14 19 20 14 12 5

4 12 21 20 21 13 7 2

**/++ 2 8 16 18 19 20 12 5

Occupation employed retired unemployed sick/disabled home caretaker student

49 23 8 3 8 9

** 61 11 6 1 9 13

** 54 23 4 3 6 10

64 13 6 1 4 13

**/++ 57 22 4 5 3 9

Ethnicity White Asian/Asian British Black/Black British other ethnic group

93 3 3 1

93 3 3 1

93 3 4 1

++ 85 7 6 3

**/++ 95 2 1 2

Disability 15

** 9

15

8

**/++ 20

38 The Oxford Internet survey provides this information only for Internet users who have looked up information (online and/or offline) about

an MP, local councillor, political party or candidate within the last year.

60 of 63

Active in groups (in last year) e)

no a community group a political group both

49 40 2 9

** 43 44 2 11

** 31 47 4 19

++ 29 40 4 27

**/++ 51 26 11 11

Pol. participation in last year a)

no activity offline online online & offline

66 19 4 11

** 60 19 6 15

-

56 13 31

++ 31 19 9 41

f) **/++ 43 10 18 28

Source: Oxford Internet Survey (OxIS) 2009, except „TheyWorkForYou users“: mySociety/UK Citizens Online Democracy user survey

2009/10

Note: ** p≤0.01; * p≤0.05 (based on χ2-test, always in relation to British population); ++ p≤0.01; + p≤0.05 (based on χ2-test, always in

relation to British Internet population);

a) political participation was defined in OxIS as signing a petition, taking part in a demonstration, deliberately buying certain products, contacting a

politician, donating money to a political or civic organisation or group, contacting a political party or joining a civic organisation or association or a

political party;

b) based on the following question (to Internet users only): “Have you done any of the following in the past year: Look for information about an MP,

local councillor, political party or candidate?”;

c) indicates number of cases in base category. Number of cases for individual items may be smaller due to nonresponse on particular variables

d) excluding those respondents who are currently students except those on a postgraduate degree

e) based on the following OxIS questions: for political groups: “Do you participate to the activities of: A trade union, an environmental or animal

welfare organisation, any other political or campaigning organisation?”; for community groups: “Do you participate to the activ ities of: any social or

sport club, a residents, neighbourhood, school or other local group, a charity organization or social aid organisation, religious or church organisation?”

f) apart from their use of TheyWorkForYou;

61 of 63

E. mySociety – a short summary39

mySociety is a charitable organisation and an open source community which work together to build the world's most popular and effective democracy and transparency websites. Despite directly employing only 12 people and focussing primarily on the UK, mySociety's websites were used by over 5 million people in the last year. Those citizens undertook practical, meaningful activities as varied as getting potholes fixed to delivering messages from individuals appealing against extradition from the UK. Our purpose is to deliver simple, tangible benefits for citizens, using the internet. What we do mySociety builds websites like TheyWorkForYou.com (parliamentary transparency), WriteToThem.com (for contacting politicians) and FixMyStreet.com (report a problem in your street to your local council). The best way to understand what any of our websites does is to visit it: they are often single-purpose, and by trying it out, the user should immediately grasp what that purpose is. Consequently, our sites tend to have a minimalist, unfussy feel. That reflects our obsession with making our tools as usable as possible for people who have never engaged before, never before written to a politician or never examined a voting record. Our metrics suggest we succeed at these goals, with 50% first timers being usual. This means that each year we are engaging enormous numbers of people in the UK's democratic system for the first time, fixing problems in their lives at the same time. Everything we build is open source software. This means that our ideas and technology can be copied and re-used in democracies across the world, from the Ukraine to New Zealand. We are a virtual community with a core of a few increasingly-renowned paid developers, and a much larger group of volunteers who do everything from code entire websites to putting up posters in workplaces. Recently a group of non-technical volunteers (including one government minister) helped chop up and time-stamp 37,000 video clips of Parliament in just a few weeks. With over 2.5 million visitors expected this year, this video addition will make a real difference to showing citizens what is happening in their name in Parliament. Why we do it Elections in most countries happen only every few years. In between those elections there are innumerable decisions made by government that the public has a right to have a say in. mySociety believes that appropriate uses of technology can help people to engage, and to get the things changed in their society that they want changed. We are not interested in direct democracy - we're interested in patching and upgrading the democracy we've already got. mySociety also believes that the institutions of government and civil society need help understanding quite how substantially the internet can be used to make change for the better. We therefore exist not just to build sites directly for members of the public, but we strive to help often recalcitrant institutions realise what's best for them and their users. When we succeed, the results can be spectacular - the previous Prime Minister's office commissioned us to build a petitions website that went on to become the biggest non-partisan democracy website in the history of the world (over 8 million signatures so far). mySociety is politically neutral. It is important that everything we do is accessible to all, no matter what political allegiance. TheyWorkForYou deals with facts, not opinions.

39

For an up-to-date version see http://docs.google.com/Doc?id=dcqnzzt5_39dzqrw2hr

62 of 63

History mySociety was founded in late 2003, building on the foundations of a small volunteer community that had been around for several years prior. It was founded by Tom Steinberg, a policy analyst in the UK government who was unusually familiar with technology in having been a systems administrator prior to his policy career. He was therefore able to see that the work of this small volunteer community was more far more meaningful and empowering to most citizens than the bulk of the multi-billion pound 'eGovernment' agenda, and set about building an organisation that could replicate and accelerate their work. That project proceeded with funding from every source possible, foundations, individuals both wealthy and not, government departments and eventually business commissions from the likes of Google, the BBC and No10 Downing Street. By its 5th birthday, mySociety had built an international reputation as the most successful engineers of impactful non-partisan websites in the world. In 2007 the UK government asked Tom Steinberg to write its new information strategy, the Power of Information Review, and implemented almost all of the recommendations. At around the same time the then Prime Minister included a whole chapter on mySociety in his book Britain's Everyday Heroes. Organisational Structure mySociety is an open source organisation that breathes the methods it preaches. We have no offices, to help keep overheads down to extremely low levels, and organise ourselves using a variety of open source internet technologies, such as email lists, chat rooms, wikis and so on. Around three quarters of the core of the organisation are unpaid, and a meritocratic system tends to mean that the people who do the most work end up having a larger and larger say over where the organisation goes, and what projects get worked on. We are a registered charity in the UK, which wholly owns a company so that we can do work for organisations wanting our skills, with the profits being returned to the charity. Future MySociety's stated aims are to build websites that give people simple, tangible benefits in the civic and community aspects of their lives; and to teach the public and voluntary sectors, through demonstration, how to use the internet most efficiently to improve lives. Both predicate a future in which mySociety has a permanent place, always innovating, and always developing. With the future of web technologies growing at such a pace, and society shaping itself to adapt to new possibilities, providing online access to civic processes will continue to be a substantial challenge, and one that mySociety will happily rise to. Our open source approach and agile methodology, together with our dispersed workforce, allows for some interesting possibilities, particularly when it comes to working with organisations overseas. In the past, we have not always been able to extend much assistance (beyond access to our code) to those expressing an interest in setting up similar projects abroad – not that this has stopped the launching of many comparable projects, from a Catalan FixMyStreet to a Lithuanian WriteToThem.

63 of 63

This state of affairs looks set to change in 2011, as for the first time we employ paid full-time staff abroad. Thanks to funding from the philanthropic Omidyar Network, two mySociety employees will be bringing the principles of TheyWorkForYou to the challenging and complicated political landscapes of Kenya and Nigeria. Our other major project for this year will be the launch of FixMyTransport, a UK specific website which takes the principles of FixMyStreet and applies them to public transport. By providing an easy interface through which passengers can report problems with their journey, we hope to provide one more channel through which citizens have the power to make improvements in their own everyday life. Like FixMyStreet, FixMyTransport will rely on the power of transparency and crowd-sourcing to get stuff done. You can read more about it here: http://www.mysociety.org/fixmytransport. mySociety operates as an open forum, within which any member is welcome to suggest new ideas for projects. With the majority of the organisation being made up of developers who understand the availability, restrictions and above all the possibilities of open data, this can lead to innovative, and surprising, ideas. The best of those ideas - time and money allowing - get built. Key Details Director: Tom Steinberg Email: [email protected] phone: +44 7811 082158 web: www.mysociety.org Marketing and Communications Manager: Myfanwy Nixon Email: [email protected] Phone: +44 7976 961845

http://www.mysociety.org/fixmytransport


http://www.mysociety.org/


analysis of users and usage for uk citizens online democracy · 2014-03-27 · this report was...

Documents