valuation office agency experience of automated valuation modelling (avm) in england tim eden bsc...

Valuation Office Agency

Experience of Automated Valuation Modelling (AVM)

in England

Tim Eden BSc MRICS IRRV

Deputy Director of Council Tax



Presentation overview

A “fly through” of the VOA AVM experience including: Background The challenge The extent of data capture Changes in data management Process improvement and change Model development Training Achievements


To be a world class organisation providing valuation and property

services for the public sector


VOA - Our purpose

To provide a fair and robust basis for taxes, which help to pay for public services; and to help drive better use of property in the public sector by: compiling and maintaining accurate and comprehensive valuation lists for

local taxation providing accurate valuations for national taxes delivering expert advice on valuations and strategic property management developing and maintaining a comprehensive

and up to date property database advising policy makers on valuation and

property issues


VOA - Our structure

Country VOA VOA Deals with: -

No. of Offices

Local Domestic Property Taxation

Local Non-domestic

Property Taxation

National Property Taxation

Other Government

Property Services

England 67 Yes Yes Yes Yes

Wales 7 Yes Yes Yes Yes

Scotland 5 No No Yes Yes


“Recent” Taxation History1956 General National Valuation – to Point Rental Value

1963 General National Revaluation

1973 General National Revaluation

1982 General National Revaluation (Cancelled)

Non-Domestic Domestic

1990 National Revaluation with Antecedent Valuation Date

(AVD) 1 April 1988

Individual-based Taxation called Community Charge (“Poll Tax”)

1993 National Property Valuation for Council Tax to Capital Value

Bands (AVD 1 April 1991)

1995 Revaluation AVD 1 April 1993



2007 Revaluation AVD 1 April 2005 (Postponed)


Council Tax (CT) - Context

CT + Non-Domestic Rates together raise c. £40bn pa

CT raises £18.5bn pa 22m domestic properties in England 1.3m domestic properties in Wales Clients

Department for Communities & Local Gov’t (DCLG) in England

Welsh Assembly Gov’t (WAG) in Wales


CT – 1993 List Bands (England)

A - up to £40,000 B - £40,001 to £52,000 C - £52,001 to £68,000 D - £68,001 to £88,000 E - £88,001 to £120,000 F - £120,001 to £160,000 G - £160,001 to £320,000 H – over £320,001


CT - Revaluation

Local Government Act 2003 included requirement to undertake revaluations in Wales (2005) and England (2007)

In Wales AVD 1 April 2005; list came into force on 1 April 2005

In England AVD 1 April 2005; list was to come into force on 1 April 2007

Draft lists were to be published 1 September 2006


The Challenge

New banding scheme not known Necessary to produce individual

valuations & then overlay new bands Scale of task Complexity of housing types Vagaries of the market Inconsistent, paper-based records


The Challenge – Market Vagaries Market analysis across England

Free flow of money Historically low interest rates Fluctuating volumes in sales

Sales Relative to Same Month in Previous Year - Aug '05 Data

0%

20%

40%

60%

80%

100%

120%

140%

160%

Jul-04 Aug-04 Sep-04 Oct-04 Nov-04 Dec-04 Jan-05 Feb-05 Mar-05 Apr-05 May-05 Jun-05

Month

Pe

rce

nta

ge

Birmingham

East Anglia

East Midlands

Leeds

Liverpool

London-City

London-North

London-South

London-Westminster

Manchester

Newcastle

North West

Reading

Sheffield

South East

South West

St Albans

Wessex

West Midlands

Western

England Total

Sales Relative to Same Month in Previous Year - Aug '05 Data

0%

20%

40%

60%

80%

100%

120%

140%

160%

Jul-04 Aug-04 Sep-04 Oct-04 Nov-04 Dec-04 Jan-05 Feb-05 Mar-05 Apr-05 May-05 Jun-05

Month

Pe

rce

nta

ge

Birmingham

East Anglia

East Midlands

Leeds

Liverpool

London-City

London-North

London-South

London-Westminster

Manchester

Newcastle

North West

Reading

Sheffield

South East

South West

St Albans

Wessex

West Midlands

Western

England Total


The Challenge – Housing Types Profile of Housing Stock

80% Houses/Bungalows 20% Flats/Maisonettes Over 1/3rd of all flats in Greater London Around 1/6th of properties at least 100 years old BUT high proportion of new properties in sales evidence!

National Age Profile

0.00%

2.00%

4.00%

6.00%

8.00%

10.00%

12.00%

14.00%

16.00%

18.00%

<1900 -1918 -1929 -1939 -1954 -1964 -1972 -1982 -1992 -1999 >1999

Age (Era)

Co

un

t (

wh

ere c

od

ed

)

National Age Profile

0.00%

2.00%

4.00%

6.00%

8.00%

10.00%

12.00%

14.00%

16.00%

18.00%

<1900 -1918 -1929 -1939 -1954 -1964 -1972 -1982 -1992 -1999 >1999

Age (Era)

Co

un

t (

wh

ere c

od

ed

)


The Solution!

Develop an AVM Skill-up staff Learn from others

Cole Layer Trumble (CLT) – computer modelling expertise, AVM software

KPMG – programme & project management support

IAAO – statistical training Digitise data


Began in 2002; worldwide research Mass data capture/enhancement Model development Many innovations and lessons

These aspects have been very important to the success of the project

• Improving Business Decisions

• AVM design

• MRA models

• data collection • 22 million values• data cleansing

• data enhancement• business case• political issues

• value reviews

• staff planning and allocation

VOA - AVM Journey


Attribute Data Investigation

Investigation commences in 2002 considering: Use of existing “attribute” codes already available on VOA IT

database (“Group”, “Type”, “Area”) Valuer and caseworker engagement to determine useful

codes Data availability & maintainability

CT List Maintenance work External standards e.g. IAAO (6 year cycle) Impact of differing local practices

Ability to undertake sales investigation activity

Pre-contractor appointment; so limited AVM expertise


Main Attributes to be digitised

Architectural Style and construction quality – “Group” Property Type e.g. House Semi-Detached – “Type” Age (coded as era e.g. G – 1965-72) Area m² - external area (houses); internal area (flats) No. of rooms No. of bedrooms No of bathrooms Number of floors (houses); floor level (flats) Parking Conservatory (type and size) Outbuilding details Value Significant Codes


What about Sales Data?

Access to Land Registry and Stamp Duty Land Tax information

Address irregularities – creates matching problems and manual effort

Need to establish whether transaction at “Market Value”, special circumstances etc.

Data understanding significantly assists AVM geared to support this process “Value” understanding working with AVM is key

to the process


Process Development

Consider existing process The impact of AVM – scale and profile Skills

Existing skills and abilities of targeted staff The need to re-train certain staff

Resource availability New process to reflect Revaluation needs

and business as usual IT constraints and capabilities Understand relationship between sales and

model performance


Before

Sales received

Suspect sales coded

Sales searched to support each valuation case “on paper”

At a Revaluation “Beacon Sales” reviewed and recorded “on paper”

Property Data

(Group, type, floor area, age, bedrooms,

rooms, garage)

Property Data

(Group, type, floor area, age, bedrooms,

rooms, garage)

Models

(MRA, algorithms

and variables)

Models

(MRA, algorithms

and variables)

Locality Definition

(area from which

comparables derived)

Locality Definition

(area from which

comparables derived)

AVM

AVM analysis output

reviewed by

qualified staff

AVM analysis output

reviewed by

qualified staff

Decision to change, retain or include current

data and coding

Decision to change, retain or include current

data and coding

Sales Data

1)New Sales continually updating

analysis set

2) Value significance considered

Sales Data

1)New Sales continually updating

analysis set

2) Value significance considered

After

The Analysis Process


Initial Issues with Modelling Process

No first hand knowledge of sale Data time lag Data gaps due to

“Permissible development” – the public need not inform the Billing Authority (BA)

BAs internally not acting in a joined-up way – billing, planning & building control

BAs not acting in joined-up way with VOA Are we maintaining lists or data or both?

“Condition” not an attribute


Integrating the AVM

Investigation work commenced late 2002 Procurement process 2003 Collaboration of in-house team and

successful contractors/partners - Cole Layer Trumble (CLT)

Working on BA areas where data capture is advanced (note had only started in 2003!)

Challenge of integrating AVM technology with existing VOA IT systems


Getting Started

Started with a simple additive model £ = B0 + B1 * size + B2 * Detached * Size + B3 * Terraced *

Size + … + Bn * Date of Sale * Size

1993 Band initially used to support analysis, but quickly stripped out of valuation models

Postcode sectors used as proxy for location – VOA developed “localities” Postal areas are too crude Designed to support mail delivery, not to reflect influences

on the property market!


Performance Improvement (1)

Better understanding of subject/sale data helped to: Determine usability rules for raw data Determine usability rules for a sale

This understanding fed Comparable selection Model specification Consideration of rarely occurring variables Consideration of locality relationships


High Correlations

Group 55 separate codes

Age11 separate codes

Type

32 separate codes

Parking

Floors/

Floor Level

BedroomsRooms

AreaBathrooms

Correlation among variables

High correlation compromises

modelling stability


Localities – illustrative photograph

Local Authority housing estate

Local Authority housing estate

Privately built housing estate

Privately built housing estate

Created bespoke localities (neighbourhoods)


Mapping & Localities

In excess of 10,000 localities Regular boundary review required Thematic mapping as part of process X-Y co-ordinate becomes necessary

data X-Y suitable for comparable selection Issue of who maintains the data

Valuation Office AgencyMapping Example



Move to multiplicative (log linear) model structureLog (£) = B0 + B1 * log (size) + B2 * Detached + B3 * Terraced

+ …

+ Bn * Date of Sale + Bp * Log (Locality Adjustment Factor)where Bi determined from sales set using MRA

This enabled improvements in: - Locality Adjustment Factor (time adjusted median

price per square metre) Locality Grouping – support for comparable selection Central Modelling across the whole country


Benefits of Central Modelling

Central Modelling enabled: Central recognition of national modelling

patterns and data issues Modelling “constraints” could be imposed

by the centre ensuring model coefficients are consistently applied

Effective direction of effort Calculation of market trend information to

support VOA modelling and wider government market appreciation



Integrating X-Y & Mapping allowed mass calculation of “plot size”

Issues with map plots are: Plot bleed Non-alignment with the property

transferred Maintenance of data


Model Development - Lessons

Create a multi-skilled and focused R&D team Select several representative areas to test Ensure proper debate on proposals Predict overall likely modelling gains Sense check & gather feedback from local

staff Business model to relate model performance

and cost/benefit Produce individual cost/benefit for each

proposal Promote external verification e.g. IAAO


Supporting Decision Making (1)

VOA recognised management of a national valuation delivery required consistency in valuation decisions

Property level confidence score required COD/COV inadequate existing banding too remote

Score needed to support decisions at several levels: Strategic:

Business case development Process definition Data collection & enhancement

Tactical: Resource planning

Operational: Resource allocation Value review or band review


Supporting Decision Making (2)

Confidence estimate for each subject property needed to reflect the aspects which would reduce accuracy:

Data quality Data availability Market variability Model accuracy

Adequate coverage for accuracy in the MRA Adequate comparables


Confidence Model Confidence model based upon indicators from MRA and comparables Related actual errors to the dispersion amongst the comparables Comparable sales approach provides a number of other indicators for

confidences: comparability distance weighted estimate (average adjusted) vs. MRA estimate the overall COV for the model on the sales set which tells us something

about the underlying uniformity in the market and sales base MRA modelling includes a control model which considers current CT band So Confidence Model is:

Likely Error = A+ B dispersion of comparables+ C x average distance between comparables+ D x absolute value of (ln (mkt est. / control model est.) ) + E x model standard error+ F x absolute value of (ln (weighted est. / MRA est.) )


Confidence Model Maintenance

Re-calibrated with every calibration iteration Periodically calibrated to valuer judgement of

estimate output Can be used individually or aggregated for

decisions at varying levels Developed between VOA, John Thompson

(Cole Layer Trumble), Dr Jim Abbott (EDS) Presented at IAAO CAMA/GIS conference

February 2006


Summary of Models (1)

Multiplicative MRA and comparable selection Operational delivery Undertaken locally Specified centrally

Broad Based Model Market Trends Linearisations e.g. Group and Type variables to

allow wider comparability Develop constraints for local models Quality assurance on operational delivery


Summary of Models (2)

Control Model Operates in background of local model Highlights data irregularities

Confidence Model Uses data from all models Calibrated to valuer opinion Provides information for use at all levels


Managing Business Change

Addressing the time lag Review whole data process Use electronic transfer to bring in sales

data Consider how to engage with taxpayer

New skills Control modelling and train professionals Develop support and maintenance staff

Rigorous application of project management methodology


IT Development

VOA has successfully integrated AVM technology with existing IT; this required: Improving data flow and management Enhanced mapping tools “Customising” 3rd party “off-the-shelf” product Introducing workflow to manage delivery

Lead-in time has created tensions IT supplier’s understanding of business and

new technology does not happen overnight!


Conclusions/Lessons Learned (1)

Other data sources create issues: Often aggregated Currency – frequency of update Niche provider with limited depth Definition – what creates the data Cost – Is it worth it to you? Your records may be adequate and better

than elsewhere anyway!



How do you measure data quality Sale data quality

Mostly measured by model Dependent upon proper sales review process

Subject data quality Adoption of external standards not always

possible e.g. IAAO 6 year cycle Data sampling using existing maintenance

activities can support the view



Time lag Understand the delivery requirement Address steps in the data process Work closer with data “partners” Central modelling can identify areas of

concern Proximity to data

You cannot easily replace local knowledge of market and meaning of sales.



Modelling Define “attributes” with knowledge of AVM

techniques Balance modellers desire for more data of

imperfect market and cost to complete/maintain

Stabilise data prior to commencement of modelling

Valuing to “Band” does not loosen the rigour of modelling and data management



Use AVM to direct effort Raw data analysis COD & COV are not the only measures “Frequently used comparables” Thematic mapping

Use formal modelling hierarchy Don’t rush to deliver. Inefficiencies result

Stabilise Model approach to ensure consistency Data stability and consistency



Create clear lines of communication Local Management Local technical/modelling

Training Make timely Consider delivery mechanism (e-learning,

workshop) Project Management Structure A quality model can be let down by a poorly

defined and delivered process!


Some VOA achievements

Digitisation of over 22 Million records Data completeness approaching 100% Over 120 surveyors trained in AVM techniques Market analysis nationwide, including mapping

localities across the whole country Model performance well within recognised

standards e.g. median COD of 353 BAs is 9.97 Proven IT platform for mass appraisal at

national level.


Questions?

Tim Eden BSc MRICS IRRVDeputy Director of Council [email protected]


Achieving World Class

valuation office agency experience of automated valuation modelling (avm) in england tim eden bsc...

Documents