ck_long_tail

Knowledge Knowledge Farming Farming

City Knowledge and the Long Tail City Knowledge and the Long Tail of Small Cities of Small Cities

Fabio CarreraFabio Carrera

UCL-CASA, March 2, 2007UCL-CASA, March 2, 2007

Born in Venice, Italy Born in Venice, Italy BSEE and MSCS @ WPI BSEE and MSCS @ WPI PhD @ MITPhD @ MIT

Urban Information Systems and Urban Information Systems and Planning Planning

Teaching @ WPI and MITTeaching @ WPI and MIT Director of Venice and Boston Project Director of Venice and Boston Project

CentersCenters Founder and Director of City Lab (WPI)Founder and Director of City Lab (WPI) Planning Board in Spencer, MAPlanning Board in Spencer, MA Consultant to municipalitiesConsultant to municipalities

BioBioFabio CarreraFabio Carrera

City KnowledgeCity Knowledge The 6 ToolsThe 6 Tools Birth Certificates Birth Certificates

Data FarmingData Farming The Long Tail of Small Cities The Long Tail of Small Cities

Presentation Presentation OutlineOutline

PromotesPromotes

the transformation of the transformation of

municipalitiesmunicipalities

from from hunter-gatherershunter-gatherers of urban of urban

data data

to to farmersfarmers of municipal information of municipal information

City KnowledgeCity Knowledge

Municipalities are the Municipalities are the locuslocus of change of change Cities = Structures + ActivitiesCities = Structures + Activities Reality = Backlog + Future ChangeReality = Backlog + Future Change Space Is the GlueSpace Is the Glue Middle-out = Top-down + Bottom-upMiddle-out = Top-down + Bottom-up Government only has 6 tools for Government only has 6 tools for

implementation and data collectionimplementation and data collection

The Premises of The Premises of CKCK

Municipalities are the Municipalities are the locuslocus of of changechange

Cities = Structures + ActivitiesCities = Structures + Activities Reality = Backlog + Future ChangeReality = Backlog + Future Change Space Is the GlueSpace Is the Glue Middle-out = Top-down + Bottom-upMiddle-out = Top-down + Bottom-up Government only has 5 (or so) tools for Government only has 5 (or so) tools for

implementationimplementation


Like politics, “all change is local”Like politics, “all change is local” Change is filtered/allowed by municipalitiesChange is filtered/allowed by municipalities with CK:with CK:

City departments implement information strategiesCity departments implement information strategies Urban information is farmed-in at a fine grainUrban information is farmed-in at a fine grain Documentation becomes InformationDocumentation becomes Information Intra- and Inter-departmental sharing is commonplaceIntra- and Inter-departmental sharing is commonplace Regional patterns (SDI) emerge upon municipal Regional patterns (SDI) emerge upon municipal

foundationsfoundations

Municipalities are the Municipalities are the locuslocus of change of change Cities = Structures + ActivitiesCities = Structures + Activities Reality = Backlog + Future ChangeReality = Backlog + Future Change Space Is the GlueSpace Is the Glue Middle-out = Top-down + Bottom-upMiddle-out = Top-down + Bottom-up Government only has 5 (or so) tools for Government only has 5 (or so) tools for



“The Fundamental problem is to decide what the form of a human settlement consists of […]

[…] the chosen ground is the spatiotemporal distribution of human actions and the physical things which are the context of those actions […]”.

Lynch, Good City Form, p. 48

Structures are more stable and permanentStructures are more stable and permanent Structural change can be captured as it occursStructural change can be captured as it occurs

Activities are more dynamic and fickleActivities are more dynamic and fickle Activities can be frozen in time and space (snapshots)Activities can be frozen in time and space (snapshots)

with CK:with CK: Information about structures is routinely updatedInformation about structures is routinely updated Activities are “spatialized”Activities are “spatialized” Activities are periodically frozenActivities are periodically frozen




There is a lot of “reality” already out there…There is a lot of “reality” already out there… But the amount of information is finiteBut the amount of information is finite with CK the backlog can be completely capturedwith CK the backlog can be completely captured

Urban change is rather slow so, with CKUrban change is rather slow so, with CK all Structural change is captured at the sourceall Structural change is captured at the source snapshots of activities are creatively obtained snapshots of activities are creatively obtained

with CK, municipal information is “farmed” with CK, municipal information is “farmed” dailydaily




within CK:within CK: Space plays a key role in municipal information Space plays a key role in municipal information

farmingfarming Addresses are no longer primary spatial identifiersAddresses are no longer primary spatial identifiers GIS means Geographic GIS means Geographic IndexingIndexing Systems Systems Space indexes our datasetsSpace indexes our datasets




Top-down is rigorous and structured…Top-down is rigorous and structured…… but is received as an “imposition” and resisted… but is received as an “imposition” and resisted

Bottom-up is passionate and self-interested…Bottom-up is passionate and self-interested…… but unstructured, unscalable and unsustainable… but unstructured, unscalable and unsustainable

with CK:with CK: Pure top-down and bottom-up approaches disappearPure top-down and bottom-up approaches disappear Middle-out combines the positive traits of bothMiddle-out combines the positive traits of both

Municipalities are the Municipalities are the locuslocus of change of change Cities = Structures + ActivitiesCities = Structures + Activities Reality = Backlog + Future ChangeReality = Backlog + Future Change Space Is the GlueSpace Is the Glue Middle-out = Top-down + Bottom-upMiddle-out = Top-down + Bottom-up Government only has 6 tools for Government only has 6 tools for

implementation (and information implementation (and information gathering)gathering)


1.1. Ownership & OperationOwnership & Operation

2.2. RegulationRegulation

3.3. Incentives/DisincentivesIncentives/Disincentives

4.4. Education & InformationEducation & Information

5.5. RightsRights

6.6. Mitigation & CompensationMitigation & Compensation

with CK:with CK: Municipalities consciously & creatively combine the 6 Municipalities consciously & creatively combine the 6

tools fortools for Information Farming Information Farming Policy/Plan ImplementationPolicy/Plan Implementation

1.1. Ownership and OperationOwnership and Operation



4.4. Education and InformationEducation and Information

5.5. Right swappingRight swapping

6.6. Mitigation and CompensationMitigation and Compensation

6 Tools of Gov.’t6 Tools of Gov.’tapplied to Data Farmingapplied to Data Farming







whereby municipalities…whereby municipalities… Adopt internal mechanisms to farm THEIR OWN dataAdopt internal mechanisms to farm THEIR OWN data Emphasize Information in Standard Operating Emphasize Information in Standard Operating

ProceduresProcedures Extract Informational Returns from all internal Extract Informational Returns from all internal

processesprocesses Change “job descriptions” for personnel to include Change “job descriptions” for personnel to include

informationinformation Catch up with their own “backlog”Catch up with their own “backlog” Intercept all future internal change as it happensIntercept all future internal change as it happens

applied to Data Farmingapplied to Data Farming6 Tools of Gov.’t6 Tools of Gov.’t







applied to Data Farmingapplied to Data Farming

whereby municipalities…whereby municipalities… Make informational Returns part of their regulationsMake informational Returns part of their regulations Force outside entities to provide information (for free)Force outside entities to provide information (for free) Change submission requirements (permits, plans…)Change submission requirements (permits, plans…) Modify maintenance and management contractsModify maintenance and management contracts Institute yearly renewals for data updatesInstitute yearly renewals for data updates Apply regulations to capture backlog as wellApply regulations to capture backlog as well Invent creative ways to acquire datasetsInvent creative ways to acquire datasets Become “validators” instead of “collectors”Become “validators” instead of “collectors”

6 Tools of Gov.’t6 Tools of Gov.’t








whereby municipalities…whereby municipalities… Routinely entice outside entities into providing Routinely entice outside entities into providing

informationinformation Change submission fee structures (permits, plans…)Change submission fee structures (permits, plans…) Make “old ways” costly (disincentives)Make “old ways” costly (disincentives) Make it cheaper to do the right thing (incentives)Make it cheaper to do the right thing (incentives) Provide benefits for data updatesProvide benefits for data updates Invent bonuses for data backlogInvent bonuses for data backlog Reward and enforce collaborationReward and enforce collaboration Validate incoming dataValidate incoming data









whereby municipalities…whereby municipalities… Constantly educate citizens about the use of dataConstantly educate citizens about the use of data Are always transparent about motives for data Are always transparent about motives for data

collectioncollection Explore potential for volunteer citizen inputExplore potential for volunteer citizen input Incite “peer-production”Incite “peer-production” Make educational institutions partners in the processMake educational institutions partners in the process Acknowledge and Reward collaborationAcknowledge and Reward collaboration Include this aspect in ALL their initiativesInclude this aspect in ALL their initiatives









Requires “real” creativity but is very powerfulRequires “real” creativity but is very powerful More useful for implementation, toMore useful for implementation, to

Trade-up “as-of” rights in exchange for desired Trade-up “as-of” rights in exchange for desired outcomesoutcomes

in the end municipalities can…in the end municipalities can… include informational returns any time rights are include informational returns any time rights are

renegotiatedrenegotiated increase “as-of” rights in exchange for dataincrease “as-of” rights in exchange for data









More useful for implementation, to…More useful for implementation, to… Mitigate negative consequences of initiativesMitigate negative consequences of initiatives Remove final obstacles to implementationRemove final obstacles to implementation

with this tool municipalities could…with this tool municipalities could… accumulate complaints and suggestions from affected accumulate complaints and suggestions from affected

partiesparties provide online tools for quantifying and logging provide online tools for quantifying and logging

problemsproblems


City KnowledgeCity Knowledge The 6 ToolsThe 6 Tools Birth CertificatesBirth Certificates



Birth CertificatesBirth Certificates

Municipalities treat their assets as newborn Municipalities treat their assets as newborn babiesbabies

Municipalities identify “parent” dept’sMunicipalities identify “parent” dept’s Dept’s produce a “birth certificate” for each Dept’s produce a “birth certificate” for each

asset asset Parent dept. assigns “name” (and code)Parent dept. assigns “name” (and code) Death and Adoption certificates are treated Death and Adoption certificates are treated

similarlysimilarly

Other dept’s refer to assets by their given Other dept’s refer to assets by their given namename

A municipal spatial data infrastructure A municipal spatial data infrastructure emergesemerges

the conceptthe concept

Data FarmingData FarmingCatch up with Backlog

Intercept Future Change

Birth Certificates SnapshotsSnapshots

New Geometry Birth Record

Current Structures Current Activities

Structures Activities

Current Reality

Spatial Reference Activity Measurement

Additional Information

Municipal Spatial Data Infrastructures Municipal Spatial Data Infrastructures emergeemerge

Towns have “plan-ready” informationTowns have “plan-ready” information Municipalities stop hunting-and-gatheringMunicipalities stop hunting-and-gathering

Information is farmed insteadInformation is farmed instead Change is captured at the source (for Change is captured at the source (for

free)free) Open-source web-GIS dominateOpen-source web-GIS dominate New business models emergeNew business models emerge Profits come from “changers” and “users”Profits come from “changers” and “users” Private sector contributes fine-grained Private sector contributes fine-grained

datadata Web services are the means/currencyWeb services are the means/currency

Data FarmingData FarmingWith the farming of City Knowledge:With the farming of City Knowledge:

Web-servicesWeb-services Open-source web-GIS are the platformOpen-source web-GIS are the platform

Light clients or AJAX apps replace standalone appsLight clients or AJAX apps replace standalone apps Systems are upgraded regularly on serverSystems are upgraded regularly on server

Municipalities get data and applications for Municipalities get data and applications for freefree

Web services are available as toolsWeb services are available as tools Dept’s mash-up web-services to suit needsDept’s mash-up web-services to suit needs

Metadata is reliably availableMetadata is reliably available Web 2.0 techniques are commonplaceWeb 2.0 techniques are commonplace

FolksonomiesFolksonomies Reputation Management, etc.Reputation Management, etc.

Urban Information Systems exploit the Long Urban Information Systems exploit the Long TailTail

and City Knowledgeand City Knowledge

MERTON BUILDINGS INFORMATION SYSTEMS

tan

dal

one

App

licat

ions

DB

DB

DB

Ordnance Survey

DTI

ONS

Onl

ine

Dat

aset

s

Permits

Plans

Notifications

OwnershipUse

Construction

Energy

data

data

Cit

y K

no

wle

dg

e

Mastermap

Energy Use

Demographics

BuildingsCKDB

data

Schoolsdata

Estatesdata

Analysis…

NewEdit

Uses…

buildingsroadsparksschoolschurchesindustries

orthoshydrology

boundaries

Energy… Maintenance…

Leases…Repairs…

Notifications… Contracts…

Permits… Inspections…

public buildingscouncil estatesbusinessesorthophotos

roadsCHP plants

50 m

LOUIS – The CITY LAB “Local On-line Urban Information System”


Data FarmingData Farming The Long Tail of Small CitiesThe Long Tail of Small Cities


The Long TailThe Long Tail

163,547 towns (local gov.t) in the world163,239 < 1 Million pop.159,349 < 100,000 pop.130,206 < 10,000 pop. 64,307 < 1,000 pop.

The Long TailThe Long Tailand City Knowledgeand City Knowledge

Size of CitiesSize of Cities

Large Large CitiesCities

Small Small CitiesCities

The total population that lives in small and medium cities is greater than the population in megacities. Small towns (“tail”) represent a huge market opportunity.

ANY TOWNANY TOWN

The Long TailThe Long TailChange managed by various Change managed by various

DepartmentsDepartments

Planning, Buildings, Planning, Buildings, DPWDPW

Other Other DepartmentsDepartments

The Long Tail is Fractal.

within a Municipalitywithin a Municipality

Target main Target main departmentsdepartmentsANY DEPARTMENTANY DEPARTMENT

The Long Tail is Fractal. Starting with the “head” makes sense, but all departments ought to eventually adopt the CK approach.

Data captured by Municipal Processes (Merton, UK)

0

100,000

200,000

300,000

400,000

500,000

600,000

700,000

Bui

ldin

g N

otic

e F

ull P

lans

RS

L H

ous

ing

Sto

ck

Land

& P

rope

rty

Gaz

zett

e db

GIS

Map

Mas

term

apP

lann

ing

App

licat

ion

Cou

ncil

Tax

Tra

de A

ss. N

otifi

catio

n

Aud

itC

ounc

il S

ocia

l Hou

sing

N

ew A

ddr

ess

App

lica

tion

SA

P s

urve

yD

atab

ase

of B

usin

esse

sO

SR

1 F

orm

Bus

ines

s R

ates

Hou

sing

& P

ollu

tion

Bus

ines

s In

spec

tion

Hou

sing

Sto

ck C

ondi

tion

Ass

essm

ent

Bus

ines

s Li

cens

eB

usin

ess

Dire

cto

ryB

usin

ess

Dire

cto

ryIn

spec

tion

Not

ifica

tion

HE

CA

sur

vey

Fire

Cer

tific

ate

Dut

y of

Car

eIn

spec

tions

Land

Rec

ords

Dat

abas

eF

oods

Reg

istr

atio

nH

ealth

& S

afet

y In

spec

tions

Cou

ncil

Pro

pert

y L

ist

Dire

ctor

y of

Est

ablis

hmen

tsIn

dust

rial M

apM

itcha

m E

nerg

y A

udit

The Long TailThe Long TailLog-log of Data Captured

0

1

2

3

4

5

6

7

0.00

0.60

0.85

1.00

1.11

1.20

1.28

1.34

1.40

1.45

1.49

1.53

Log (rank)

log

(fr

equ

ency

)

Number of Urban Elements (Venice)

0

5,000

10,000

15,000

20,000

25,000

30,000

35,000

40,000

Ad

dres

ses

Bu

ildin

gs

Str

eets

Las

tric

ato

Bo

at P

arki

ng

Pav

imen

tazi

oni

Sto

res

Sew

er h

oles

Pu

blic

Art

Do

cks

Pla

teat

ici

Gre

en S

pace

sB

rid

ges

Ho

tels

Pal

aces

Seg

men

tsN

ode

sF

onda

men

te

Wel

lsC

anal

sC

hur

ches

Isol

eB

ellto

wer

sZ

ones

Mon

umen

tsL

unet

teC

onv

ents

Rii_

tera

Insu

leS

esti

eri

Mun

icip

alita

'


Log-log of Venice Elements

0

1

2

3

4

5

0.00

0.48

0.70

0.85

0.95

1.04

1.11

1.18

1.23

1.28

1.32

1.36

1.40

1.43

1.46

1.49

Log (rank)

Lo

g (

freq

uen

cy)

Number of Urban Elements (Cambridge, MA)

0

2,000

4,000

6,000

8,000

10,000

12,000

14,000

16,000

18,000

20,000

Addresse

s

Buildin

gs

Parce

ls

Trees

Man

holes

Curbs

Street

Lig

hts

Segm

ents

Met

ers

Nodes

Roads

Hydra

nts

Traffi

c Lig

hts

Street

s

Open

Spac

e

Parks

Met

er z

ones


Log-log of Cambridge Elements

0

1

2

3

4

5

0.00

0.30

0.48

0.60

0.70

0.78

0.85

0.90

0.95

1.00

1.04

1.08

1.11

1.15

1.18

1.20

1.23

Log (rank)

Lo

g (

freq

uen

cy)

Number of Urban Elements (Merton, UK)

0

10,000

20,000

30,000

40,000

50,000

60,000

70,000

80,000

90,000

100,000

Address

es

Buildin

gs

Trees

Unlit S

igns

Gully

Bollard

s Unlit

TPO

Illum

inat

ed S

igns

Adopted H

wy

Bollard

s

Beacons

Privat

e st

reet

s

Unadopte

d Hwys

Dogbins

School F

lash

ers

TFL Red

Route

s


Log-log of Merton Elements

0

1

2

3

4

5

6

0.00

0.30

0.48

0.60

0.70

0.78

0.85

0.90

0.95

1.00

1.04

1.08

1.11

1.15

1.18

1.20

Log (rank)

Lo

g (

freq

uen

cy)


From RedfishGroup (redfish.com) – Santa Fe

The Long TailThe Long TailAmount of Change by different Amount of Change by different

“agents”“agents”

specific developers, specific developers, contractors, staffcontractors, staff

Other agentsOther agents

Again, the “head” will yield instant benefits, although the change generated by agents in the tail may be quantitatively just as large. Target all agents eventually

major agentsmajor agents

within a Departmentwithin a Department

ANY AGENTANY AGENT

The Long TailThe Long TailChange produced via various Change produced via various

processesprocesses

subdivision subdivision approvals, approvals,

construction construction permits, contractspermits, contracts

Other ProcessesOther Processes

Processes in the head are major vehicles of change. Minor processes in the tail still add up to major change. Eventually all processes will be addressed by CK.

within an administrative processwithin an administrative process

Low-hanging Low-hanging fruitsfruits

ANY PROCESSANY PROCESS

The Long TailThe Long TailChange produced over time Change produced over time

BACKLOGBACKLOG

Future Future ChangeChange

The backlog may be huge but it is finite and worth catching up with. Focusing on the long tail of future piecemeal change will close the loop forever.

past and futurepast and future

Municipal Spatial Data Infrastructures Municipal Spatial Data Infrastructures flourishflourish

Departments farm their “data plots”Departments farm their “data plots” The 6 tools make data farming The 6 tools make data farming

perpetual/freeperpetual/free Fine-grain is achieved routinelyFine-grain is achieved routinely Backlog is completely captured Backlog is completely captured Change is intercepted as it happensChange is intercepted as it happens Technologies automate/facilitate data Technologies automate/facilitate data

collectioncollection Web-services enable intra-/inter-dept. Web-services enable intra-/inter-dept.

sharingsharing Information is treated like an infrastructureInformation is treated like an infrastructure

In SummaryIn Summary

More about CKMore about CK

[email protected]@wpi.eduhttp://www.wpi.edu/~carrera/Publications/http://www.wpi.edu/~carrera/Publications/

Publications.htmlPublications.htmlhttp://www.wpi.edu/~carrera/MIT/dissertation.htmlhttp://www.wpi.edu/~carrera/MIT/dissertation.html

mailto:[email protected]

ck_long_tail

Technology

future change space

thelocusof change cities

urban change

local change

future internal change

activities reality

information intra

permanent structural