cesar working document 2 erreichbarkeitsatlas experiment 2

CESAR Working Document Series no. 2 Erreichbarkeitsatlas as PSS

CESAR WORKING DOCUMENT SERIES Working document no.2

Erreichbarkeitsatlas as Planning Support System Testing the performance in supporting strategy making

M. te Brömmelstroet 26 January 2011

This working document series is a joint initiative of the University of Amsterdam, Utrecht University, Wageningen University and

Research centre and TNO

The research that is presented in this series is financed by the NWO program on Sustainable Accessibility of the Randstad: http://www.nwo.nl/nwohome.nsf/pages/nwoa_79vlym_eng


TABLE OF CONTENT

1. INTRODUCTION ............................................................................................................... 4

2. ASSESSING THE PERFORMANCE OF A PSS ....................................................................... 5

2.1 Dual goals of PSS .................................................................................................................. 5 2.2 Operationalizing PSS goals ................................................................................................... 5

3. SETUP OF THE EXPERIMENT ............................................................................................ 7

3.1 Intervention: The Erreichbarkeitsatlas ................................................................................. 7 3.2 Strategy making trial with students ...................................................................................... 8 3.3 Testing the performance ...................................................................................................... 8

APPENDIX 1: PROCESS QUESTIONNAIRE .............................................................................. 14

APPENDIX 2: OUTCOME PERFORMANCE QUESTIONNAIRE .................................................. 17

APPENDIX 3: INSTRUMENT USABILITY QUESTIONNAIRE ...................................................... 18


1. INTRODUCTION

There are a large number of computer based systems that aim to support integrated land use and transport planning; more than 100 in the Netherlands alone (Al 2005). These so-‐called Planning Support Systems (PSS) have been developed since the 1970s and are still continuously improved. In a recent survey among land use and transport planners in the Netherlands we found that, just as in almost all planning fields, these instrument fail to support an important phase of planning where land use and transport should come together; the strategic planning phase (Te Brömmelstroet 2010). Figure 1 lists the reasons that were found to block the widespread use of these tools in daily practice of integrated land use and transport strategy making. Figure 1 Bottlenecks for PSS in integrated land use and transport strategy making (% of respondents (124)) that think it is a (highly) problematic.

Figure 1 shows that it is mainly a set of soft characteristics that hamper widespread use of PSS. Transparency, low communication value, user friendliness and interactiveness are seen as (highly) problematic by more than half of the respondents. There is no shortage of ideas to bridge what has been coined the implementation gap. Some of these focus on improving PSS software by adding new functionality: PSS that are more integrated (i.e. What If developed by Klosterman 1999), more interactive (i.e. Urban Strategy developed by TNO 2011) or more user-‐friendly (i.e. UrbanSim developed by Waddell 2002, 2011). Others focus more on the hardware, such as Maptables (see Vonk & Ligtenberg 2009) and other visual gadgets. Then there is the process-‐oriented line that focuses on bridging the human gap between the potential end-‐users and the PSS developers with more participative, iterative PSS development structures (i.e. Lee 1973; 1994; Te Brömmelstroet & Schrijnen 2010; Vonk 2006). One of the instruments that aim to follow the strategy of improving the software is the Accessibility Atlas (Erreichbarkeits Atlas) that has recently been developed by the Technical University of Munich. Here, we report on a randomized clinical trial that was set up in 2011 to test the performance of this innovative instrument in supporting strategy making processes. To do so, I first define how the performance of a PSS can be measured (section 2). Then, I describe the methodological choices and set up of the experiment (section 3). In section 4, the results are presented and this report closes by discussing the findings and reflecting on its meaning.


2. ASSESSING THE PERFORMANCE OF A PSS

2.1 Dual goals of PSS Recently we have proposed an integrated performance framework that allows us to structurally test if PSS, or specific improvement strategies for them, are effective in supporting strategy making processes (Te Brömmelstroet, 2012). This framework is based on the concept of dual goals of PSS. First, many PSS explicitly aim to improve planning processes, e.g. by structuring them better and/or making them more interactive, integrative and participatory. Next to that PSS aim to improve the outcomes of these processes (e.g. strategies, plans and projects), e.g. by providing relevant content (knowledge, information) and facilitating design-‐analyse loops. In strategic planning this link is particularly problematic. Following Couclelis, who addressed land use planning, we can assert that strategic planning

“is a hopelessly complex human endeavour. It involves actions taken by some to affect the use of land controlled by others, following decisions taken by third parties based on values not shared by all concerned, regarding issues no one fully comprehends, in an attempt to guide events and processes that very likely will not unfold in the time, place, and manner anticipated” (Couclelis, 2005; p. 1355).

Following this, it is often said that strategy making problems are ‘wicked’ (Rittel and Webber, 1984; Christensen, 1985); problems on which there is no consensus and for which there are no clear-‐cut answers or solutions. Pelzer (2012) refers to this problem as a double complex one: the subject of planning is complex, as is the process of planning itself. In such situations, a rigid protocol of planning steps is hardly helpful. It is the development of the capacity to deal with these problems, rather than final solutions that is the general aim of strategic planning (Healey, 2006). Planners here (should) aim to become aware and learn to cope with complexity and the “unknown unknowns” (Taleb, 2007) instead of collecting knowledge to reduce it/them.

2.2 Operationalizing PSS goals Both goals of PSS (i.e. improving process and improving outcomes) have been operationalized into a multidimensional framework. Based on academic literature on ideational output (e.g. Dean et al. 2006) the quality of a planning outcome can be rated on four dimensions; • novelty (originality, paradigm relatedness), • workability (implementability, acceptability), • relevance (applicability effectiveness), and • specificity (completeness, Implicational explicitness, clarity). For a planning outcome to be of high quality, it has to score on all these dimensions. Based on academic work on Group Model Building (Rouwette et al. 2002) we operationalized the quality of the process in nine dimensions; • reactions (enthusiasm, satisfaction, credibility), • insight (in problems, in others’ assumptions), • commitment, • behavioural change, • communication, • shared language, • consensus (on problem, goal, strategies), • cohesion and • efficiency.


Again, this means that for a planning process to be of high quality it needs to score high on all these dimensions. Table 1 Dimensions of outcome Table 2 Dimensions of process Dimension Operationalization

1. Novelty 1a. Originality

1b. Paradigm relatedness

2. Workability 2a. Implementability

2b. Acceptability

3. Relevance 3a. Applicability

3b. Effectiveness

4. Specificity 4a. Completeness

4b. Implicational explicitness

4c. Clarity

Dimension Operationalization

5. Reaction 5a. Enthusiasm

5b. Satisfaction

5c. Credibility

6. Insight 6a. Insight in problem

6b. Insight in assumptions

7. Commitment

8. Behaviour

9. Communication

10. Shared language

11. Consensus 11a. On problem

11b. On goals

11c. On strategies

12. Cohesion

13. Efficiency gains


3. SETUP OF THE EXPERIMENT

3.1 Intervention: The Erreichbarkeitsatlas In 2011, the Faculty for Urban form and mobility of the Technical University of Munich developed an accessibility atlas for the Munich Metropolitan region. This instrument allows users to explore the opportunities and treats for this growing region in terms of potential activities to be reached within different travel times by different modes of transport. Examples of maps that can be explored, and that can be interactively developed by the user via an online platform, are the number of people that can reach intercity train stations within 30 minutes or mapping the commuting flows compared to the accessibility quality of these connections. The Erreichbarkeitsatlas is an attempt to transferuse the potential of accessibility as a professional language for integrated land use and mobility planning issues to the realm of urban and regional planning. Accessibility concepts are increasingly acknowledged as fundamental to understand the functioning of cities and urban regions. In particular, accessibility instruments are able to provide a framework for understanding the reciprocal relationships between land use and mobility.

Figure 2 A map from the Erreichbarkeitsatlas: commuting flows vs. accessibility quality

A collection of these maps is used in the experiment in printed form. In the next section, I will describe the experiment in more detail.


3.2 Strategy making trial with students To test the performance of this instrument in supporting strategy making a randomized controlled trial with students was set up. A total of 34 voluntary students took part in the trial, from the Master Environmental Engineering and from the Master Transport Engineering. These students were randomly divided in a control group (that received only a set of empty maps of the region) and a treatment group (that received the maps of the Erreichbarkeitsatlas, introduction to these maps and support by one of the developers). These groups were then again divided in random groups of three students that then engaged in a strategy making process (of 60 minutes) for the Munich Metropolitan region. The planning problem that all groups received on paper is presented in the box below. In the near future, the Europäische Metropolregion München is expecting a strong growth in both economic and demographic terms. In the coming 20 years, the region faces considerable spatial and mobility challenges: within the region, a total of 60.000 new working places (offices and industry), 60.000 new houses and supporting leisure and shopping areas have to be allocated. The region aspires that these new developments are primarily located in places with high public transport accessibility. There, sustainable mobility with limited negative impacts can be guaranteed. You will form a strategic planning team together with two colleagues. This team is asked to develop a rough spatial strategy for this allocation challenge. You will have a map of the region and an empty sheet to your disposal on which you can formulate a strategy in text and in geographical drawings (please remember to include a legend). In total you will have a maximum of 60 minutes to perform this. In this competition you are asked to find good locations (on the level of municipalities) and use the complete region as an “empty sheet” where all locations are potentially possible. Don’t worry about the size, form of these locations. Your strategy will be judged by external experts on the quality of the final result. This quality will be judged in terms of: • Novelty; • workability; • relevance and; • specificity.

3.3 Testing the performance After 60 minutes all groups had to hand in their strategy. For this, they could use a map with an attached empty sheet to write down their ideas and argumentation. These strategies were then our input for the performance test. This test was done in three steps. First, all participants were asked to fill in a questionnaire related to the quality of the planning process. This questionnaire consisted of 32 statements that each related to the dimensions of table 1. For each of these statements, the participant was asked to rate his/her agreement on a 7-‐point Likert scale. This questionnaire is attached in appendix 1. The outcomes of these ratings were grouped and this allowed us to compared the process dimensions of the control group with these of the treatment group. The resulting strategies were rated on their outcome dimensions as listed in table 2. This was extended with one statement on the amount of accessibility logic followed. Again, the same method of (28) 7-‐point Likert scale statements was used. The questionnaire is listed in appendix 2.


Each strategy was rated by two external raters. For this, two PhD students that are specialized in integrated land use and transport from the University of Amsterdam were asked. The third and last measurement questionnaire was focused on the perceived quality of the Erreichbarkeitsatlas. For that, again statements were offered, but now only to the participants of the treatment group. They were asked to respond to several dimensions of the instrument that are generally related to the usability of such instruments (see e.g. figure 1). This questionnaire is attached in appendix 3.


4. RESULTS OF THE EXPERIMENT

4.1 Effects on the planning process

The average scores on the planning process dimensions are listed in table 3. The table distinguishes the groups that did not receive planning support from the Erreichbarkeitsatlas from the groups that did. On the bottom, the differences between the two are presented with the level of significance (deviation from the zero hypothesis that there is no effect). Although both groups score above average (4) on all dimensions, all dimensions but cohesion (no effect) show a negative effect of the support of the Erreichbarkeitsatlas. These negative effects are significant on a 0.05 level for the dimensions (bold for grouped dimensions) satisfaction, reaction, insight in the problem, insight in assumptions, insight, communication, shared language, consensus on problem, consensus on goals, consensus and efficiency.

4.2 Effects on the planning outcome

The average scores on the planning outcome dimensions are listed in table 4. The table is set up similar to table 3. Here, most dimensions score below or just above average (4). Statements on two of them (paradigm relatedness and Implementability) were not filled in by one of the external raters due to a lack of insight in them. These should therefore not be considered as valuable scores. The effects of the Erreichbarkeitsatlas are more mixed here. Small positive scores are found for bold for grouped dimensions) accessibility logic (0.39), applicability (0.19), completeness and specificity (both 0.13). None of these effects are significant. The other dimensions all score negatively, but also not significantly so.

4.3 Perceived quality of the instrument

The usability characteristics of the Erreichbarkeitsatlas are very positively rated by the participants that received support from it, see table 5. Especially, the participants found the instrument easy to understand (5.32). They rated the ability of the instrument to support creating (5.22) and evaluating (5.17) strategic ideas as very positive as well. Important factors as transparency and user-‐friendliness also received a score over 5 (on a scale from 1to 7). The lowest score is for the understandability of the indicators used (still 4.58).

cesar working document 2 erreichbarkeitsatlas experiment 2

Technology