International Centre for Integrated Assessment and Sustainable Development, Maastricht University, PO Box 616, Maastricht 6200 MD Netherlands (email: firstname.lastname@example.org)
Abstract: A comprehensive systems approach is a prerequisite for the development of sustainability policy strategies. Quantitative models are frequently proposed as useful building blocks in this regard. We examine the value of such models in a case study on regional sustainability carried out in cooperation with the administration of the Dutch province of Limburg. With the participation of an interdisciplinary group of civil servants, we developed an influence diagram representing the region and compared this to the coverage of quantitative models used in provincial strategic planning. The significant discrepancies between the two types of system representation lead to a more diligent interpretation of results, help to improve the models, and set challenges for future model use in regional sustainability planning. This article provides policy makers with practical advice on strategic planning and encourages scientists to improve the models by developing new techniques for the integration of quantitative and qualitative analyses.
Keyword: regional planning, sustainable development, prediction models, systems analysis
Citation: Grosskurth J. 2007. Ambition and reality in modeling: a case study on public planning for regional sustainability. Sustainability: Science, Practice, & Policy 3(1):3-11. Published online Mar 20, 2007. http:///archives/vol3iss1/0610-036.grosskurth.html
Regional administrations aiming for sustainability face complex issues resulting from interactions, feedbacks, and trade-offs among various aspects of sustainability, as well as from dynamic structural change. The normative character of sustainability further enhances this systemic complexity—individual and social preferences differ and evolve over time. The difficulties of trying to manage, control, or manipulate such complex systems, even with the best of intentions, have been well documented (see, e.g., Forrester, 1971a; Sterman, 1994; Meadows, 1999; Sterman & Booth Sweeney, 2002; Dörner, 2003). It is widely agreed that addressing these characteristics requires a comprehensive systems approach (see, e.g., Schellnhuber & Wenzel, 1998; Ehrenfeld, 2005).
The exact definition of a systems approach is as elusive as that of sustainability, but researchers propose quantitative models as at least one building block of such an approach (Rotmans, 1998; Robinson, 2004). However, there are significant discrepancies between the ambitions and the practical reality of such models. According to van der Sluijs (1997), the ambition is to
model the complete so-called causal chain, including all the feedbacks within this chain. The causal chain starts with socio-economic drivers leading to economic activity and other practices, leading to emissions and other pressure on the environment[,] leading to environmental changes, leading to physical impacts on societies and ecosystems, leading to socio-economic impacts, eventually returning to cause changes in the socio-economic drivers.
Other ambitions include the transparency of models; their capability to explore uncertainties, trend breaks, and discontinuities; their potential to foster deliberation; and their relevance to decision makers. We are currently far from meeting these objectives. In an overview of challenges for Integrated Environmental Assessment, Toth (2003) reflects on models referring to sustainability. He concludes that, despite significant progress in recent years, the available models are not suited for addressing the complex problems on current policy agendas. Progress on this front will require innovative integration techniques.
We come to a similar conclusion after assessing 13 European and global integrated scenario models, including a set of integrated models regarding sustainability (Greeuw et al. 2000). In particular, the evaluated models do not fulfill the objectives of horizontal and vertical integration and policy relevance. Horizontal integration refers to the amalgamation of different aspects of a model’s domain. Issues of assimilation between the environmental, economic, and sociocultural domains, as well as integration within each of these domains, are important in this respect. Vertical integration refers to the incorporation of different stages of cause-effect chains in the sense of the pressure-state-response cycle (OECD, 1993). Greeuw et al. (2000) also identify a significant neglect of sustainability’s sociocultural domain. The indicators chosen to represent this domain are largely demographic or economic, and only marginally correlated to the underlying system. The assessment further reveals a trade-off between horizontal and vertical integration. While the sociocultural domain is not explored in satisfying depth, the economic domain seems to privilege depth over breadth.
A case study commissioned by the administration of the Dutch province of Limburg allowed us to study in detail the discrepancies between ambition and reality in models for public strategic planning and regional sustainability. The next section introduces the project. We subsequently describe the methods and result of drafting a conceptual integrated system representation of the province in a participatory process. This qualitative approach is followed by an inventory of quantitative forecasting models used by the provincial administration to support its strategic planning. The differences between the relationships covered in the conceptual representation and in the structure of the forecasting models are analyzed. In the concluding section, we offer some practical advice on how to better use existing models and on how to bridge their discrepancies.
In regional strategic planning, forecasting and exploratory studies are crucial. Integrated and forward-looking research supports the planning process by structuring relevant discussions and by facilitating the development of robust strategies. In the Netherlands, legal mandates require that these studies be conducted at regular intervals for some sectors. The Dutch province of Limburg was the first of the country’s twelve provinces to combine its sector plans into a single, integrated strategic plan: the Provinciaal Omgevingsplan Limburg, or POL (Provincie Limburg, 2001). To better harmonize the underlying forecasting models, the provincial government commissioned a year-long participatory project.
For purposes of this article, two of the four project phases are of particular interest. One phase was devoted to the participatory drafting of an influence diagram representing the dynamics of regional development in the province. The other phase was predicated upon an inventory and evaluation of the quantitative indicators and forecasting models used by the provincial administration for strategic planning. Six senior civil servants with different backgrounds formed the core group for the participatory process. Especially early in this process, additional members joined this group to provide specific practical expertise, to increase internal transparency, and to satisfy general interest. During the year, the group worked with our team during 17 workshop sessions of approximately three hours each.1
To facilitate horizontal integration in the administration and among the different forecasting models, we dedicated the first phase of the project to the participatory development of a conceptual integrated system representation of the province. The purpose of this process was to strengthen mutual understanding, to sharpen awareness for the interdependencies between different characteristics and regional developments, and to structure the available information and knowledge in an integrated manner.
We drafted the conceptual integrated system using the SCENE approach (Grosskurth & Rotmans, 2005). The acronym SCENE stands for the three domains of sustainability: SoCial, ENvironmental, and Economic. This approach was developed to foster a better understanding of sustainability’s underlying dynamics and of related issues. SCENE is based on the participative and qualitative representation of stocks and flows in the format of an influence diagram. The three domains of sustainability provide the fundamental structure for SCENE. Stocks describe core elements of a system that change relatively slowly. In contrast to the system-dynamic notion given to the terms “stock” and “flow,” SCENE stocks can be generic titles such as “lifestyle” or “economic vitality.” Moreover, these titles can be interpreted multidimensionally. In the SCENE approach, we generally take four dimensions of a stock into account: quantity, quality, function, and spatial dimension. This methodology breaks with the legacy of system-dynamic modeling where only one dimension of a stock—mostly quantity—is generally taken into account. Flows are relationships between stocks. Flows can represent material flows, information flows, or other relations that follow a cause-effect line. The resultant description of the system is a conceptual model of the real world. During the past five years, we have drafted such models by applying the SCENE approach at national, provincial, and urban levels. Figure 1 is a schematic representation of the “naked” SCENE model.
Figure 1 The conceptual structure of the SCENE model.
With SCENE, the “sustainability triangle” is transformed from a concept for the structuring of sets of sustainability indicators to a framework that allows for the representation of underlying dynamics. For an integrated description, it is crucial that all three capital domains are conceived at the same level of conceptual detail.
Drafting the influence diagram for the POL project was a two step process: 1) selecting and describing the stocks that comprehensively represent the province of Limburg and 2) mapping the influences between these stocks. The participants were advised to develop the diagram of the province in the present and to try not to anticipate possible future changes. The concept of sustainability was implicit in the process as the balanced development of the economic, sociocultural, and environmental domains.
During the first several sessions, the group proposed a set of issues and topics. In an interactive and iterative process of screening, revising, reformulating, and restructuring, this initial list was brought together into a set of 18 stocks that implicitly comprised all indicators regularly monitored by the administration. The stocks were compatible with other documents published by the Province, but covered a wider range. The structure and wording of the stocks corresponded to the administration’s internal organization and communication habits.
The 18 stocks (and their labels) are: population (Popu), consumption (Cons), social structure (Soci), public amenities (Amnt), housing (Hous), security (Secu), space (Spac), air (Air), groundwater (G-wat), surface water (S-wat), soil (Soil), nature (Natu), entrepreneurship (Entr), production (Prod), knowledge and innovation (Know), work (Work), mobility (Mobi), and infrastructure (Infr). For each stock, the group formulated an abstract with a description of the stock, its characteristics, and its scope. For example, the characteristics of the stock “knowledge and innovation” include knowledge institutions, public and private research and development, applications of knowledge and innovation, as well as the spatial distribution of these factors.
The set was also screened for comprehensiveness based on Vester’s (2002) system criteria. Vester is author of the Sensitivity Model Prof. Vester®, a computerized planning and mediation tool for complex systems. The software has frequently been applied in urban and regional planning where stakeholders describe their region as a system. These system representations form the basis for subsequent simulation and intervention steps. To check their comprehensiveness, Vester derived a set of criteria based on his experience and system understanding and meant to guarantee the balanced representation of all relevant aspects of the system. These system criteria require at least the implicit inclusion of seven specific areas of life (e.g., economy, population, and infrastructure), three physical categories (material, energy, and information), four dynamic categories (e.g., flow or time dynamic), and four system-relational categories (that open the system to outside influence or can be influenced from within). The 18 stocks selected satisfied the areas of life and the physical categories of Vester’s criteria. The dynamic and system-relational categories were not satisfied at this stage as they can only be met after the relationships among the stocks have been represented.
To draft these relationships, the participants individually filled in an empty matrix with the 18 stocks as row and column titles. Each cell represented a potential unidirectional flow from one stock to another (though not to itself), resulting in 306 possible flows. Working independently, the participants marked the cells where they perceived a direct influence. They then added a strength estimate (weak, medium, or strong) and a short description of the influence they had in mind. We then combined the individual matrices into a single document, checked the results with experts and, in instances of doubt or dissent, presented it to the group. Among themselves, the participants noted their agreements, discussed and clarified their disagreements, and identified uncertainties. We added comments from the expert scan. These remarks were mostly concerned with additional information about previously identified relationships, thus enriching the group’s discussions. In total, 95 flows were selected and documented—15 strong, 46 medium, and 34 weak. For example, both participants and experts agreed that the stocks “population,” “public amenities,” and “entrepreneurship” all influence the stock “knowledge and innovation.” This latter stock, in turn, influences “entrepreneurship,” “population” (both are bidirectional relationships), and “production.” Figure 2 shows the complete influence diagram (without the flow strength). The stocks are coded by color and shape to make clear the distinction between stocks in the various domains: sociocultural (red circle), environmental (green box), and economic (blue diamond).
Figure 2 The conceptual SCENE model for the Province of Limburg.
To compare the completed SCENE model with the coverage of the available quantitative models, we conducted an extensive inventory and represented the underlying structures in the same format as in the previous section. In total, we identified 13 quantitative models that the administration had previously used for strategic planning:
Overall, four models had a demographic focus, five had a socio-economic orientation, two pertained to the labor market, and two concentrated on land use. Some of these tools were simple shift-share models that translated national extrapolations down to the regional level, while others were sophisticated stand-alone constructions based on statistical regression, general equilibrium, or GIS approaches. Approximately half were developed specifically for the region and the other half are nationally run models that are disaggregated and optimized over all Dutch provinces. These two model types often cover similar topics.
Due to divergent assumptions, regional and national models are often at odds in their projections. The differences are a consequence of statistical estimation biases to which outliers in panel-data sets are subject. For example, labor is highly mobile between most Dutch provinces when one location has vacancies in employment and another has unemployment. However, this mobility does not extend to the peripheral provinces, including Limburg. As Limburg historically had many vacancies unfilled, the national forecasts assumed high immigration into the region and thus projected a higher population than region-specific models that neglected this relationship. In the past, the lower population forecasts have turned out to be the better estimates for Limburg.
As many of the evaluated models are either run at the national level or shared among the provinces, it is safe to assume that the set is representative for other provinces in the Netherlands. Members of the Limburg administration, who frequently took part in interprovincial meetings on monitoring and strategic planning, supported this view. At smaller-scale levels, the coverage of the models seems to get even thinner; at higher-scale levels, the coverage is slightly improved as more general and global models become relevant.
We translated each model into an influence diagram using the stocks described in the previous section. To accomplish this, we screened the available primary and secondary model documentation and conducted extensive interviews with the model developers and/or owners where documentation was insufficient. We accepted a flow as being covered by a model when its framework incorporated any aspect of the flow (either exogenously or endogenously). For example, the flow from “knowledge and innovation” to “entrepreneurship” is represented in the relevant models as an exogenous constant by which productivity increases each year. This facet of the model would obviously be insufficient to explore the effects of changes in the education system and it certainly does not answer potential questions of public officials about increasing entrepreneurship through stimulating knowledge and innovation. More generally, the description provided in the conceptual influence diagram is much richer than in the models themselves.
The illustration below combines the separate influence diagrams for each model into a single overview.
Figure 3 Flows covered by quantitative forecasting models.
Quantitative models used for strategic planning cover 30 of the 95 flows in the conceptual influence diagram. It is important that these modeled flows are not necessarily comprehensively addressed and the different models are not at all compatible in their underlying assumptions or the methods used to make the ultimate calculations. The models are certainly not calibrated for use within a single system of models. This fragmentation of coverage is lost in the illustration above.
The stocks “social structure” and “groundwater” are not covered at all. A third of the remaining stocks are only connected to the system via a single remaining flow. Some flows are covered by several models, most of which concern population and labor-market dynamics. We did not detect a single flow covered in any of the models that group participants had not previously identified. Well-covered stocks include the extent and condition of “housing,” the effects (but not the causes) of changes in the “population,” the economic effects of “entrepreneurship,” changes in land use (“space”), and the situation regarding “mobility.” Other economic—and most social and environmental—stocks are not covered at all or, at best, their treatment is highly superficial. A more favorable picture emerges when the weights of the flows are taken into account. There is coverage of 12 (out of 15) strong flows, 16 (out of 46) medium flows, and only two (out of 34) weak flows.
It is immediately obvious that the loss of information moving from qualitative to quantitative analysis is dramatic. Figure 4 shows the 65 flows that are not covered in the models, but that were identified during the project’s first phase. It shows the discrepancy between the ambition of van der Sluijs’s ideal model (1997) and the reality of models that typically inform public strategic planning.
Figure 4 Flows not covered by quantitative forecasting models.
The discrepancies below the surface of Figure 4 are likely even more significant. While the models covered well the dynamics between the stocks labeled “population,” “housing,” “entrepreneurship,” and “mobility,” the majority of other flows were reduced to exogenous parameters. This observation implies that the dynamics of the system rely fully either on a statistical analysis of the past or on a rough estimate of parameter value. In many cases, the exact parameter values were not documented. Inaccessible parameters mean that flows are putatively covered, but not documented in an accessible way. Because of this lack of transparency, even simple scenario studies cannot be executed in an informed manner. The richness of the qualitative influence diagram is almost completely lost.
Not surprisingly, there is a clear relationship between the ease with which a stock can be quantified and its inclusion in the models. We were able to deduce this straightforward relationship from the inventory of provincial indicators. On the one hand, “security” and “social structure” are lost from the inventory and from the models. Their core characteristics are “soft” and notoriously difficult to quantify. “Population,” on the other hand, is a stock with very high data availability and, as such, is prevalent in the models.
We also witnessed a clear relationship between the coverage of different stocks and the modeling methods used to forecast them. More dynamic modeling techniques, such as general equilibrium of system dynamics, allow for scenario-type, longer-term explorations and often provide insights into counterintuitive developments (Forrester, 1971b). Integrated models of causal chains, as described above, require such dynamic techniques (Gilbert & Troitzsch, 1999). Parameter-based shift-share modules largely ignore these connections, give little insight into possible surprises, and only very rarely produce counterintuitive insights.
In our case, “population,” “housing,” “work,” “entrepreneurship,” and, to some extent, “mobility” were modeled dynamically and included a large number of feedbacks and interactions. General equilibrium modeling is the method of choice for the more sophisticated of these models. “Consumption” and “production,” two stocks related to lifestyles and type of economic activity, are mostly covered as parameters in shift-share modules (e.g., translating economic activity into environmental impacts by assuming emissions per unit of production based on statistical analysis). All other stocks are covered in the form of calculated results of these simple shift-share procedures. This pattern is illustrated in Figure 5. A better integration of models would require coverage of a larger number of aspects using dynamic modeling techniques.
Figure 5 Stocks and modeling methods
One might respond by noting that the modelers apply Occam’s razor quite skillfully. This principle requires that any explanation of a phenomenon should rely on as few assumptions as possible. The modelers minimize the number of assumed relationships by “shaving off” those with little statistical significance and thus reduce the system’s complexity without significantly changing the dynamic properties. For the short run that might be true, but over the longer term, discontinuities in trends and other surprises are likely to result from causes located virtually anywhere in the system. Strategic planning must not, by definition, exclude these causes. Qualitative aspects of the stocks are not considered and influences currently deemed weak are omitted, severely limiting the models’ adequacy for exploring the consequences of novel events and developments. Occam’s razor is not compatible with rich, exploratory, forward-looking studies.
However, the fact that flows with high strength estimates are generally incorporated into the models indicates that the modelers are judicious in their application of Occam’s razor. For pragmatic business-as-usual scenarios they do appear to set the right priorities. Just as the participants tended to give a higher strength estimate to flows subject to little uncertainty, volatility, and discussion, modelers focus on easy-to-measure and relatively stable flows. Yet, in E. F. Schumacher‘s classic phrasing (1977), “one restriction entails another. We attain objectivity, but we fail to attain knowledge of the object as a whole.”
Practical Advice on Better Use of the Models
Based on our analysis of the conceptual SCENE model and the corresponding model inventory, we formulated three recommendations for the provincial administration on how to better use its available models and how to efficiently improve the overall set.
First, a better understanding of the structure of the models is imperative for better use of their results. Our primary advice was to require disclosure of the parameter values whenever a model was commissioned. This small amount of extra insight would enable public officials to place a particular model in its proper context and to improve interpretation of its output in terms of validity and robustness. Also, simple scenario exercises that introduced different change rates—or even discontinuities—into the development of the parameters over time would create opportunities for transparency. Different futures can thus be explored in a rule-of-thumb manner. The results could also illustrate narrative scenarios of various futures.
Second, extension of the individual models would be an inefficient way to improve the integrated dynamics of the whole set. Harmonization of the underlying assumptions (and thus the possibility to vary them with full intention) would be a much more important step.
Finally, a fully functional integrated model with a sufficient level of detail is indeed possible. A model of this kind would, however, require considerable resources to implement while its comparability with the model runs of other provinces would be lost.
Bridging the Discrepancies
The first step in bridging the discrepancies between qualitative and quantitative analyses of complex systems must be a better understanding of the potential and the limits of both approaches.
Quantitative models have often been critically appraised. Godet (1983) criticized dependency on inaccurate data, coupled with unstable models and the explanation of the future in terms of the past, as an inherent property of statistical models. In response, he called for a global, qualitative approach.
The ensuing two decades have seen no fundamental progress in this respect. “Mathematical modelling is only possible if one is willing to except important parts of the problem and to limit an originally comprehensive understanding of the problem to the computable part of the problem” (Weimer-Jehle, 2006). Also, uncertainties are notoriously difficult to represent in quantitative studies that suggest precision and objectivity (Jaeger et al. 1998; 2001). Modeling results have little value once deprived of their underlying assumptions and context.
However, we found that the insights gained by provincial administration officials during the project helped them to make more appropriate use of the model results. An understanding of which factors were accounted for in the different studies, as well as an appreciation of the implications of different methods, helped the participants to use model results as arguments rather than indisputable facts. This experience confirms observations in similar projects. In reflecting on the ULYSSES project on urban lifestyles and sustainability, De Marchi et al. (1998) describe indications of mutual learning where the participatory processes included a somewhat odd “Mr. Computer.” The perception of the computer as a group member indicates the experience of mutual learning among human participants, the computer (model) and the modelers. Siebenhüner & Barth (2005) evaluate three major integrated assessment projects that embedded models in their participatory approaches: the ULYSSES project described above, the COOL project (focused on long-term climate options), and the VISIONS project (focused on developing integrated visions for a sustainable Europe). Even though these processes were mainly concerned with learning from models rather than about models, the authors indicate that some models “served as a trigger for debates about the implied uncertainties.”
A qualitative influence matrix also has limited value as a stand-alone product. Experiments on learning about dynamic systems have shown quite impressively that people have great difficulties conceptually learning about dynamic behavior, even under the most favorable circumstances (Sterman, 1994; Booth Sweeney & Sterman, 2001; Sterman & Booth Sweeney, 2002). Thus, the analytical power of quantitative models is currently the only technique available to structurally explore complex long-term dynamics (though not all modeling techniques are suited for this).
However, even without an exact understanding of the dynamics, mapping the province in the form of an influence diagram facilitated horizontal integration in organizational decision making. Civil servants from different provincial departments are now more likely to contact colleagues from other administrative units when their decisions might generate ripple effects and their mutual comprehension has improved. Vester (2002) provides anecdotal evidence of similar effects for a series of case studies: a working group on systems thinking set up in Bad Aibling (which continued to be active after his intervention), an altered structure for a regional planning group in Frankfurt, more integrated project perceptions at NERIS (Netzwerk in der Risiko-Sensitivitätsanalyse—Networkin Risk and Sensitivity Analysis). Unfortunately, empirical studies on the effects of participation, the drafting of system diagrams, and the implications of learning about factual information—including model structures—are few and far between and quality criteria for participatory processes are disputed and generally hard to test (van de Kerkhof, 2004).
The direct path towards more comprehensive coverage in the sense of the cause-effect cycle described above would be to extend existing models to capture a larger number of dynamics. Such an effort would shift some stocks in Figure 4 from the right side to the left side, from impact and parameter status toward becoming integrated elements of the dynamic model. However, in many cases this would be mere window dressing as these models have been developed with a more focused target in mind. Toth (2003) warns modelers that they “should not extend their tools far beyond their original objectives because this might jeopardize the internal consistency and integrity of the [model].” Every flow added to a model exponentially multiplies the number of indirect influences and feedback cycles. This increase in complexity limits the dynamics that can be modeled without losing an overall understanding of the system’s behavior.
Many model extensions would also require more data. The most far-reaching project in this respect is probably the Sustainability Geoscope that attempts to be an observation instrument for the anthropocene, the era in which natural and human dimensions of the earth system have become inseparable (see http://www.sustainability-geoscope.net). But it remains to be seen whether these additional data are actually useful for modeling purposes when there are breaks in the trends and the meaning of indicators changes. Bossel (1999) illustrates this effectively when he observes:
As systems change and develop in a changing environment, individual indicators may lose their relevance and may have to be replaced by others that are more relevant under current conditions. Where once coal consumption per capita may have been a useful indicator of living standard, the number of computer chips in use per person may be a better indicator at another time.
Time-series analysis obviously becomes problematic under such circumstances.
The most ambitious option is to develop better ways to integrate qualitative and quantitative analyses. Such methods would combine some of the rigor of quantitative analysis with some of the richness of qualitative analysis. Numerous scholars have been calling for years for the development of such tools and some notable progress to that end is now being made (Rotmans, 1998; Kates et al. 2001; Toth, 2003; Robinson, 2004). A good example is the Georgia Basin project described in Tansey et al. (2002). The research team applied user-friendly model-based tools in participatory processes, triggering the participants to test the consistency of their narratives of a desirable future, to explore their choices and the consequences that followed, and to experience the resultant trade-offs. In addition to combining quantitative and qualitative exploratory methods, the exercise incorporated a strong element of learning.
We have observed a significant loss of information, combined with a gain in analytical possibilities, when moving from qualitative analysis to quantitative modeling. We have presented some indications that this result is transferable to other regions and other scale levels. Whether the loss of information outweighs the gain in analytical possibilities must be determined for each case individually. We have the choice between knowing a little of the whole or a whole lot of a little. Combining both approaches will remain a challenge for the future. Experiments to resolve these problems are being conducted. Models are increasingly open to “soft” factors, while qualitative scientists are strengthening the analytical aspect of their methods. It remains to be seen how fast the field will progress from clumsy first steps to a brisk walk. To advance along this path, current modeling tools appear unsuitable. A change is needed in the research paradigms of both modelers and qualitative scientists. Neither methodological approach can be expected to meet the other party on its own terms. Policymakers are caught between the two as numbers provide them with authority and suggest objectivity while narratives and the simplification of complex issues down to strong statements are the daily bread of politics.
This project was financed by the provincial government of Limburg and owes everything to the commitment, enthusiasm, and constructive criticism of the process participants. The project team included the author, Nicole Rijkens, Martin van de Lindt, and Jan Rotmans. This initiative (and the resultant publication) has benefited significantly from the comments of my colleagues at ICIS, especially Pim Martens. Two anonymous reviewers and the editor of this journal have been very helpful and diligent. The system illustrations have been drawn with the software package Pajek (Batagelj & Mrvar, 2004).
1 Full documentation of the project, the influence diagram, a detailed explanation of each stock and flow, as well as the full inventory of models, is contained in the project documentation and available from the author (in Dutch).
4cast. 2006. Modeltoepassing. http://www.4-cast.nl/defaultphp?id=1028. October 30, 2006.
ABFresearch. 2006a. Primos Prognose. http://www.abfresearch.nl/default.asp?p=33. October 30, 2006.
ABFresearch. 2006b. Ruimtelijk Demografisch Prognosemodel. http://www.abfresearch.nl/default.asp?r=64. October 30, 2006.
ABFresearch. 2006c. IPB Model 2002. http://www.abfresearch.nl/default.asp?r=58. October 30, 2006.
Batagelj, V. & Mrvar, A. (2006). Pajek-Program for Large Network Analysis. http://vlado.fmf.uni-lj.si/pub/networks/pajek/. June 2004.
Buck Consultants International (BCI). 1998. Marktanalyse Bedrijventerreinen Zuid-Limburg. Nijmegen: Buck Consultants International.
Booth Sweeney, L. & Sterman, J. 2001. Bathtub dynamics: initial results of a systems thinking inventory. System Dynamics Review 16(4):249–286.
Bossel, H. 1999. Indicators for Sustainable Development: Theory, Method, Applications. Winnipeg: International Institute for Sustainable Development.
Centraal Planbureau (CPB). 1996. Omgevingsscenario's Lange termijn verkenning 1995-2020. Working Paper 89. The Hague: CPB.
De Marchi, B., Funtowicz, S., Gough, C., Guimarães Pereira, Ã., & Rota, E. 1998. The ULYSSES Voyage: The ULYSSES Project at the JRC. Report No. EUR 17760EN. Ispra: Joint Research Centre, European Commission.
Dörner, D. 2003. Die Logik des Misslingens: Strategisches Denken in Komplexen Situationen. Hamburg: Rowohlt.
Ehrenfeld, J. 2005. The roots of sustainability. Sloan Management Review 46(2):23–25.
Engelen, G., White, R., & de Nijs, T. 2003. Environment explorer: a spatial support system for the integrated assessment of socio-economic and environmental policies in the Netherlands. Integrated Assessment 4(2):97–105.
ETIL. 2000. Harmonisatie Prognoses POL, Onderdeel: Beoordeling Gebruikte Prognoses en Voorstellen Nieuwe Prognoses. Maastricht: ETIL.
Forrester, J. 1971a. World Dynamics. Cambridge, MA: Wright-Allen Press.
Forrester, J. 1971b. Counterintuitive behavior of social systems. Theory and Decision 2(2):109–140.
Gilbert, N. & Troitzsch, K. 1999. Simulation for the Social Scientist. Buckingham: Open University Press.
Godet, M. 1983. Reducing the blunders in forecasting. Futures 15(3):181–192.
Greeuw, S., van Asselt, M., Grosskurth, J., Storms, C., Rijkens-Klomp, N., Rothman, D., & Rotmans, J. 2000. Cloudy Crystal Balls: An Assessment of Recent European and Global Scenario Studies and Models. Copenhagen: European Environment Agency.
Grosskurth, J. & Rotmans, J. 2005. The SCENE model: getting a grip on sustainable development in policy making. Environment, Development and Sustainability 7(1):135–151.
Heida, H. & Poulus, C. 1993. COMBI: Een Combinatie van Modellen. Delft: Focus Onderzoek & Advies.
Jaeger, C., Renn, O., Rosa, E., & Webler, T. 1998. Decision analysis and rational action. In S. Rayner & E. Malone (Eds.), Human Choice and Climate Change, Volume 3. pp. 141-215. Columbus, OH: Battelle Press.
Jaeger, C., Renn, O., Rosa, E., & Webler, T. 2001. Risk, Uncertainty, and Rational Action. London: James & James/Earthscan.
Kates, R., Clark, W., Corell, R., Hall, J., Jaeger, C., Lowe, I., McCarthy, J., Schellnhuber, H., Bolin, B., Dickson, N., Faucheux, S., Gallopin, G., Grubler, A., Huntley, B., Jager, J., Jodha, N., Kasperson, R., Mabogunje, A., Matson, P., Mooney, H., Moore III, B., O’Riordan, T., & Svedlin, U. 2001. Sustainability science. Science 292(5517):641–642.
Meadows, D. 1999. Leverage Points: Places to Intervene in a System. Hartland, VT: The Sustainability Institute.
Organization for Economic Cooperation and Development (OECD). 1993. Environmental Indicators: Basic Concepts and Terminology. Paris: Group on the State of the Environment. Workshop on Indicators for Use in Environmental Performance Reviews. Paris: OECD.
Provincie Limburg. 2001. Liefde voor Limburg-Provinciaal Omgevingsplan Limburg (POL). Maastricht: Provincie Limburg.
Robinson, J. 2004. Squaring the circle? Some thoughts on the idea of sustainable development. Ecological Economics 48(4):369–384.
Rotmans, J. 1998. Methods for IA: The challenges and opportunities ahead. Environmental Modelling and Assessment 3(3):155–179.
Schellnhuber, H. & Wenzel, V. 1998. Earth System Analysis: Integrating Science for Sustainability. New York: Springer-Verlag.
Schumacher, E. 1977. A Guide for the Perplexed. New York: Harper Collins.
Siebenhüner, B. & Barth, V. 2005. The role of computer modelling in participatory integrated assessments. Environmental Impact Assessment Review 25(4):367–389.
Sterman, J. 1994. Learning in and about complex systems. System Dynamics Review 10(2–3):291–330.
Sterman, J. & Booth Sweeney, L. 2002. Cloudy skies: assessing public understanding of global warming. System Dynamics Review 18(2):207–240.
Tansey, J., Carmichael, J., VanWynsberghe, R., & Robinson, J. 2002. The future is not what it used to be: participatory integrated assessment in the Georgia Basin. Global Environmental Change 12(2):97–104.
Teunis, U. 1996. De ontwikkeling van een Euregionaal inform-atiesysteem onderwijs-arbeidsmarkt voor de Euregio Maas-Rijn. Working Paper ROA-W-1996/7. Maastricht: Research Centre for Education and the Labour Market.
Toth, F. 2003. State of the art and future challenges for integrated environmental assessment. Integrated Assessment 4(4):250–264.
van de Kerkhof, M. 2004. Debating Climate Change. Amsterdam: Lemma.
van der Sluijs, J. 1997. Anchoring amid Uncertainty: On The Management of Uncertainties in Risk Assessment of Anthropogenic Climate Change. Utrecht: Universiteit Utrecht.
van der Woerd, K., Dellink, R., Boelens, J., Masurel, E., & Verberne, T. 1997. The “MIOW-PROV” Model to Analyse Social Economic Consequences of Provincial Environmental Policies. IVM R-97/11. Amsterdam: Institute for Environmental Studies, Vrije Universiteit.
Vester, F. 2002. Die Kunst vernetzt zu denken. Munich: Dtv.
Weimer-Jehle, W. 2006. Cross-impact balances: a system-theoretical approach to cross-impact analysis. Technological Forecasting and Social Change 73(4):334–361.