Uses and Limitations of Insights from Integrated Assessment Modeling
Essay Prepared for SEDAC by:
Milind Kandlikar and James Risbey
Department of Engineering and Public Policy
Carnegie Mellon University
Pittsburgh, PA 15213
- Use and Limits of Large Scale Modeling
- Insights from Integrated Assessment Modeling
This document discusses the uses and limitations of insights drawn from integrated assessment models (IAMs). The focus here is on models, and we do not explicitly address non-modeling methods for carrying out integrated assessments (IAs) and the insights they may generate. In the rest of this section, we describe the goals of IAMs. These goals characterize the types of insights that might be drawn from IAMs. Section 2 provides a general framework for assessing insights from IA models by describing the reasons for using models and the limitations of large scale modeling efforts. In section 3 we list insights from IAM exercises, and provide informal evaluations of their merit according to several different criteria.
The literature on definitions of IA and IA modeling reveals several different kinds of goals. For instance, in Morgan and Dowlatabadi (1996), IA models are viewed as research tools, where a model provides an organizing framework for conducting research by ensuring consistency and pointing to substantive areas where more information is required. The actual research program emerges iteratively from the insights that the model provides and from investigations in the substantive domains of the sub-components. In Rotmans, Dowlatabadi, and Parson (1996) the definition of IA emphasizes that IAs should provide useful information for making policy decisions. This is a more ambitious goal, though it seems to accord with the general intent of IA modeling as carried out by most IA modeling groups.
In the process of running IAMs to develop useful information for policy makers, IA modelers usually make a distinction between use of IAMs as 'truth machines' or 'forecasting tools' and 'heuristic tools'--favouring the latter view. That is, the models should not be viewed as forecasting tools for use in policy making. Rather, a heuristic approach should be taken in which less confidence is placed on the actual model results. In this view, quantitative model outputs are used to make qualitative judgments about impacts (e.g., large, small), abatement (e.g., high, low) and dynamics (e.g., now, later), and about the interplay of processes in the model. These qualitative judgments have been termed 'insights'--hence the focus on 'insights' in this document.
In order to assess the uses and limitations of insights from IAMs it is useful to review the uses and limits of past large scale modeling efforts. There is a rich literature on this subject, and we draw from it in the next section.
There are a variety of advantages and disadvantages to developing IAMs. Models provide analysts with opportunities to formalize their reasoning and ideas in a single consistent framework. This can be helpful in the creative process. In addition, some results entailing complex interactions between subsystems require formal computation, and could not be obtained using purely theoretical, analytical, or empirical techniques. Furthermore, in complex systems with feedbacks and nonlinearities, results may often be non-intuitive (Ascher 1981); models may help describe the dynamical properties of such systems, for example by revealing stable and unstable equilibria (Simon 1990).
The Limits to Growth Models of the 1970s provide sobering lessons on long term predictive modeling; long range models suffer from an inherent inability to characterize and parameterize long term interactions between the economy, society, and environment. Even if it were possible to describe all interactions between model variables, one would still need to parameterize the model. A non-linear dynamic system of more than four variables can show essentially any behavior (Smale 1976); the form of the model does not significantly constrain its behavior. The parameters and initial conditions of the assumed model form are just as important. Additionally, individual choice and co-evolution may make socio-economic models inherently unpredictable at the macro-level (Land and Schneider 1987).
Such unpredictability is observed in practice. Models of socio-economic phenomena are reasonably good predictors in the very short-term. However, they show poor performance and make noticeably overconfident predictions for longer time periods (Keepin 1986; Shlyakter et al. 1993), partly because they do not seem to capture long term "weak" feedback relationships. As a result, long range models may never be able to make meaningful predictions; the goal should instead be to achieve qualitative realism without imposing assumptions of causal determinism between variables (Ayres 1984). Echoing similar ideas, Simon (1990) suggests an approach to modeling for achieving prescriptive (as opposed to predictive) goals.
The constraint of qualitative realism in complex modeling requires a shift away from producing forecasts to developing insights. Embedded in the notion of an insight is the idea that qualitative judgments are more robust than quantitative forecasts. However, use of the model as a tool for generating insights relies on having some level of confidence in model forecasts because qualitative judgments must always be inferred from quantitative model output. Thus the distinction between IAMs as truth machines and heuristic tools is not a clean one. This blurring of roles presents a 'Catch-22' for IA modeling. On the one hand, the generation of insights from IAMs depends to some degree on their ability to provide reliable quantitative information. Yet, evaluating the reliability of the forecast inevitably reduces to an evaluation of insights (Risbey, Kandlikar, and Patwardhan 1996).
The discussion in the preceding section suggests that the potential of IA modeling is limited to providing qualitative understanding about the dynamics and interactions of global systems. Although this may sound like a modest goal, achieving it would put climate change IA modeling well ahead of previous global modeling efforts. In what follows we outline and discuss insights drawn from current IA efforts, examine their limitations and point to opportunities for extracting more utility from IA models.
According to definitions of IAs, insights from modeling should be of use for policy. Broadly speaking this places two constraints on IA insights: they should be specific enough to help in formulation of concrete policy decisions, and they should be robust enough that some measure of confidence can be ascribed to them. Additionally, IA insights hope to "have added valued compared to single disciplinary oriented assessment" (Rotmans, Dowlatabadi, and Parson 1996). It is therefore appropriate to ask if these insights are unique to IA, or whether they could be achieved through disciplinary analyses and qualitative reasoning. The level of specific policy relevance of IA insights, the subjective level of confidence and the underlying core assumptions, and their uniqueness to IA modeling together form the criteria we use for assessing the limitations and uses of IA model insights. In table 1 below we examine insights from IA models of climate change on the basis of these criteria.
The list of insights below, though not exhaustive, is an informed distillation based on literature, review papers, workshops and conversations with modelers. Our judgments on the specificity, confidence, and uniqueness of the insights are, of course subjective, and space limits explication of every judgment. More details on the insights and their sources can be found in literature on overviews of IAs, such as Rotmans, Dowlatabadi, and Parson (1996) and IPCC (1996). The cites below are offered as examples of each insight only, and are not intended to be exhaustive.
TABLE 1: INSIGHTS FROM IA MODELSInsight Specificity Core Assumptions Confidence Uniqueness to IAMs 1 Near term delays medium--high political feasibility; low medium--high (10--30 years) in technological Requires formal emissions reductions development exogenous; computation are reasonable intragenerational equity issues manageable; smooth changes in nat. and soc. systems 2 Low abatement is medium discounting; low high optimal impacts small; Requires formal single rational actor; computation ecosystem valuation 3 Impacts from climate medium adaptation rapid; low low--medium change are a small ecosystems resilient; Derives from sectoral percent of GNP dollar metrics impact research appropriate and economic analysis 4 Stabilization of medium knowledge of carbon high low--none concentrations at cycle Derives from carbon present levels cycle models requires drastic emissions reductions 5 Sulphate aerosols are medium aerosol physics high low--none important to take into Initially derived account in assessing from atmospheric abatement policies science 6 Cooperation between low virtually high low nations leads to unfalsifiable Follows from elementary higher net benefits economic considerations 7 Value judgments low diversity of values high low dominate scientific is important uncertainty 8 R \ D in institutions low past and present high low and technologies for experience; adaptation and lower engineering analyses material use are worthwhile 9 Changes in ecosystems, low past and present high low agricultural experience management, and urbanization are important
We begin our discussion of the table with insights (1, 2, and 3) that have the highest specificity, and which are unique to IA. Thus, these IA insights are best placed for making contributions to policy. Model results supporting near term delays in emissions reduction (insight 1) have been put forth by several investigators (Schlesinger and Jiang 1991; Hammitt, Lempert, and Schlesinger 1992; Wigley, Richels, and Edmonds 1996). These results are of critical importance to near term implementation and long term consequences of climate change. However, the assumptions on which these conclusions are based are tenuous and extensively debated in the literature (Risbey, Handel, and Stone 1991; Grubb 1996; Rotmans, Dowlatabadi, and Parson 1996). As a result, we have ascribed a low level of confidence to this insight.
The insight that low abatement is optimal (insight 2--e.g., Nordhaus 1994) stems from a balancing of the costs of emissions reduction against the potential future benefits of emissions reduction. This insight is based on the assumption that future impacts from climate change will be smaller than the costs of extensive abatement (insight 3--e.g., Nordhaus 1994); it also depends on the choice of a single exponential discount rate. The insight is also drawn from the perspective of a mythical global commoner who faces globally averaged impacts, and values ecosystems using valuation techniques such as 'willingness to pay' that have known flaws (Diamond and Hausman 1994; Fischoff 1991). A more detailed assessment of how these and related assumptions breakdown in the context of global climate change is given in Morgan et al. (1996). We ascribe a low level of confidence to these insights because of the low level of robustness in the underlying assumptions described above.
The list of insights in table 1 includes two other insights which are reasonably specific (insights 4 and 5--IPCC 1996). For these insights high levels of confidence can be applied. However, these insights were not originally developed from IAMs. They were first developed within the scientific disciplines--biogeochemistry (insight 4) and atmospheric physics (insight 5). Disciplinary literature describing these results predates their discussion in the IA literature. Examples from the disciplinary literature are Harvey (1989), Watson et al. (1990) based on Enting and Pearman (1985), and Goudriaan (1989) for insight 4; and Wang et al. (1986), Charlson, Langer, and Rodhe (1990), Charlson et al. (1992), Kiehl and Briegleb (1993) for insight 5.
Insights 6 through 9 have a high level of confidence associated with them, but are of low specific relevance to policy. Examples of these insights may be found in Richels et al. (1996) for insight 6, Lave and Dowlatabadi (1993) for insight 7, and IPCC (1996) for insights 8 and 9. The validity of these insights is broadly accepted, but they provide only the most general policy implications. Furthermore, these insights can be derived independently of IAMs using qualitative reasoning and theoretical considerations. To be sure, the role of IA in confirming broad insights derived elsewhere should not be dismissed, since independent confirmation may contribute to their ultimate acceptance.
If the goal of IA is the provision of policy recommendations, then the best insights would need to be specific, robust, and unique to IA: i.e., the insights would score medium or better along the respective categories. In our discussion in section 2, we noted that enterprises like IA modeling can only hope for a measure of qualitative realism. The 'Catch-22' of IA modeling described earlier implies that development of qualitative insight is limited by the same kinds of model ambiguity and uncertainty that hinder long term forecasting--the more specific the insight, the more it depends on the actual quantitative results. It is therefore unlikely that IA modeling can provide specific policy recommendations with high levels of confidence, and it is not surprising that such combinations are not evident in the table. Rather the general observation from table 1 is that more specific insights have low confidence and vice versa. In other words, there is an inverse relationship between the degree of specific relevance of the insight and the levels of confidence associated with it. Note that insights 4 and 5, provide exceptions to the inverse relationship in featuring high specificity and confidence. However, these insights arise from more narrow natural science assessments and are not subject to the full set of indeterminacies that long term global models face (see section 2).
If the inverse relationship described above is accurate, then the goal of providing direct policy recommendations may be too ambitious. Yet, we see opportunites for IA to fulfil a more realistic and useful role indirectly. For policy relevance, we seek high levels of specificity on insights without being compromised by the indeterminacies which plague long term modeling efforts. A tentative path forward is pointed to by insights with higher specificity and confidence associated with them (insights 4 and 5). These insights are derived from more focused 'sub-integrated' assessments that do not seek to incorporate the whole cause effect chain of natural and social systems over long time scales and multiple regions. In a sub-integrated framing the complexity of the analysis is truncated by focusing on some specific sub-systems or specific regions. Insights 4 and 5 are of the former type, stemming from a focus on specific sub-systems. Nonetheless, the insights are more directly relevant to setting policy responses than those derived from more holistic IAMs. Regional sub-integrated assessments attempt to encompass multiple interactions within a specific region, while making compromises on the inclusion of external interactions. Such approaches usually include a mix of modeling and qualitative studies, and incorporate the interests of particular stakeholders in the region. Regional studies have the potential to produce more specific policy insights, albeit at a regional level. An example of such an approach is provided by Cohen (1995).
However, assessments that truncate the complexity of the analysis by leaving out some system interactions or focusing on specific regions may miss some of the insights that follow from the attempt at holism in IA. This suggests an approach akin to the research mode described earlier where the IAM is used as an organising framework to spawn more limited detailed studies. In this way IAMs may be used indirectly to generate policy insights with higher levels of specificity and confidence. A useful framework for combining IAMs with sub-integrated assessments can be drawn from Root and Schneider's (1995) strategic cyclical scaling (SCS) paradigm in which "[l]arge-scale associations are used to focus small-scale investigations to ensure that tested causal mechanisms are generating the large-scale relations." The application of SCS to IA studies would lead to IA models providing meta-insights. Focused empirical and disciplinary studies could be used to probe the causal relations implicit in these insights and to suggest further avenues for exploration in the modeling framework. Thus IA models could serve as organising frameworks for coordinating insights, while spawning detailed process studies to inform and refine those insights.
The above analysis of insights drawn from IAMs is based on insights drawn from IA modeling exercises to date. It may be that the real potential of IA models is yet to be revealed, and that this list represents the immature gropings of a nascent research field. There is of course an element of truth to this. However, given the more theoretical and practical limitations on large scale modeling efforts described in section 2, we are not optimistic that IAMs will be able to provide policy recommendations that are both highly specific and robust.
The definitions of IA in the literature suggest dual uses for IA as research frameworks and as providers of policy relevant insights. Past experience with large scale modeling efforts suggests that IAMs have the potential to provide qualitative realism at best, consistent with the provision of insights, but not forecasts. However, the distinction between insights and forecasts is blurry because insights depend upon quantitative model forecasts. Attempts to evaluate the robustness of forecasts inexorably lead to evaluations of insights. This 'Catch-22' also applies to the drawing of policy relevant insights from IAMs. The more specific the insight, the more it relies upon detailed quantitative simulation of processes in the model, and the less robust the resulting insight. This leads to a situation where one has little confidence in the potentially most useful insights from IAMs, and high confidence in the more generic insights that can be learned through other sources. In order to generate policy insights that have the dual characteristics of being robust and specific enough to be useful, it may be necessary to pursue more limited sub-integrated assessments. In this approach, IA models can still serve the role of an organizing framework for directing activity in more detailed studies outside of the model.
Cohen, S.J. 1995. An interdisciplinary assessment of climate change on northern ecosystems: The Mackenzie Basin Impact Study. In Human Ecology and Climate Change--People and Resources in the Far North, eds. D.L. Peterson and D.R. Johnson, 301-316. Washington, D.C.: Taylor and Francis.
Goudriaan, J. 1989. Modelling biospheric control of carbon fluxes between atmosphere, ocean and land in view of climatic change. In Climate and Geo-sciences (NATO-ASI Series C:285), eds. A. Berger et al., 481-499. Dordrecht, The Netherlands: Kluwer.
Rotmans, J., H. Dowlatabadi, and E. Parson. 1996. Integrated Assessment of Climate Change: Evaluation of Methods and Strategies. In Human Choice and Climate Change: A State-of-the-Art Report. In Press.
Watson, R., H. Rodhe, H. Oeschger, and U. Siegenthaler. 1990. Greenhouse gases and aerosols. In Climate Change: The IPCC Scientific Assessment, eds. J.T. Houghton et al., 1-40. Cambridge: Cambridge University Press.
Weyant, J., and 10 others 1996. Integrated assessment of climate change: An overview and comparison of approaches and results. In Intergovernmental Panel of Climate Change: Working Group III Report. In press.
Center for International Earth Science Information Network (CIESIN). 1995. Thematic Guide to Integrated Assessment Modeling of Climate Change [online]. University Center, Mich.
CIESIN URL: http://sedac.ciesin.org/mva/iamcc.tg/TGHP.html