2 Model validation and verification

2.2 Verification

Verification by comparing model output with statistical data series is too well established to warrant detailed consideration here. Numerical forecasting models have been shown on a number of occasions to be improved by expert intervention ( [3], [4], [5]). Integrating statistical forecasting models with rulebases incorporating expert judgement has been shown by Collopy and Armstrong [6] and by Moss, Artis and Ormerod [7] to improve forecasts while making the interventions explicit. Moss, Artis and Ormerod, in particular included in their system an explanations facility which provided the user with a qualitative account of the reasons for interventions by the rulebase. These qualitative reasons were couched in much the same language as that given by model operators for making similar interventions by hand.

There is therefore some precedent for including qualitatively expressed, domain expertise in models of social or economic processes and verifying the qualitative elements of such models through assessment by domain experts. A further development in this area is to integrate well verified theories from other disciplines into our computational models.

One such theory, used in two of the models reported below, is Alan Newell's unified theory of cognition [8]. The theory itself was guided by the requirement to mimic the time required by humans for cognitive acts of varying degrees of complexity. The Soar software architecture [9] is an implementation of the Newell theory which performs well when assessed against the performance of subjects in a large number of psychological experiments.

Cooper, Fox, Farringdon and Shallice [10] showed that Soar is not the only possible implementation of the Newell theory. Moss, Gaylard, Wallis and Edmonds [1] found that reimplementing Ye's and Carley's Radar-Soar model [11] in SDML reduced the number of computations and the time required to run the model by two to three orders of magnitude while replicating the Radar-Soar results.

An important issue which remains is the extent to which further verification can be obtained by comparing numerical outputs from simulation models with appropriate statistical data series. The argument of this paper is that the verification of computational models with qualitative elements can and should include empirical tests of the behavioural elements of the models, assessments by domain experts and, when possible, statistical tests of the model's numerical outputs. The descriptions of the three models are used to investigate the limits to such verification for different applications of social simulation.

Simulation and Reality - 20 MAY 98

[Next] [Previous] [Up] [Top] [Contents]

Generated with CERN WebMaker