Model-independent comparison of simulation output

  title={Model-independent comparison of simulation output},
  author={Nuno Fachada and Vitor V. Lopes and Rui Costa Martins and Agostinho C. Rosa},
  journal={Simul. Model. Pract. Theory},
Abstract Computational models of complex systems are usually elaborate and sensitive to implementation details, characteristics which often affect their verification and validation. Model replication is a possible solution to this issue. It avoids biases associated with the language or toolkit used to develop the original model, not only promoting its verification and validation, but also fostering the credibility of the underlying conceptual model. However, different model implementations must… 
Chapter 9 Verifying and Validating Simulations
Verification and validation are two important aspects of model building. Verification and validation compare models with observations and descriptions of the problem modelled, which may include other
Verifying and Validating Simulations
This chapter deals with a comprehensive outline of methodological perspectives and practical uses of verification and validation in the context of simulating social complexity and the problem of evaluating simulations.
Use of Graphics Processors for Agent-Based Simulations
Agent-based modelling is a programming and simulation paradigm. It is a bottom-up approach where interacting agents are its basic building blocks. However, ABMs must undergo a process of independent
Towards a standard model for research in agent-based modeling and simulation
This paper presents PPHPC, a model which aims to serve as a standard in agent based modeling research, namely, but not limited to, conceptual model specification, statistical analysis of simulation output, model comparison and parallelization studies.
Theory Development Via Replicated Simulations and the Added Value of Standards
This work shows how a replicated simulation model can be used to develop theory and shows the potential of replicated simulation models for theory refinement, such as analyzing in-depth the relationship between memory functions and routine performance or routine adaptation.
Lessons learned on development and application of agent-based models of complex dynamical systems
  • R. Williams
  • Computer Science
    Simul. Model. Pract. Theory
  • 2018
The view that agent-based models can be useful for answering questions of the real-world domain through predictive modelling is reinforced, but it is emphasised that all modellers must make a concerted effort to adopt robust methods and techniques for constructing, validating and analysing their models, if the result is to be meaningful and grounded in the system of interest.
Model alignment using optimization and design of experiments
This paper presents an extension of the model alignment methodology for comparing the outcome of two simulation models that searches the response surface of both models for significant differences.
The use of simulation modeling for scientific tasks demands that these models be replicated and independently verified by other members of the scientific community. However, determining whether two
Assessing the feasibility of OpenCL CPU implementations for agent-based simulations
It is shown that there are considerable gains in using CPU-based OpenCL for developing and implementing ABMs, with speedups up to 10x over the parallel Java version on a 10-core hyper-threaded CPU.
Using Multiple Scale Space-Time Patterns to Determine the Number of Replicates and Burn-In Periods in Spatially Explicit Agent-Based Modeling of Vector-Borne Disease Transmission
This study proposes to use multiple scale space-time patterns to determine the number of required replicates and burn-in periods in spatially explicit ABMs, and develops an indicator for these purposes, which enables to take into consideration a successive coefficient of variance (CV) over replicate and simulation years.


Validating Simulations
Why Read This Chapter? To help you decide how to check your simulation – both against its antecedent conceptual models (verification) and external standards such as data (validation) – and in this
A template model for agent-based simulations
A template ABM is presented which aims to serve as a basis for a series of investigations, including, but not limited to, conceptual model specification, statistical analysis of simulation output, model comparison and model parallelization, and providing a canonical implementation of the model, such that it serves as a complete reference to the presented model.
Aligning simulation models: A case study and results
This paper develops the concepts and methods of a process we will call “alignment of computational models” or “docking” for short. Alignment is needed to determine whether two models can produce the
Making Models Match: Replicating an Agent-Based Model
This paper presents a case study of an agent-based model from political science that was developed by Axelrod and Hammond and details an effort to replicate that model and the challenges that arose in recreating the model and in determining if the replication was successful.
Replicating and breaking models: good for you and good for ecology
It is argued that a culture of replication would lead to increased credibility, coherence and efficiency of computational modelling and thereby facilitate theory development.
The Complexities of Agent-Based Modeling Output Analysis
Issues surrounding variance stability, sensitivity analysis, spatio-temporal analysis, visualization, and effective communication of all these to non-technical audiences, such as various stakeholders are examined.
Validation of Simulation Models via Simultaneous Confidence Intervals
SYNOPTIC ABSTRACTThe purpose of this paper is to present state-of-the-art and new research results on the use of simultaneous confidence intervals (and joint confidence regions) for determining the
Horizontal and Vertical Multiple Implementations in a Model of Industrial Districts
This paper discusses strategies concerning the implementation of an agent-based simulation of complex phenomena, and provides some important suggestions in terms of model validation, showing how models of complex systems tend to be extremely sensitive to implementation details.
Standardised and transparent model descriptions for agent-based models: Current status and prospects
A minimum standard of model description is suggested for good modelling practice, namely the provision of source code and an accessible natural language description, and a classification for structuring types of model descriptions is suggested.
Modeling Complex Systems
Modelling of ecological systems should include and combine strategic interactions among economic agents, nonconvexities induced by non-linear feedbacks, separate spatial and temporal scales and modeling of spatiotempo-ral dynamics, and allowance of alternative time scales.