Verificación y Validación De moDelos De Simulacion

Proceedings of the 2011 Winter Simulation Conference
S. Jain, R. R. Creasey, J. Himmelspach, K. P. White, and M. Fu, eds.

TOWARDS A MEASUREMENT TOOL FOR
VERIFICATION AND VALIDATION OF SIMULATION MODELS

Zhongshi Wang
ITIS GmbH
Werner-Heisenberg-Weg 39
D-85577 Neubiberg, GERMANY
ABSTRACT
Model deficiencies, despite their negative influences on assessment of modeling and simulation (M&S)applications, carry a large amount of insightful information, which can be used to measure different
aspects of the M&S development process and its verification and validation (V&V). Although there already
exist various categorizations of model deficiencies, none of which can be used as a measurement tool to
classify and analyze deficiency data collected in practice. This paper describes aframework for developing
model deficiency classifications and the pilot application of the established classification scheme as a
quantitative method to measure and control the V&V process being conducted. Investigations indicate that
the proposed approach is capable of providing process diagnoses reflecting the real problems and identifying
the improvement potentials. Based on the findings achieved inpractice, a planning and tailoring concept
is being developed for efficiently applying different V&V techniques in a simulation study.
1

INTRODUCTION

Development of modeling and simulation (M&S) applications involves human- and knowledge-intensive
activities, in which errors, uncertainties, or inadequacies leading to quality deficiencies in models and
simulation results are inevitable.Verification and validation (V&V) focuses on assessing the accuracy
quality characteristic of an M&S application with respect to its objectives (Balci 2004, Shannon 1975),
and is intended to ensure that only correct and suitable models and simulation results are used in practice.
Model deficiencies, on the one hand, due to their negative impact on the assessment issue, should be
prevented from thedevelopment’s point of view, or be identified and corrected from the V&V’s point of
view. On the other hand, a large amount of information is attached to the model deficiencies, characterizing
the different aspects of model development and its V&V. In order to extract the semantic information of
interest from the collected deficiency data, classification of model deficiencies is required. However,there
is currently not sufficient research in this area.
In the M&S literature, the term “errors” is frequently used to refer to model deficiencies. Its meaning
is sometimes, however, extremely different, such as the definitions of type I, II, and III errors (Balci and
Nance 1985), and of acknowledged or unacknowledged errors (Oberkampf et al. 2000). Typically, model
deficiencies are classified intoseveral coarse-grained categories relating to the corresponding development
phases in the M&S life cycle (Shannon 1975, Carson 2002, Schmeiser 2001), such as data model errors,
logic model errors, programming errors, experimentation errors, or interpretation errors. Such taxonomies
cannot be used as a basic instrument to measure and analyze the semantics included in model deficiencies.
In thesoftware community, several defect classification schemes and even an IEEE standard (IEEE
1993) have already been developed. In publications, the most referenced classification approaches are the
IBM Orthogonal Defect Classification (ODC) scheme (Chilarege et al. 1992) and the Hewlett-Packard
(HP) scheme (Grady 1992). The basic concept of software defect classifications can be outlined as follows:978-1-4577-2109-0/11/$26.00 ©2011 IEEE

581

Wang
1. defining a classification scheme with different attributes;
2. using the defined classification scheme to extract the semantic information from individual defects;
3. analyzing and evaluating the aggregated defect data.
Findings of the investigation indicate that the software defect classifications (ODC and other schemes)
can not be...