Bride and prejudice

Proceedings of the 1999 Winter Simulation Conference P. A. Farrington, H. B. Nembhard, D. T. Sturrock, and G. W. Evans, eds.

DESIGNING SIMULATION EXPERIMENTS

W. David Kelton Department of Quantitative Analysis and Operations Management College of Business Administration University of Cincinnati Cincinnati, OH 45221-0130, U.S.A.

ABSTRACT This tutorial introduces some of the ideas, issues,challenges, solutions, and opportunities in deciding how to experiment with a simulation model to learn about its behavior. Careful planning, or designing, of simulation experiments is generally a great help, saving time and effort by providing efficient ways to estimate the effects of changes in the model’s inputs on its outputs. Traditional experimental-design methods are discussed in thecontext of simulation experiments, as are the broader questions pertaining to planning computer-simulation experiments. 1 INTRODUCTION

kind of optimal system configuration. Specific questions of this type might include: • • • • • What model configurations should you run? How long should the runs be? How many runs should you make? How should you interpret and analyze the output? What’s the mostefficient way to make the runs?

The real meat of a simulation project is running your model(s) and trying to understand the results. To do so effectively, you need to plan ahead before doing the runs, since just trying different things to see what happens can be a very inefficient way of attempting to learn about your models’ (and hopefully the systems’) behaviors. Careful planning of how you’regoing to experiment with your model(s) will generally repay big dividends in terms of how effectively you learn about the system(s) and how you can exercise your model(s) further. This tutorial looks at such experimental-design issues in the broad context of a simulation project. The term “experimental design” has specific connotations in its traditional interpretation, and I will mention some ofthese below, in Section 5. But I will also try to cover the issues of planning your simulations in a broader context, which consider the special challenges and opportunities you have when conducting a computer-based simulation experiment rather than a physical experiment. This includes questions of the overall purpose of the project, what the output performance measures should be, how you use theunderlying random numbers, measuring how changes in the inputs might affect the outputs, and searching for some

These questions, among others, are what you deal with when trying to design simulation experiments. My purpose in this tutorial is to call your attention to these issues and indicate in general terms how you can deal with them. I won’t be going into great depth on a lot of technicaldetails, but refer you instead to any of several texts on simulation that do, and to tutorials and reviews on this subject in this and recent Proceedings of the Winter Simulation Conference. General book-based references for this subject include chapter 12 of Law and Kelton (1991), chapter 11 of Kelton, Sadowski, and Sadowski (1998), Banks, Carson, and Nelson (1996), and Kleijnen (1998), all of whichcontain numerous references to other books and papers on this subject. Examples of application of some of these ideas can be found in Hood and Welch (1992, 1993) and Swain and Farrington (1994). Parts of this paper are taken from Kelton (1997), which also contains further references and discussion on this and closely related subjects. 2 WHAT IS THE PURPOSE OF THE PROJECT?

Though it seems likepretty obvious advice, it might bear mentioning that you should be clear about what the ultimate purpose is of doing your simulation project in the first place. Depending on how this question is answered, you can be led to different ways of planning your experiments. Worse, failure to ask (and answer) the question of just what the point of your project is can often

33

Kelton leave you...