The Variability in Design-Quality Measures for Multiple Types of Space-Filling Designs Created by Leading Software Packages

Abstract

Space-filling designs (SFDs) underpin many large-scale simulation studies. The algorithms that construct SFDs are mostly stochastic and cannot guarantee that optimal solutions can be found within a practical amount of time. This paper uses massive experimentation to find the empirical distributions of a diverse set of design-quality measures in highly-used classes of SFDs constructed by leading software packages. The objective is to provide simulation practitioners with a better understanding of what they can expect from different SFD choices. The results show substantial variability in measures of correlation and space-fillingness in the design classes and dimensions investigated. Therefore, computer experimenters should generate and assess several candidate designs using different random-number-generator seeds to reduce the risk of using a poor design simply due to random chance. We also find that in the largest designs investigated, the uniform designs generally perform best for both our correlation and uniformity measures.