Evaluating the probability of a good selection

We present a two-stage experiment design for use in simulation experiments that compare systems in terms of their expected (long-run average) performance. This procedure simultaneously achieves the following with a prespecified probability of being correct: (a) find the best system or a near best system; (b) identify a subset of systems that are more than a practically insignificant difference from the best; and (c) provide a lower bound on the probability that the best or near best system has actually been selected. The procedure assume normally distributed data, but allows unequal variances.