ENHANCING KRIGING BASED OPTIMIZATION THROUGH NEW

SEARCH CRITERIA

KEVIN CHRISTIAN

DEPARTMENT OF INDUSTRIAL SYSTEMS ENGINEERING AND

MANAGEMENT
NATIONAL UNIVERSITY OF SINGAPORE

2016/2017
Enhancing Kriging Based Optimization Through New Search Criteria

Submitted by
Kevin Christian

Department of Industrial Systems Engineering and Management

In partial fulfillment of the requirements for

the Degree of Bachelor of Engineering
National University of Singapore

2016/2017
Summary

This project aims to improve the current kriging-based simulation optimization,

namely the Two Stage Sequential Optimization (TSSO) algorithm developed by Quan et

al. (2013). Two ranking and selection procedures are explored in this project. The

procedures are Stop and Go with fixed first stage sample size (SaG-F) and Stop and Go

with variable first stage sample size (SaG-V). Each of these procedures are incorporated to

the TSSO by replacing Optimal Computing Budget Allocation (OCBA) algorithm by

SaG-F and SaG-V, thus resulting in two new algorithms TSSO with SaG-F and TSSO

with SaG-V. Three test functions are used in this project to test the two new algorithms

and compare their results with the original TSSO algorithm. In each function, different

levels of noise and budget are used to test the algorithms. The results of this project are

that TSSO with SaG-F is a promising algorithm that may be used in the future for kriging-

based optimization while TSSO with SaG-V does not seem to be a very promising

algorithm. Future research may explore the settings of probability of correct selection

(PCS) and indifference zone (IZ) parameter of SaG-F and SaG-V, and the first stage

sample size function of SaG-V.

I
Acknowledgement

I would like to give all glory and praise to Lord Jesus Christ, in whom all things

are held together, because all things were created through him and for him. It was through

his providence that I could complete this project. I would like to express my gratitude to

my supervisor, Associate Professor Ng Szu Hui from NUS Department of Industrial

Systems Engineering and Management, whose guidance, insight, and expertise have been

invaluable for the completion of this project. I would like to thank Ms. Meng Qun,

researcher at National University of Singapore, who have provided me help with the

technical aspects of this project. I would also like to give credit to Mr. Stefanus Lie, my

friend, who has helped me with the unfamiliar mathematical formulations in the journal

papers. Lastly, I would like to thank my family and friends for the continued support

during the course of this project.

II
Table of Contents

Summary ................................................................................................................................I
Acknowledgement ............................................................................................................... II
Table of Contents ................................................................................................................ III
List of Figures ..................................................................................................................... IV
List of Tables ....................................................................................................................... V
Chapter 1. Introduction ......................................................................................................... 1
Chapter 2. Background ......................................................................................................... 2
Chapter 3. Methodology ....................................................................................................... 9
Chapter 4. Results ............................................................................................................... 20
Chapter 5. Discussion ........................................................................................................ 27
Chapter 6. Conclusion and Recommendation ..................................................................... 31
References ........................................................................................................................... 33
Appendices .......................................................................................................................... 35

III
List of Figures

Figure 2.1 Typical Steps in Kriging Based Optimization

Figure 3.1 Rescaled Branin

Figure 3.2 Candidate Points and Contour Plot of Rescaled Branin

Figure 3.3 Six-hump camel-back

Figure 3.4 Candidate Points and Contour Plot of Six-hump camel-back

IV
List of Tables

Table 3.1 Test Functions

Table 4.1 Results for Rescaled Branin Light Noise-Low Budget

Table 4.2 Results for Rescaled Branin Heavy Noise-Low Budget

Table 4.3 Results for Rescaled Branin Light Noise-High Budget

Table 4.4 Results for Rescaled Branin Heavy Noise-High Budget

Table 4.5 Results for Six-hump Camel-back Light Noise-Low Budget

Table 4.6 Results for Six-hump Camel-back Heavy Noise-Low Budget

Table 4.7 Results for Six-hump Camel-back Light Noise-Low Budget

Table 4.8 Results for Six-hump Camel-back Light Noise-Low Budget

Table 4.9 Results for Hartmann-3 Light Noise-Low Budget

Table 4.10 Results for Hartmann-3 Heavy Noise-Low Budget

Table 4.11 Results for Hartmann-3 Light Noise-High Budget

Table 4.12 Results for Hartmann-3 Heavy Noise-High Budget

V
Chapter 1. Introduction

Kriging is one of many methodologies for simulation metamodeling. Optimization

of computationally costly simulations can be estimated by optimizing metamodels as

surrogates for the costly simulation response functions (Barton & Meckesheimer, 2006).

In general, there are five different types of metamodels, namely, response surface,

regression spline, kriging, radial basis function and neural network metamodels (Barton &

Meckesheimer, 2006). In this project, we will focus on kriging based optimization.

In kriging based optimization, different methods are employed to find an optimal

solution. Some of the kriging-based methods for simulation optimization are minimum

quantile (MQ) (Picheny, Wagner, & Ginsbourger, 2013), sequential kriging optimization

(SKO) (Huang, Allen, Notz, & Zeng, 2006), correlated knowledge-gradient (CKG)

(Frazier, Powell, & Dayanik, 2009), expected quantile improvement (EQI) (Picheny,

Ginsbourger, Richet, & Caplin, 2013a), two-staged sequential optimization (TSSO)

(Quan, Yin, Ng, & Lee, 2013) and extended two-stage sequential optimization (eTSSO)

(Liu, Pedrielli, & Ng, 2014). Every method usually has these steps, namely, search,

replication (optional) and identification steps.

A research which compared the performances of all the six kriging based methods

mentioned above was conducted by Jalali, Van Nieuwenhuyse, and Picheny (2016). One

of the conclusions states that TSSO simulates the most promising alternatives but usually

fails to identify them at the end (Jalali et al., 2016). In this project, we are going to explore

the possibility of modifications of these steps to improve TSSO performance by using two

ranking and selection (R&S) procedures, namely Stop and Go with fixed first-stage

sample size (SaG-F) and Stop and Go with variable first-stage sample size (SaG-V).

1
Chapter 2. Background

In this chapter, the background on Kriging-based optimization, Two Stage

Sequential Optimization (TSSO), and the two SaG procedures mentioned in Chapter 1

will be explained.

2.1 Kriging-Based Optimization

Assume that we would like to find the solution that minimizes some goal function

𝑓(x): min 𝑓(x) , where 𝑓 ∶ Θ → ℝ and x = (𝑥1 , 𝑥2 , … , 𝑥𝑑 ) (with d the dimension of the
x∈Θ

solution space). This goal function cannot be directly observed, and stochastic simulation

model to be employed to provide an estimation: it means we can only look at the noisy

observations 𝑓̃𝑗 (x𝑖 ) = 𝑓(x𝑖 ) + 𝜀𝑗 (x𝑖 ), where 𝑓̃𝑗 (x𝑖 ) represents the observed goal value in

the jth simulation replication at point xi. We have a heterogeneous noise, 𝜀𝑗 (x𝑖 ) which has

mean zero, and its variance depends on xi. We usually estimate the value of 𝑓(x𝑖 ) by

𝑛 𝑓̃𝑗 (x𝑖 )
̅ 𝑖) = ∑ 𝑖
performing ni simulation replications: 𝑓(x .
𝑗=1 𝑛𝑖

When Θ is continuous, Kriging-based or Bayesian optimization is among the few

techniques that can handle this problem with low problem dimensionality (d ≤ 20)

̅ 𝑖 ) for several xi,

(Brochu et al., 2010; Preuss et al., 2012). Based on the observed 𝑓(x

Kriging provides a metamodel for 𝑓(x). In recent years, several Kriging-based

optimization algorithms have been proposed that can handle heterogeneous noise, based

on stochastic Kriging models (see Cressie (1993), Staum (2009), Ankenman et al. (2010),

and Yin et al. (2011)).

2
 In general, kriging-based simulation optimization methods consist of these steps

(Jalali, et al. 2016):

1. Initial Fit: Fit an initial kriging metamodel based on a set of n0 design points: xi,

𝑓 (̅ x𝑖 ), 𝑉𝑎𝑟
̂ [𝑓(̅ x𝑖 )] for i = 1, …, n0

2. Search step: This is the step where the algorithm chooses a point (infill point),

simulates it for some number of replications to obtain the mean and variance, and

then refits the kriging model with the new point included.

3. Replication step (Optional): In this step, we perform extra replications on some

points already sampled to get a more accurate measure of the mean in those points.

4. Identification step: The algorithm looks at all the simulated points and picks the

best solution of the optimization problem.

Figure 2.1 Typical Steps in Kriging Based Optimization (Jalali et al., 2016)

3
2.2 Two Stage Sequential Optimization (TSSO)

Two Stage Sequential Optimization (TSSO) is developed by Quan et al. (2013). To

fit the model, TSSO uses Modified Nugget Effect Kriging (MNEK) which is suitable for

simulation with heteroscedastic variance (Yin et al. 2011). TSSO algorithm consists of

search, replication, and identification steps.

In the search step, the following modified expected improvement (MEI) function is

used:

MEI = E(max(fmin − 𝑓𝑃∗ (x), 0)

fmin is the predicted response at the sampled point with the lowest mean. 𝑓𝑃∗ (x) is a normal

random variable with mean given by the MNEK predictor at x and variance given by the

predictor’s spatial prediction uncertainty (Quan et al., 2013).

In the replication stage, the TSSO uses the optimal computing budget allocation

(OCBA) which allocates most of the replication budget to the sampled points with low

sample mean and high variance (Chen et al., 2000). The explanation below shows how the

replication budgets are allocated. Assuming there are n sampled points, with each point xi

having a sample mean given by 𝑌̅i, sample variance 𝜎̂𝜀 (x𝑖 ) and replication number Ri,

according to Theorem 1 provided by Chen et al. (2000), the Approximate Probability of

Correct Selection (APCS) can be asymptotically maximized when available computing

budget tends to infinity and when

2
𝑁𝑖 𝜎̂𝜀 (x𝑖 )/∆𝑏,𝑖
=( ) 𝑖, 𝑗 ∈ {1, 2, . . . , 𝑛} and 𝑖 ≠ 𝑗 ≠ 𝑏,
𝑁𝑗 𝜎̂𝜀 (x𝑗 )/∆𝑏,𝑗

4
 𝑛
𝑁𝑖2
𝑁𝑏 = 𝜎̂𝜀 (x𝑏 )√ ∑
𝜎̂𝜀2 (x𝑖 )
𝑖=1,𝑖≠𝑏

Where Ni is the number of replications allocated to point xi, xb is the point with the lowest

sample mean and ∆𝑏,𝑖 is the difference between the lowest sample mean and the sample

mean at point xi.

In the identification step, the sampled point with the lowest sample mean is chosen

as the location of best response.

In the paper by Quan et al. (2013), the performance of TSSO is only compared

with the EQI method. To get a clearer comparison of how TSSO stands among the

methods of Kriging-Based Simulation Optimization, please refer to Jalali et al. (2016).

2.3 Stop and Go (SaG) Procedure

Stop and Go is a ranking and selection (R&S) procedure used for evaluating

systems generated by simulation that are revealed sequentially. This procedure is

introduced by Hong and Nelson (2007). The SaG will select the best system by using a

frequentist approach. In this procedure, probability of correct selection (PCS) and

indifference zone parameter (IZ) need to be specified. The selection procedures select the

best system with a Probability of Correct Selection (PCS) greater than or equal to 1 − α

from a given set of alternative systems whenever the true mean performance of the best

system in the set is at least δ greater than the true mean of the second-best system. The IZ

parameter δ > 0 is set by the experimenter to the minimum difference in expected

performance that it is important to detect. PCS and IZ parameters are user specified. To

5
understand the SaG better, let us consider a generic system generating algorithm (SGA)

below (Hong & Nelson, 2007).

System Generating Algorithm (SGA)

Step 0. We start with k0 simulated systems, π1, π2, . . . , πk0 , and k0 ≥ 2. Let i and Ki denote

the iteration count and the total number of alternative systems generated through iteration

i, respectively. Set i = 0 and K0 = k0. Go to Step 2.

Step 1. Let i = i + 1. Generate ki ≥ 1 alternatives, πKi−1+1,πKi−1 +2, . . . , πKi−1+ki . Let Ki =

Ki−1 + ki.

Step 2. Select the best system in all Ki alternatives π1, π2, . . . , πKi .

Step 3. If the stopping rule is satisfied, stop; otherwise, go to Step 1

From the description above, the SaG should be employed in step 2 of the SGA to

select the best system. Detailed procedures of two different types of SaG can be found in

the section 2.3.1 and 2.3.2 (Hong & Nelson, 2007).

2.3.1 SaG with fixed first-stage sample size (SaG-F)

Fix the first-stage sample size n0

Input: If the iteration counter i = 0, then set I = {π1, π2, . . . , πk1} and K−1 = 0; otherwise,

set I = {π1, π2, . . . , πKi−1, πKi−1+1, πKi−1+2, . . . , πKi−1+ki}, and input the overall sample sizes,

overall sample means and first-stage sample variances of systems π1, π2, . . . , πKi−1

6
 ̂𝑖∗ and the overall sample sizes, overall sample means and first-
Return: The best system 𝜋

stage sample variances of systems π1, π2, . . . , πKi .

Procedure:

Setup: Select the PCS 1/2 < 1 −α < 1, IZ parameter δ > 0, and first-stage sample size n0

≥ 2. Let λ = δ/2.

Initialization: For all p = Ki−1 + 1,Ki−1 + 2 . . . ,Ki, take n0 observations from πp, calculate

the first stage sample mean ̅̅̅̅

𝑋𝑝(n0) and the first-stage sample variance 𝑆𝑝2 (n0), and set rp =

n0, where rp is the number of observations for πp. Let r = n0.

Computing parameters: Let βi = α/(Ki − 1). For any πp and πq in I and p ≠ q, calculate:

−2
𝜂𝑖(𝑛0 − 1) [ 𝑆𝑝2 (𝑛0 ) + 𝑆𝑞2 (𝑛0 )]
apq = , where ηi = (2𝛽𝑖)𝑛0−1 - 1
2𝛿

Elimination: Set Iold = I, and update I to be

I = {πp : πp ∈ Iold and r [𝑋𝑝

̅̅̅̅(rp) − ̅̅̅̅
𝑋𝑞 (rq )] ≤ max(0, apq − rλ), ∀πq ∈ Iold, p ≠ q }.

̂𝑖∗ be the system in I, and Return. Otherwise, for every

Stopping rule: If |I| = 1, then let 𝜋

πp ∈ I such that rp = r , take an observation of system πp, let rp = rp + 1, and update

̅̅̅̅
𝑋𝑝(rp). Set r = r + 1 and go back to Elimination.

7
2.3.2 SaG with variable first-stage sample size (SaG-V)

Input: Systems π1, π2, . . . , πKi and all sample information on systems π1, π2, . . . , πKi−1. If

iteration counter I equals zero, let K−1 = 0 and let the identities and all sample information

on previously generated systems be null.

Return: System 𝜋̂𝑖∗ and all the sample information on systems π1, π2, . . . , πKi .

Procedure:

Setup: Select the PCS 1/2 < 1 −α < 1, IZ parameter δ >0 and first-stage sample size

function n0(·). Let λ =δ/2.

Initialization: Take max{0, n0(Ki) − rp} observations from πp and set rp = rp + max{0,

n0(Ki) − rp} for all p = 1, 2, . . . ,Ki−1; also take n0(Ki) observations from πp and set rp =

n0(Ki) for all p = Ki−1 + 1, . . . ,Ki. Calculate the first-stage sample mean ̅̅̅̅
𝑋𝑝(n0(Ki)) and the

first-stage sample variance 𝑆𝑝2 (n0(Ki)) for all p = 1, 2, . . . ,Ki. Let r = n0(Ki) and I = {π1,

π2, . . . , πKi}.

Computing parameters: Same as the step in SaG-F, but using n0(Ki) instead of n0.

Elimination: Same as the step in SaG-F.

Stopping rule: Same as the step in SaG-F.

8
Chapter 3. Methodology

This chapter is divided into two parts. The first part describes the modifications we

make on the TSSO algorithm. Two algorithms will be presented in this part. The second

part is the evaluation of the algorithms by the use of test functions and scenarios to

compare the new algorithms with the current TSSO algorithm.

3.1 Modifications on TSSO

The first new TSSO algorithm is developed by replacing the OCBA in the

replication step with SaG-F, while the second new TSSO algorithm is created by replacing

the OCBA with SaG-V. Apart from these changes, the algorithms still use MNEK to build

the metamodel and MEI in the search stage.

This is how SaG works in the TSSO. After the search step, SaG procedure will be

employed on all the sampled points. The SaG selects the best point based on a user

specified probability of correct selection (PCS) and indifference zone (IZ) parameter. In a

minimization problem, the PCS is the probability that a system or a point that is selected is

the best system or the best point. The PCS is specified as any number between 0.5 to 1

(Hong & Nelson, 2007). The higher the PCS, the more replications in each iteration will

be made, thus the algorithm will explore less infill points, and vice versa. The IZ

parameter is often referred to as the smallest difference worth detecting. It is usually

specified as the difference between the means of the best and second best points, Δ,

however, due to the lack of knowledge about Δ, a more conservative IZ parameter may be

chosen, such as a positive number close to zero (Fan & Hong, 2014). The higher the IZ

9
parameter, the less replications in each iteration will be made, thus the algorithm will

explore more infill points, and vice versa.

The SaG procedure may run for several iterations until it finds the best point

among the sampled points. The SaG procedure will eliminate the points that are not

promising, leaving behind the more promising points. Promising points are mainly the

points that have low sample mean and points with high sample variance. The eliminated

points are the points that are clearly not the minimum points, in this case, mainly points

with high sample mean and low sample variance. One more replication will be performed

on each sample points that are not eliminated. The replications are required to further

evaluate the uneliminated points in order to select one sampled point as the global

optimum. After these replications are performed, the remaining points will be evaluated

again and the sampled points that are not promising will be eliminated. This cycle will go

on until only one point remains, this point is the current best point. The current best point

is the fmin in the next iteration of the MEI. In summary, these are the parameters and steps

for the two algorithms.

Parameter Definition

T Total number of replications at the start

n0 First-stage sample size, n0 ≥ 2 (For TSSO with SaG-F only)

n0(Ki) First-stage sample size function (For TSSO with SaG-V only)

A Size of initial space filling design. An0 ≤ T

i Current iteration. i = 0, 1, 2, . . . , I, I = (T − n0B)/B

10
Ki Total number of simulated points at iteration i

R Total number of replications used

xp pth simulated point

α 1-PCS (Probability of Correct Selection), 1/2 < 1 −α < 1

δ IZ (Indifference Zone), δ > 0

λ λ = δ/2

TSSO with SaG-F

Step 1: Initialization: Run a A size space filling design, with n0 replications allocated to

each point. Set R = R + An0

Step 2: Validation: Fit a MNEK response model to the set of sample means. Use leave-

one-out cross validation (LOOCV) to check if the initial MNEK model fit is satisfactory.

Step 3: Set i = 1,

While R < T

Step 3a: Sample a new point that maximizes the modified Expected Improvement

criterion with n0 replications. (Search Step)

MEI = E(max(fmin − 𝑓𝑃∗ (x), 0)

When i=1, fmin is the predicted response at the sampled point with the lowest mean. When

i> 1, fmin is determined by SaG procedure in Step 3b. 𝑓𝑃∗ (x) is a normal random variable

with mean given by the MNEK predictor at x and variance given by the predictor’s spatial

prediction uncertainty.

Set R = R + n0.

11
Step 3b: Use SaG-F. (Replication Step)

Initialization: Set I = {x1, x2, . . . , xKi}. For all p = Ki−1 + 1, Ki−1 + 2 . . . ,Ki, take n0

̅̅̅̅(n0) and the first-

observations from xp, calculate the first stage sample mean 𝑌𝑝

stage sample variance 𝑆𝑝2 (n0), and set rp = n0, where rp is the number of

observations for xp. Let r = n0.

Computing parameters: Let βi = α/(Ki − 1). For any xp and xq in I and p ≠ q,

calculate:
−2
𝜂𝑖(𝑛0 − 1) [ 𝑆𝑝2 (𝑛0 ) + 𝑆𝑞2 (𝑛0 )]
apq = , where ηi = (2𝛽𝑖)𝑛0−1 - 1
2𝛿

Elimination: While R<T

Set Iold = I, and update I to be I = {xp : xp ∈ Iold and r [𝑌𝑝

̅̅̅̅(rp) − 𝑌𝑞
̅̅̅̅ (rq )] ≤ max(0,

apq − rλ), ∀xq ∈ Iold, p ≠ q }.

Stopping rule: If |I| = 1, then let x̂𝑖∗ be the system in I, set fmin = 𝑓(x̂𝑖∗ ) and go to

Step 3c. Else, if R≥T, go to Step 3c. Otherwise, for every xp ∈ I such that rp = r ,

̅̅̅̅(rp). Set r = r + 1, R
take an observation of system xp, let rp = rp + 1, and update 𝑌𝑝

= R+1 and go back to Elimination.

Step 3c: Fit a MNEK response surface to the set of sample means. i = i + 1

The point of the global optimum at the end will be the best point selected by the SaG-F.

TSSO with SaG-V

Step 1: Initialization: Run a A size space filling design, with n0(Ki) replications allocated

to each point. Set R = R + An0(Ki)

12
Step 2: Validation: Fit a MNEK response model to the set of sample means. Use leave-

one-out cross validation (LOOCV) to check if the initial MNEK model fit is satisfactory.

Step 3: Set i = 1,

While R < T

Step 3a: Sample a new point that maximizes the modified Expected Improvement

criterion with n0(Ki) replications. (Search Step)

MEI = E(max(fmin − 𝑓𝑃∗ (x), 0)

When i=1, fmin is the predicted response at the sampled point with the lowest mean. When

i> 1, fmin is determined by SaG procedure in Step 3b. 𝑓𝑃∗ (x) is a normal random variable

with mean given by the MNEK predictor at x and variance given by the predictor’s spatial

prediction uncertainty.

Set R = R + n0(Ki)

Step 3b: Use SaG-V. (Replication Step)

Initialization: Take max{0, n0(Ki) − rp} observations from xp and set rp = rp +

max{0, n0(Ki) − rp} for all p = 1, 2, . . . ,Ki−1; also take n0(Ki) observations from xp

and set rp = n0(Ki) for all p = Ki−1 + 1, . . . ,Ki. Calculate the first-stage sample mean

̅̅̅̅(n0(Ki)) and the first-stage sample variance 𝑆𝑝2 (n0(Ki)) for all p = 1, 2, . . . ,Ki.
𝑌𝑝

Let r = n0(Ki) and I = {x1, x2, . . . , xKi}.

Computing parameters: Same as the step in TSSO with SaG-F, but using n0(Ki)

instead of n0

Elimination: Same as the step in TSSO with SaG-F

13
 Stopping rule: Same as the step in TSSO with SaG-F

Step 3c: Fit a MNEK response surface to the set of sample means. i = i + 1

The point of the global optimum at the end will be the best point selected by the SaG-V.

Selecting the n0(Ki) for TSSO with SaG-V is a complicated task. An ideal n0(Ki) is

one that minimizes the expected total number of observations needed to make the

selection decision (Hong & Nelson, 2007). However, this is an almost impossible task

because it requires knowing the true means and variances of all generated alternatives

before starting the experiment. Hong and Nelson (2007) proposes conditions for choosing

n0(Ki), which are, the n0(Ki) increases at the rate of log(Ki) and E[N(Ki)] increases at the

rate of Ki log(Ki).

While the OCBA implemented in the original TSSO maximizes the approximate

probability of correct selection (Chen et al., 2000), it has no statistical guarantee of correct

selection of best point at each iteration. The advantage of the two TSSO modifications is

that it will find the best point with a statistical guarantee of correct selection at each

iteration (Hong & Nelson, 2007). This aims to provide a more accurate fmin at each

iteration. If the selection of fmin is more accurate, it will improve the selection of new point

in the search step by the modified EI criterion.

Moreover, the SaG procedures will make replications in points that are potential

minimum, mainly points with low sample mean and high sample variance, ignoring points

that are clearly not potential minimum, which are mainly points with high sample mean

and low sample variance. In this respect, SaG procedures are similar to OCBA procedure

in the original TSSO (Quan et al., 2013). These replications in points that are potential

minimum will help in estimating the Kriging parameters which will help to fit the Kriging

model.
14
3.2 Evaluation of Algorithms

To evaluate the two algorithms, comparison between TSSO, TSSO with SaG-F,

and TSSO with SaG-V is made. To conduct the comparison, test functions and scenarios

are implemented.

3.2.1 Test Functions

The algorithms are applied to minimize three test functions namely, Rescaled

Branin, Six-hump camel-back, and Hartmann-3. The objective is to find the global

minima of these functions. Detailed descriptions of the functions can be found in Table 1.

For the candidate points, we take 2500 points for each function. Figure 3.1 and Figure 3.3

are the illustrations of Rescaled Branin and Six-hump camel-back functions respectively.

Figure 3.2 and Figure 3.4 are their candidate points and contour plots. (𝑥 ∗ , 𝑓(𝑥 ∗ )) are

denoted by an arrow.

Figure 3.1 Rescaled Branin Figure 3.2 Candidate Points and Contour Plot

15
 Figure 3.3 Six-Hump Camel-back Figure 3.4 Candidate Points and Contour Plot

As seen from the figures above, the global minimum of Branin lies on a large flat

valley while Six-hump camel-back’s global minimum lies within a small valley. Branin

has two local minima and one global minimum while Six-hump camel-back has six local

minima, one of which is the global minimum.

Table 3.1 Test Functions

Function Source Description

Rescaled Branin Dixon and dimension=2

Szegö (1978) 2
1 5.1𝑥̅ 12 5𝑥̅ 1
𝑓(𝑥1 , 𝑥2 ) = [(𝑥2 − + − 6) +
51.95 4𝜋 2 4𝜋

10
(10 − ) 𝑐𝑜𝑠(𝑥̅1 ) − 44.81], 𝑥̅1 = 15𝑥1 − 5,
8𝜋

𝑥̅2 = 15𝑥2 , 0 ≤ 𝑥1 ≤ 1 𝑎𝑛𝑑 0 ≤ 𝑥2 ≤ 1

𝑥 ∗ = (0.541, 0.1348), 𝑓(𝑥 ∗ ) = −1.0459

16
 Six-hump Dixon and dimension = 2

Camel-back Szegö (1978) 𝑥16

𝑓(𝑥1 , 𝑥2 ) = 4𝑥12 − 2.1𝑥14 + + 𝑥1 𝑥2 − 4𝑥22 −
3

4𝑥24 , −2 ≤ 𝑥1 ≤ 2 𝑎𝑛𝑑 − 1 ≤ 𝑥2 ≤ 1

𝑥 ∗ = (0.0977, −0.6973), 𝑓(𝑥 ∗ ) = −1.0294

Hartmann-3 Global dimension = 3

Optimization

Test Problems

(2013)

0 ≤ 𝑥1 ≤ 1, 0 ≤ 𝑥2 ≤ 1, 𝑎𝑛𝑑 0 ≤ 𝑥3 ≤ 1

𝑥 ∗ = (0.114614, 0.555649, 0.852547)

𝑓(𝑥 ∗ ) = −3.86278

3.2.2 Test Scenarios

For the initial design, we follow the suggestions of Jones et al. (1998): A =

10×dimension, where A is the number of initial points. Thus, the initial points for rescaled

Branin and Six-hump camel-back functions are 20 for each function, while there are 30

initial points for Hartmann-3 function.

We test our algorithms using low and high replication budgets; this refers to the

replication budget available after simulating the A initial points, for performing the search

and replication steps. For the low budget scenario, a total of 550 replications are available,

while for the high budget scenario, a total of 2750 replications are available (Jalali et al.,

17
2016). For TSSO the number of replications for every iteration is 55 for low budget and

110 for high budget. For TSSO with SaG-F and TSSO with SaG-V there is no fix number

of replications in each iteration due to the nature of SaG procedure that will keep making

replications until the best point is selected. For the TSSO with SaG-F and TSSO with

SaG-V we set the default PCS to 0.95 and IZ parameter to 0.1. Moreover, for light noise

we add another setting for IZ parameter to 0.05 to allow for more replications in each

iteration and for heavy noise we add another setting where the IZ parameter is 0.5 to

explore more infill points. For the replication stage of TSSO with SaG-V we set the first

𝐾
stage sample-size function to be: 𝑛0 (𝐾𝑖 ) = 𝑛0 log ( 2𝑖 ) /log(5), where n0 is the initial first

stage sample size and Ki is the number of sampled points at iteration i. Moreover, we have

two noise scenarios namely, heavy and light noise scenario.

We have 4 settings for evaluating the algorithms, namely:

1. Light Noise – Low Budget

2. Heavy Noise – Low Budget

3. Light Noise – High Budget

4. Heavy Noise – High Budget

100 macroreplications are taken for the rescaled Branin and Six-hump camel-back

functions, while 50 macroreplications are taken for the Hartmann-3 function.

3.2.3 Performance Measures

We use three performance measures to compare the algorithms in each scenario.

Let 𝑦 be the final best function value identified by the algorithm and 𝑦 ∗ be the true

18
minimum. Let 𝑥(𝑘) be the best solution found by the algorithm in the 𝑘th dimension and

𝑥(𝑘)∗ be the true best value in the 𝑘th dimension.

The first performance measure is 𝑦 − 𝑦 ∗ which is the difference between the final

best function value identified by the algorithm in one and the true minimum in one

macroreplication. This value can be either positive or negative. The closer to zero the

value, the better the performance of the algorithm.

The second performance measure is |𝑥 − 𝑥 ∗ |, it is the euclidean distance between

the estimated and the true optimum location in one macroreplication. The lower the value,

the better the performance of the algorithm.

We would then compare the mean and standard deviation of these performance

measures from all macroreplications for every scenario in every function. T-Test at 5%

significance level is used to compare the performance measures.

Lastly, the third performance measure is the average time (in seconds) needed for

one microreplication of every algorithm for the three functions and four settings, this is

denoted by t. The shorter the time needed the better.

19
Chapter 4. Results

The simulation results for the 3 functions and 4 settings of TSSO, TSSO with SaG-

F, and TSSO with SaG-V are shown below. The mean and standard deviations for the first

two performance measures of TSSO with SaG-F and TSSO with SaG-V are colored.

Green means it performs better than TSSO at 5% significance level, red means it performs

worse than TSSO at 5% significance level, while yellow means the T-test is inconclusive

at 5% significance level.

4.1 Rescaled Branin Results

Table 4.1 Results for Rescaled Branin Light Noise-Low Budget

Rescaled Branin, Light Noise – Low Budget

Algorithm Mean Standard Mean Standard t (seconds)

𝑦 − 𝑦∗ Deviation |𝑥 − 𝑥 ∗ | Deviation
𝑦 − 𝑦∗ |𝑥 − 𝑥 ∗ |
TSSO 0.029871508 0.032623538 0.463875493 0.276185281 22.02749

TSSO with 9.80993

0.035952 0.031318 0.514617 0.285592
SaG-F
TSSO with 5.07662
SaG-F IZ =
0.062123 0.059935 0.299331 0.294161
0.05
TSSO with 26.85143
SaG-V
0.034083 0.026142 0.465357 0.301464
TSSO with 8.03892
SaG-V IZ =
0.033928 0.028606 0.630254 0.22291
0.05

20
 Table 4.2 Results for Rescaled Branin Heavy Noise-Low Budget

Rescaled Branin, Heavy Noise – Low Budget

Algorithm Mean Standard Mean Standard t (seconds)

𝑦 − 𝑦∗ Deviation |𝑥 − 𝑥 ∗ | Deviation
𝑦 − 𝑦∗ |𝑥 − 𝑥 ∗ |
TSSO 19.18549
0.473992 0.378957 0.419875 0.226265
TSSO with 4.20208
SaG-F
0.380026 0.314693 0.348422 0.191597
TSSO with 4.31187
SaG-F IZ =
0.5 0.329415 0.306289 0.3407 0.256322
TSSO with 5.18697
SaG-V
0.48361 0.34427 0.423819 0.260628
TSSO with 6.89035
SaG-V IZ =
0.5 0.279939 0.208818 0.421879 0.258487

Table 4.3 Results for Rescaled Branin Light Noise-High Budget

Rescaled Branin, Light Noise – High Budget

Algorithm Mean Standard Mean Standard t (seconds)

𝑦 − 𝑦∗ Deviation |𝑥 − 𝑥 ∗ | Deviation
𝑦 − 𝑦∗ |𝑥 − 𝑥 ∗ |
TSSO 43.23625
0.020408 0.021263 0.503362 0.285058
TSSO with 23.53405
SaG-F 0.013823 0.01183 0.5589 0.289774
TSSO with 9.62888
SaG-F IZ =
0.05 0.016457 0.015517 0.557067 0.320964
TSSO with 140.2018
SaG-V
0.021911 0.021731 0.500396 0.292203
TSSO with 24.75005
SaG-V
IZ=0.05 0.015911 0.012869 0.480805 0.265295

21
 Table 4.4 Results for Rescaled Branin Heavy Noise-High Budget

Rescaled Branin, Heavy Noise – High Budget

Algorithm Mean Standard Mean Standard t (seconds)

𝑦 − 𝑦∗ Deviation |𝑥 − 𝑥 ∗ | Deviation
𝑦 − 𝑦∗ |𝑥 − 𝑥 ∗ |
TSSO 40.5174
0.176548 0.160892 0.443923 0.259284
TSSO with 5.2031
SaG-F 0.216497 0.201835 0.444635 0.315428
TSSO with 9.50243
SaG-F IZ =
0.5 0.130948 0.116097 0.353045 0.173431
TSSO with 5.73658
SaG-V
0.177298 0.139059 0.381434 0.232328
TSSO with 13.64396
SaG-V IZ =
0.5 0.150892 0.116551 0.416998 0.278099

4.2 Six-hump Camel-back Results

Table 4.5 Results for Six-hump Camel-back Light Noise-Low Budget

Six-hump Camel-back, Light Noise – Low Budget

Algorithm Mean Standard Mean Standard t (seconds)

𝑦 − 𝑦∗ Deviation |𝑥 − 𝑥 ∗ | Deviation
𝑦 − 𝑦∗ |𝑥 − 𝑥 ∗ |
TSSO 20.27435
0.027479 0.024088 1.15503 0.519097
TSSO with 15.23388
SaG-F
0.027065 0.021309 0.459568 0.63671
TSSO with 6.01628
SaG-F IZ =
0.05 0.0377 0.028657 1.209822 0.470694
TSSO with 33.78761
SaG-V
0.041569 0.027698 0.76053 0.663676
TSSO with 13.8943
SaG-V IZ =
0.5 0.044598 0.034998 0.604426 0.661695

22
 Table 4.6 Results for Six-hump Camel-back Heavy Noise-Low Budget

Six-hump Camel-back, Heavy Noise – Low Budget

Algorithm Mean Standard Mean Standard t (seconds)

𝑦 − 𝑦∗ Deviation |𝑥 − 𝑥 ∗ | Deviation
𝑦 − 𝑦∗ |𝑥 − 𝑥 ∗ |
TSSO 21.54566
0.436514 0.285189 0.902949 0.621394
TSSO with 4.76506
SaG-F
0.581527 0.306304 1.3835 0.350745
TSSO with 4.40737
SaG-F IZ=0.5 0.352999 0.268447 1.077082 0.568184
TSSO with 4.67181
SaG-V
0.687888 0.271837 1.143779 0.755538
TSSO with 4.67181
SaG-V
IZ=0.5 0.687888 0.271837 1.143779 0.755538

Table 4.7 Results for Six-hump Camel-back Light Noise-Low Budget

Six-hump Camel-back, Light Noise – High Budget

Algorithm Mean Standard Mean Standard t (seconds)

𝑦 − 𝑦∗ Deviation |𝑥 − 𝑥 ∗ | Deviation
𝑦 − 𝑦∗ |𝑥 − 𝑥 ∗ |
TSSO 45.452

0.015149 0.010978 0.401442 0.604431

TSSO with 42.07138
SaG-F 0.012332 0.00985 0.578973 0.676254
TSSO with 20.1503
SaG-F IZ =
0.05 0.015607 0.01277 0.847443 0.675444
TSSO with 194.7581
SaG-V
0.015138 0.012562 0.89887 0.659636
TSSO with 48.26607
SaG-V IZ =
0.05 0.017041 0.013368 0.593098 0.682459

23
 Table 4.8 Results for Six-hump Camel-back Light Noise-Low Budget

Six-hump Camel-back, Heavy Noise – High Budget

Algorithm Mean Standard Mean Standard t (seconds)

𝑦 − 𝑦∗ Deviation |𝑥 − 𝑥 ∗ | Deviation
𝑦 − 𝑦∗ |𝑥 − 𝑥 ∗ |
TSSO 47.20844
0.199553 0.160178 0.480816 0.5856
TSSO with 4.50997
SaG-F 0.182935 0.125849 1.382489 0.14976
TSSO with 7.86931
SaG-F IZ=0.5 0.320142 0.219588 1.036963 0.674474
TSSO with 5.6886
SaG-V
0.207611 0.200104 1.284865 0.476313
TSSO with 16.69251
SaG-V
IZ=0.5 0.246572 0.180828 1.110464 0.476548

4.3 Hartmann-3 Results

Table 4.9 Results for Hartmann-3 Light Noise-Low Budget

Hartmann-3, Light Noise – Low Budget

Algorithm Mean Standard Mean Standard t (seconds)

𝑦 − 𝑦∗ Deviation |𝑥 − 𝑥 ∗ | Deviation
𝑦 − 𝑦∗ |𝑥 − 𝑥 ∗ |
TSSO 20.79484
0.253232 0.148989 0.210673 0.186672
TSSO with 6.58918
SaG-F
0.221703 0.123432 0.156185 0.099214
TSSO with 5.41134
SaG-F
IZ=0.05 0.199122 0.135061 0.280173 0.081089
TSSO with 8.70702
SaG-V
0.330114 0.133652 0.166488 0.169095
TSSO with 6.13874
SaG-V
IZ=0.05 0.45797 0.248091 0.201653 0.143335

24
 Table 4.10 Results for Hartmann-3 Heavy Noise-Low Budget

Hartmann-3, Heavy Noise – Low Budget

Algorithm Mean Standard Mean Standard t (seconds)

𝑦 − 𝑦∗ Deviation |𝑥 − 𝑥 ∗ | Deviation
𝑦 − 𝑦∗ |𝑥 − 𝑥 ∗ |
TSSO 21.52234
3.385748 0.461661 0.696703 0.272894
TSSO with 6.3917
SaG-F
3.322011 0.254258 0.589682 0.190353
TSSO with 10.38902
SaG-F IZ=0.5 2.915216 0.70065 0.566529 0.224272
TSSO with 10.40596
SaG-V
3.373676 0.382183 0.517918 0.20927
TSSO with 8.16954
SaG-V
IZ=0.5 3.39224 0.401279 0.518612 0.202608

Table 4.11 Results for Hartmann-3 Light Noise-High Budget

Hartmann-3, Light Noise – High Budget

Algorithm Mean Standard Mean Standard t (seconds)

𝑦 − 𝑦∗ Deviation |𝑥 − 𝑥 ∗ | Deviation
𝑦 − 𝑦∗ |𝑥 − 𝑥 ∗ |
TSSO 48.13584
0.071153 0.052965 0.123394 0.086985
TSSO with 16.49988
SaG-F 0.076523 0.054254 0.049963 0.064307
TSSO with 11.04032
SaG-F
IZ=0.05 0.139605 0.106512 0.112425 0.065459
TSSO with 33.49008
SaG-V
0.075905 0.049564 0.155469 0.150974
TSSO with 12.87658
SaG-V
IZ=0.05 0.087511 0.067308 0.12697 0.069352

25
 Table 4.12 Results for Hartmann-3 Heavy Noise-High Budget

Hartmann-3, Heavy Noise – High Budget

Algorithm Mean Standard Mean Standard t (seconds)

𝑦 − 𝑦∗ Deviation |𝑥 − 𝑥 ∗ | Deviation
𝑦 − 𝑦∗ |𝑥 − 𝑥 ∗ |
TSSO 55.44196
1.50847 0.640686 0.377425 0.205539
TSSO with 10.18902
SaG-F 2.130533 0.411467 0.347031 0.161878
TSSO with 8.36646
SaG-F IZ=0.5 1.134588 0.545795 0.306274 0.209976
TSSO with 9.31818
SaG-V
2.126811 0.387538 0.373021 0.172016
TSSO with 9.59844
SaG-V
IZ=0.5 1.091292 0.447778 0.31679 0.195512

26
Chapter 5. Discussion

This chapter is divided into four sections. The first to third section discusses the

performances of the algorithms in every test function. The last section discusses the

performances of the algorithms across functions. In this discussion, the term ‘better’ that

we are going to use to compare the algorithms means that either the algorithm’s 𝑦 − 𝑦 ∗ or

|𝑥 − 𝑥 ∗ | performs better than TSSO at 5% significance level while the other performance

measure (excluding time) is either better or there is no significant difference at 5%

significance level.

5.1 Performances in Rescaled Branin

Except for the Light Noise and Low-Budget, there is at least one setting of TSSO

with SaG-F (either IZ=0.1, IZ = 0.5 or IZ = 0.05) which performs better than TSSO.

Whereas for the TSSO with SaG-V, there is mostly no significant difference from the

TSSO except for the Heavy Noise-Low Budget setting. Moreover, the time taken by

TSSO with SaG-F is generally the shortest among the three algorithms. From this we may

conclude that TSSO with SaG-F looks most promising as improvements to the TSSO in

the rescaled Branin function

5.2 Performances in Six-hump Camel-back

TSSO with SaG-F performs better than TSSO in the light noise cases, although for

some IZ parameters, TSSO with SaG-F performs worse than TSSO. TSSO outperforms

both TSSO with SaG-F and TSSO with SaG-V in the heavy noise cases, while also

performing better than TSSO with SaG-V in the light noise cases. Similar to the

27
performance in rescaled Branin function, TSSO with SaG-F generally performs the fastest

in Six-hump camel-back function. From this we may conclude that TSSO with SaG-V

performs the worst in the Six-hump camel-back, while TSSO with SaG-F is promising for

the light noise cases.

5.3 Performances in Hartmann-3

Except for the Light Noise-Low Budget scenario, TSSO with SaG-F has at least

one setting in each scenario that outperforms TSSO. For the heavy noise case, TSSO with

SaG-F clearly outperforms TSSO in both 𝑦 − 𝑦 ∗ and |𝑥 − 𝑥 ∗ |. SaG-V has at least one

setting that performs better than TSSO in the heavy noise case. The time taken for TSSO

with SaG-F is also generally the shortest. In conclusion, TSSO with SaG-F looks

promising as an improvement of TSSO in this function except for the Light Noise-Low

Budget.

5.4 Performances Across Functions

TSSO with SaG-F performs better than TSSO in the heavy noise cases in rescaled

Branin and Hartmann-3. A possible explanation is that TSSO with SaG-F selected better

infill points in every iteration. The fmin that is predicted by the SaG-F has a statistical

guarantee of correct selection, this helps the modified EI criterion to select a better point.

Moreover, for each iteration, there is generally more replication budget used, so this helps

to fit a more accurate MNEK model at each iteration, especially with heavy noise where

more replications are needed, resulting in better selection of infill points. This comes at

the expense of generally fewer total infill points in TSSO with SaG-F. However, in these

28
cases, selecting better infill points results in a better performance than exploring more

infill points.

TSSO with SaG-F performs worse than TSSO in the six-hump camel-back

function for heavy noise cases. This may be due to the complexity of six-hump camel-

back function which has more local minima (six local minima) than Branin and

Hartmann-3 (2 and 4 respectively). TSSO with SaG-F which does not have a fixed

replication budget probably gets trapped in the local minima more easily than TSSO

which has a fixed replication budget in each iteration, thus many replications performed

by TSSO with SaG-F may have been wasted in the local minima, especially in the

presence of heavy noise. Moreover, TSSO with SaG-F generally has fewer infill points

than TSSO. In a function where there are many local minima, there is a need to do more

exploration of infill points to prevent the algorithm from getting trapped in one of the

local minima.

TSSO with SaG-V seems not very promising in all functions, except for the heavy

noise cases in Hartmann-3 and heavy noise-low budget in rescaled Branin. A possible

explanation for the better performances in some of the cases is similar to the explanation

of TSSO with SaG-F, which is the TSSO with SaG-V selects better infill points than

TSSO.

However, the generally not very promising performances of TSSO with SaG-V

may be caused by the unnecessary replications in points that are not promising. As seen in

section 3.1, additional replications may be added to unpromising points when the first

stage sample size, n0(Ki), increases. These may cause computation budget to be wasted on

points that are not promising, while these budgets may be better used for performing

29
replications in more promising points or in the search step. Thus, leading to suboptimal

uses of the budget, which then results in poorer performances.

TSSO with SaG-V is not as promising as TSSO with SaG-F across all functions,

because, in general, where TSSO with SaG-V performs better than TSSO, TSSO with

SaG-F also performs better. Moreover, TSSO with SaG-V performs better in less number

of scenarios than TSSO with SaG-F.

30
Chapter 6. Conclusion and Recommendation

In conclusion, this project has explored the possibility of modifying the TSSO

algorithm with ranking and selection procedures, namely SaG-F and SaG-V. Two

algorithms were developed in this project. The first one is TSSO with SaG-F which

replaces the OCBA in the replication step of TSSO with SaG-F. The second one is TSSO

with SaG-V which replaces the OCBA in the replication step of TSSO with SaG-V. These

two algorithms were tested using three functions with four settings, two possible

modifications to the parameters of TSSO with SaG-F and TSSO with SaG-V were also

explored.

The SaG-F looks promising as a possible improvement to the TSSO, while the

SaG-V procedure may not be very promising. TSSO with SaG-F performs better than

TSSO in the heavy noise cases in rescaled Branin and Hartmann-3, while performing

comparatively well in the light noise cases. However, TSSO with SaG-F performs worse

than TSSO in the heavy noise cases in the six-hump camel-back function, while

performing comparatively well in the light noise cases.

TSSO with SaG-V seems not very promising in all functions, except for the heavy

noise cases in Hartmann-3 and heavy noise-low budget case in rescaled Branin. In general,

where TSSO with SaG-V performs better than TSSO, TSSO with SaG-F also performs

better than TSSO. Moreover, TSSO with SaG-V performs better in less number of

scenarios than TSSO with SaG-F.

Possible limitations to these algorithms lies in the implementation of SaG-F and

SaG-V. The implementation of these two procedures needs to be done with careful

31
attention to the PCS and IZ parameter which currently must be specified by the user.

Future research may explore a better way to determine the PCS and IZ parameter used in

the TSSO with SaG-F or TSSO with SaG-V.

Moreover, the first stage sample size function of SaG-V is also a possible future

research, since the one that is used in the TSSO with SaG-V algorithm is fixed. A better

way of selecting the first stage sample size function of SaG-V may be explored.

Lastly, future research on TSSO with SaG-F and TSSO with SaG-V may explore

the possibility of fixing the budget for each replication step. The current algorithms do not

fix the budget for the replication step and perform worse than TSSO in function with

many local minima such as the six-hump camel-back function with heavy noise. Fixing

the budget for the replication step may allow the algorithm to avoid wasting too many

replications in one of the local minima. Thus, allowing it to explore more infill points

which may lead to more accurate results. The tradeoff between exploration of new points

and exploitation of sampled points needs to be taken into consideration when exploring

this possibility.

32
References

Ankenman, B., Nelson, B. L., and Staum, J. (2010). Stochastic kriging for simulation

metamodeling. Operations Research, 58:371–382.

Barton, R. R., & Meckesheimer, M. (2006). Metamodel-based simulation

optimization. Handbooks in operations research and management science, 13, 535-574.

Chen, C. H., Lin, J., Yücesan, E., & Chick, S. E. (2000). Simulation budget allocation for

further enhancing the efficiency of ordinal optimization. Discrete Event Dynamic

Systems, 10(3), 251-270.

Cressie, N. (1993). Statistics for spatial data. John Wiley & Sons, New York.

Dixon, L. C. W. and Szegö, G. P. (1978). Towards global optimisation 2. North-Holland

Amsterdam, The Netherlands.

Fan, W., & Hong, L. J. (2014, December). A frequentist selection-of-the-best procedure

without indifference zone. In Simulation Conference (WSC), 2014 Winter (pp. 3737-

3748). IEEE.

Frazier, P., Powell, W., and Dayanik, S. (2009). The knowledge-gradient policy for

correlated normal beliefs. INFORMS Journal on Computing, 21(4):599–613.

Hong, L. J., & Nelson, B. L. (2007). Selecting the best system when systems are revealed

sequentially. IIE transactions, 39(7), 723-734.

Huang, D., Allen, T. T., Notz, W. I., and Zeng, N. (2006). Global optimization of

stochastic black-box systems via sequential kriging meta-models. Journal of Global

Optimization, 34:441–466.

Jalali, H., Van Nieuwenhuyse, I., & Picheny, V. (2016). Comparison of Kriging-based

methods for simulation optimization with heterogeneous noise.

33
Jones, D. R., Schonlau, M., and Welch, W. J. (1998). Efficient global optimization of

expensive black-box functions. Journal of Global Optimization, 13:455–492.

Liu, C., Pedrielli, G., & Ng, S. H. (2014). eTSSO: Adaptive Search Method for Stochastic

Global Optimization Under Finite Budget. Working paper.

Picheny, V., Ginsbourger, D., Richet, Y., and Caplin, G. (2013a). Quantile-based

optimization of noisy computer experiments with tunable precision. Technometrics, 55:2–

13.

Picheny, V., Wagner, T., and Ginsbourger, D. (2013b). A benchmark of kriging-based

infill criteria for noisy optimization. Structural and Multidisciplinary Optimization,

48:607–626.

Quan, N., Yin, J., Ng, S. H., & Lee, L. H. (2013). Simulation optimization via kriging: a

sequential search using expected improvement with computing budget constraints. Iie

Transactions, 45(7), 763-780.

Staum, J. (2009, December). Better simulation metamodeling: The why, what, and how of

stochastic kriging. In Proceedings of the 2009 Winter Simulation Conference (WSC) (pp.

119-133). IEEE.

Yin, J., Ng, S. H., & Ng, K. M. (2011). Kriging metamodel with modified nugget-effect:

The heteroscedastic variance case. Computers & Industrial Engineering, 61(3), 760-777.

34
Appendices

Appendix A. Matlab Codes

Appendix A.1 Matlab Code for SaG-F

%initialize SAG
PCS=0.95; % Probability of Correct Selection: 1/2<PCS<1
IZ=0.5; % Indifference Zone Parameter
SAG_n0=B_n0; % 1st stage sample size for SAG
lambda=IZ/2;
y_SAG = y;
v_SAG = v;
rep_n0 = SAG_n0;
beta_i = zeros(MAXIMUM,1);
n_i = zeros(MAXIMUM, 1);
if(i==1)
curr_best = inf;
for j = 1:counter
if(y(j) < curr_best)
curr_best = y(j);
end
end
end
[EI_values Varextrinsic] =
EIcalc_MNEK_kd(x_axis,x(1:counter,:),model,curr_best);
[maxEIk index] = max(EI_values);

%%compute the terms you need for the new budget allocation
rule

[BESTY index1] = min(y(1:counter));

[WORSTY index2] = max(y(1:counter));
v_BARint = v(index1);
maxV_barInt=max(v(1:counter));
v_BARext = Varextrinsic(index).^2;
if (stopCritMET*stopCriterionINUSE==0 && usedBudget(kk)+ SAG_n0
< T)
%% search stage
x_new = x_axis(index,:);
T_X =
normrnd(fhandle(x_new(1,:)),noise_f(x_new(1,:)),SAG_n0,1);
y_new = mean(T_X);
v_new = var(T_X)/SAG_n0;
counter = counter + 1;
x(counter,:) = x_new;
y(counter) = y_new;

35
 v(counter) = v_new;
rep_cur(counter) = SAG_n0;
for k=1:SAG_n0
observations(counter,k)=T_X(k,1);
end
usedBudget(kk) = usedBudget(kk) + SAG_n0;
end

if(stopCritMET*stopCriterionINUSE==0)
beta_i(i) = (1-PCS)/(counter-1); % beta at iteration i
n_i(i) = (2*beta_i(i))^(-2/(SAG_n0-1))-1; % n at
iteration i
a_pq = zeros(counter,counter); % initialize a_pq
observations_SAG = observations(1:counter,1:SAG_n0); %
get observations to compute 1st stage sample stage variance
y_SAG = y;
rep_n0 = SAG_n0;

% compute 1st stage sample stage variance

for p=1:counter
v_SAG(p)=var(observations_SAG(p,1:SAG_n0))/SAG_n0;
end

% compute a_pq matrix

for p=1:counter
for q=1:counter
if(p~=q)
a_pq(p,q)= (n_i(i)*(SAG_n0-
1))*(v_SAG(p)+v_SAG(q))/(2*IZ);
end
end
end
%perform elimination
I_old = y_SAG;
I_new = zeros(MAXIMUM,1);
SAG_counter=counter;
I_counter = 0;
add_repSAG = zeros(MAXIMUM,1);
indexSAG = zeros(MAXIMUM,1);

for p=1:SAG_counter
pass = 0;
for q=1:SAG_counter
if p~=q
LHS = rep_n0*(y(p)-y(q));
Zero = 0;
a_pq_minus_rlambda = a_pq(p,q)-rep_n0*lambda;
RHS = max(Zero,a_pq_minus_rlambda);
if LHS<=RHS
pass = pass + 1;
end
end

36
 end
if pass == SAG_counter-1
I_counter = I_counter + 1;
I_new(I_counter) = I_old(p);
add_repSAG(p) = 1;
indexSAG(I_counter)=p;
end
end

while I_counter ~= 1
rep_n0 = rep_n0+1;
p = 1;
%add 1 replication for each uneliminated point
for j = 1:counter
if (add_repSAG(j)==1 && usedBudget(kk) < T)
T_D =
normrnd(fhandle(x(j,:)),noise_f(x(j,:)),1,1);
k=rep_cur(j)+1;
observations(j,k)=T_D(1,1);
y(j)=mean(observations(j,1:rep_cur(j)+1));
I_new(p)=y(j);
p=p+1;

v(j)=var(observations(j,1:rep_cur(j)+1))/(rep_cur(j)+1);
rep_cur(j) = rep_cur(j) + 1;
usedBudget(kk)= usedBudget(kk)+1;
end
end
if (usedBudget(kk)>= T)
break
end
%perform the elimination
I_old = I_new;
I_new = zeros(MAXIMUM,1);
SAG_counter = I_counter;
I_counter = 0;
indexSAG = zeros(MAXIMUM,1);
add_rep_new = zeros(MAXIMUM,1);

for p=1:counter
pass = 0;
if(add_repSAG(p)==1)
for q=1:counter
if(add_repSAG(q)==1)
if p~=q
LHS = rep_n0*(y(p)-y(q));
Zero = 0;
a_pq_minus_rlambda = a_pq(p,q)-
rep_n0*lambda;
RHS = max(Zero,a_pq_minus_rlambda);
if LHS<=RHS

37
 pass = pass + 1;
end
end
end
end
end
if pass == SAG_counter-1
I_counter = I_counter + 1;
I_new(I_counter) = y(p);
add_rep_new(p) = 1;
indexSAG(I_counter)=p;
else
add_rep_new(p) = 0;
end
end
add_repSAG = add_rep_new;
end

curr_best = I_new(1,1);

38
Appendix A.2 Matlab Code for SaG-V

%initialize SAG
PCS=0.95; % Probability of Correct Selection: 1/2<PCS<1
IZ=0.5; % Indifference Zone Parameter
SAG_n0=B_n0; % 1st stage sample size for SAG
lambda=IZ/2;
y_SAG = y;
v_SAG = v;
rep_n0 = zeros(MAXIMUM,1);
beta_i = zeros(MAXIMUM,1);
n_i = zeros(MAXIMUM, 1);
if(i==1)
curr_best = inf;
for j = 1:counter
if(y(j) < curr_best)
curr_best = y(j);
end
end
end
[EI_values Varextrinsic] =
EIcalc_MNEK_kd(x_axis,x(1:counter,:),model,curr_best);
[maxEIk index] = max(EI_values);

%%compute the terms you need for the new budget allocation
rule

[BESTY index1] = min(y(1:counter));

[WORSTY index2] = max(y(1:counter));
v_BARint = v(index1);
maxV_barInt=max(v(1:counter));
v_BARext = Varextrinsic(index).^2;
if (stopCritMET*stopCriterionINUSE==0 && usedBudget(kk)+ SAG_n0
<= T)
%% search stage
x_new = x_axis(index,:);
T_X =
normrnd(fhandle(x_new(1,:)),noise_f(x_new(1,:)),SAG_n0,1);
y_new = mean(T_X);
v_new = var(T_X)/SAG_n0;
counter = counter + 1;
x(counter,:) = x_new;
y(counter) = y_new;
v(counter) = v_new;
rep_cur(counter) = SAG_n0;
for k=1:SAG_n0
observations(counter,k)=T_X(k,1);
end
usedBudget(kk) = usedBudget(kk) + SAG_n0;
end

39
 if (usedBudget(kk) + SAG_n0 >=T)
T = usedBudget(kk);
end

if(stopCritMET*stopCriterionINUSE==0)
rep_n0(i) = floor(SAG_n0*log(counter/2)/log(5));
beta_i(i) = (1-PCS)/(counter-1); % beta at iteration i
n_i(i) = (2*beta_i(i))^(-2/(rep_n0(i)-1))-1; % n at
iteration i
a_pq = zeros(counter,counter); % initialize a_pq
for j=1:counter
add_rep = rep_n0(i)-rep_cur(j);
if (usedBudget(kk) + add_rep >=T)
T = usedBudget(kk);
break
end
if (add_rep >=1 && usedBudget(kk) + add_rep <= T)
T_D =
normrnd(fhandle(x(j,:)),noise_f(x(j,:)),add_rep,1);
for k=rep_cur(j)+1:rep_cur(j)+add_rep
observations(j,k)=T_D(k-rep_cur(j),1);
end

y(j)=mean(observations(j,1:rep_cur(j)+add_rep));

v(j)=var(observations(j,1:rep_cur(j)+add_rep))/(rep_cur(j)+add_rep
);
rep_cur(j) = rep_cur(j) + add_rep;
usedBudget(kk) = usedBudget(kk) + add_rep;
end
end
observations_SAG =
observations(1:counter,1:rep_n0(i)); % get observations to compute
1st stage sample stage variance
y_SAG = y;

% compute 1st stage sample stage variance

for p=1:counter

v_SAG(p)=var(observations_SAG(p,1:rep_n0(i)))/rep_n0(i);
end

% compute a_pq matrix

for p=1:counter
for q=1:counter
if(p~=q)
a_pq(p,q)= (n_i(i)*(rep_n0(i)-
1))*(v_SAG(p)+v_SAG(q))/(2*IZ);
end
end
end
%perform elimination

40
 I_old = y_SAG;
I_new = zeros(MAXIMUM,1);
SAG_counter=counter;
I_counter = 0;
add_repSAG = zeros(MAXIMUM,1);
indexSAG = zeros(MAXIMUM,1);

for p=1:SAG_counter
pass = 0;
for q=1:SAG_counter
if p~=q
LHS = rep_n0(i)*(y(p)-y(q));
Zero = 0;
a_pq_minus_rlambda = a_pq(p,q)-
rep_n0(i)*lambda;
RHS = max(Zero,a_pq_minus_rlambda);
if LHS<=RHS
pass = pass + 1;
end
end

end
if pass == SAG_counter-1
I_counter = I_counter + 1;
I_new(I_counter) = I_old(p);
add_repSAG(p) = 1;
indexSAG(I_counter)=p;
end
end

while I_counter ~= 1
rep_n0(i) = rep_n0(i)+1;
p = 1;
%add 1 replication for each uneliminated point
for j = 1:counter
if (add_repSAG(j)==1 && usedBudget(kk) + 1
<=T)
T_D =
normrnd(fhandle(x(j,:)),noise_f(x(j,:)),1,1);
k=rep_cur(j)+1;
observations(j,k)=T_D(1,1);
y(j)=mean(observations(j,1:rep_cur(j)+1));
I_new(p)=y(j);
p=p+1;

v(j)=var(observations(j,1:rep_cur(j)+1))/(rep_cur(j)+1);
rep_cur(j) = rep_cur(j) + 1;
usedBudget(kk)= usedBudget(kk)+1;
end
end

if (usedBudget(kk)>= T)

41
 break
end

%perform the elimination

I_old = I_new;
I_new = zeros(MAXIMUM,1);
SAG_counter = I_counter;
I_counter = 0;
indexSAG = zeros(MAXIMUM,1);
add_rep_new = zeros(MAXIMUM,1);

for p=1:counter
pass = 0;
if(add_repSAG(p)==1)
for q=1:counter
if(add_repSAG(q)==1)
if p~=q
LHS = rep_n0(i)*(y(p)-y(q));
Zero = 0;
a_pq_minus_rlambda = a_pq(p,q)-
rep_n0(i)*lambda;
RHS = max(Zero,a_pq_minus_rlambda);
if LHS<=RHS
pass = pass + 1;
end
end
end
end
end
if pass == SAG_counter-1
I_counter = I_counter + 1;
I_new(I_counter) = y(p);
add_rep_new(p) = 1;
indexSAG(I_counter)=p;
else
add_rep_new(p) = 0;
end
end
add_repSAG = add_rep_new;
end

curr_best = I_new(1,1);

42
Appendix B. Results

Appendix B.1 Rescaled Branin Results

43
Appendix B.2 Six-hump Camelback Results

44
Appendix B.3 Hartmann 3D Results

45