030717

PERKINS+WILL
RESEARCH PAPER

DESIGN OF EXPERIMENTS FOR MULTI-OBJECTIVE PARAMETRIC ENERGY

ANALYSIS

1. Abstract

Although the importance of multi-objective parametric energy analysis is recognized in research

literature, in practice it is difficult to implement. Some of the challenges are that it requires a large
number of runs to adequately explore a design space, it often relies on statistical methods that are
not easily accessible to architectural practitioners, the reporting mechanisms do not suit architectural
design exploration, among others. This paper explores a workflow for parametric energy analysis
that seeks to address the above challenges. The workflow combines design of experiment methods
with parametric energy analysis. It describes a research study undertaken at Perkins+Will using the
workflow.

2. Introduction

Comparisons between different approaches to building performance analysis show that parametric
energy analysis (PEA) leads to more optimal results than traditional building simulation. Traditional
simulation typically simulates a small number of design options and then uses human judgement to
infer the best cause of action. PEA, on the other hand, varies all relevant input factors through
significant ranges and thereby develops large design spaces for exploration. Research shows that this
can lead to dramatically better building performance optimization results.

Naboni et al. (2013) compared traditional simulation to PEA and optimization using a genetic
algorithm. They designed and built a project prototype of a 14m2 test lab made from cross laminated
timber and located at the School of Architecture in Copenhagen (Figure 1). Ten design options were
simulated in EnergyPlus during design, and after construction a calibrated model was analyzed. It
revealed a total energy consumption of 98.6 kWh/m2. A PEA was performed in EnergyPlus with 11
factors and 139,968 individual runs. It was executed on a 256 core cluster and took 30 hours to run.
Using the results modifications were identified that would bring the energy consumption down to
8.5 kWh/m2. Finally jEplus+EA, a genetic algorithm optimization tool for EnergyPlus simulations,
was used with 135 generations and 1350 simulations. It took 3 hours to run and revealed pareto
optimal results similar to the parametric energy analysis (Figure 2).

It is clear that despite the obvious benefit of PEA it has the major shortcoming of requiring long
durations to run, even on high powered computing resources that are unavailable to many
practitioners. Genetic algorithms and other statistical methods can help but they are often times too
obtuse for the architectural practitioners comfort. Moreover, architectural design is not engineering
design. Architects seek to explore ranges of options in the context of design geometry and other
subjective considerations and are not simply seeking the most optimal set of numbers. This requires
better interactive visualization methods than the reporting from typical PEA and statistical tools.
Finally, architectural design problems are invariably multi-objective seeking to optimize performance
factors across several response variables. Many PEA approaches are, however, restricted to single
objective optima.

1
 030717

Figure 1 (Naboni et al.)

In discussing multi-objective optimization it is important to distinguish between multi-disciplinary

optimizations and single discipline multi-objective optimization. Many authors referring to multi-
objective optimization are referring to the single discipline optimization of thermal performance
responses like annual heating load, annual cooling load and HVAC performance (Chlela et al. (2009);
(Magnier & Haghighat, 2010). A multi-disciplinary optimization gives a more holistic understanding
of the building performance factors in relation to overall human experience and requirements and is
as such more desirable. It requires a careful selection of input factors, a weighting of responses used
in the optimization value function and coordination of simulation results from different software
platforms. In this paper the multi-disciplinary study is comprised of daylight performance and
thermal performance.

The paper describes the use of design of experiments (DoE) in conjunction with PEA for the multi-
disciplinary, multi-objective optimization of building performance. The need for DoE arises from
the size of design spaces associated with rigorous multi-objective optimization, sometimes requiring
millions of simulations. The large size is caused by the need for several factors in order to express
the complex multi-objective problem as well as the granularity of the factors needed for the rigorous
exploration of the design spaces. This size of design space is for all practical purposes
computationally infeasible.

The DoE is, as such, seen as a filtering mechanism that allows for drastic reductions in the design
space to be explored, centered around estimated optimal values of the factor inputs. By combining
the DoE method with a full-factorial analysis of a reduced design space, the authors expect to
maintain the rigor of analytical search with the accuracy of parametric analysis using a practical
amount of computational resources.

2
 030717

Figure 2 (Naboni et al)

In order to distinguish this effort from other similar research, five components of PEA have been
identified as important for meeting the requirements of rigor, accuracy and practicality. First, the
PEA needs to be multi-disciplinary and multi-objective. Building performance analysis problems are
complex and multi-faceted and good solutions need to recognize the multi-dimensionality of the
problem. Second, not all responses are equally important in a multi-objective circumstance. The
PEA must prioritize what are considered more important design goals for a project over other less
important criteria. Third, the PEA must provide interactive visual feedback to the architect. The
output, particularly to an architectural audience, cannot simply be numbers and statistical metrics.
There needs to be a visual exploratory interface that permits the designer to interactively examine
ranges of alternatives in the proposed solution. Fourth, the PEA needs to be computationally
feasible which means it does not extend much beyond a few thousand simulation runs. Finally, the
PEA must retain a link to parametric design geometry. Architects must be able to see the impact of
building performance decisions on the physical geometry of the building design.

3
 030717

3. Literature Review
Table 1

Multi- Weighted Interactive Small number Parametric

Disciplinary Responses Visualization of runs Geometry
Dhariwal & Banerjee
Ritter et al.
Sadeghifam et al.
Qian & Lee
Pratt & Bosworth
Sadeghifam et al.
Jabi
Chlela et al.
Magnier & Haghighat
*Perkins+Will
(Dhariwal & Banerjee, 2017) observe that multi-objective optimization methods rely on exhaustive
search which can be computationally intensive. For example to iterate over 20 design variables each
with three parametric levels on an Intel Core i7-2670QM processor could take years. Sensitivity
analysis methods like Global Sensitivity Analysis and Fractional Factorial Designs can help reduce
the number of variables in the design space to those that are significant. Optimization methods are
also more effective with only significant variables present. Genetic Algorithms like NSGA-II are
popular for these optimizations.

They propose the use of surrogate models, specifically response surface models (a form of DoE),
that use a limited number of simulation runs. They note however that these methods require expert
knowledge and this is the main reason for their limited use. Their own method involves creating a
base model for simulation in EnergyPlus, applying Global Sensitivity Analysis to reduce the variable
space, using this reduced variable space for response surface modeling and then running an
optimization Genetic Algorithm on the surrogate model for final optimization. On a case study of a
three story office building in New Delhi they found that the surrogate model approach reduced
simulation time substantially at the cost of incurring up to 10% in prediction error. Although the
authors perform a multi-disciplinary analysis of life cycle cost and energy use intensity, they do not
weight their model and they do not provide interactive visualization for their result. Their study is
also not based on parametric geometric modeling.

Ritter et al. (2015) propose a design space exploration assistance method that utilizes available
information and applies parametric modeling and simulation techniques in the early design stage. It
includes response surface methods for optimizing the design space to support decision making.
They note that current methods have challenges such as relying on existing databases which does
not help with projects that have unique conditions, and the use of generative methods together with

4
 030717

optimization algorithms which can only return singular optimal solutions. These methods also lack
acceptable interfaces for designers to interact with.

They define design space construction as exploring all possible solutions to the design problem. This
can be time consuming if attempted manually. Parametric methods, while more automated, hinder
interaction between designer and the design. They propose generating a parametric geometric model
in Autodesk Dynamo, defining input parameters to describe the design space, creating a DoE in
Matlab to rapidly calculate the response values based on the input parameters and then output the
results into a parallel coordinates plot for interactive visualization. Their method, however, does not
perform a multi-disciplinary optimization as defined here and also does not weight the relative
importance of responses.

Sadeghifam et al. (2015) observe that altering a single factor, insulation, in hot humid climate did not
have a big effect on building performance. Altering a combination of factors (set point
temperatures, low-e double glazing, the use of VAV systems, adjustable lighting and energy efficient
lamps), on the other hand, combined for a 36% reduction in annual energy consumption. This
motivated their parametric energy analysis of factor combinations on a two story residential building
in Kuala Lumpur. Their process involved developing a Revit Architecture model, running a baseline
energy analysis based on the Revit model, identifying significant factors and their ranges and then
using a DoE to select building envelope materials in order to optimize thermal performance. Their
method was not a multi-disciplinary optimization, did not include interactive visualization or
parametric geometry.

Jabi (2014) suggests that the difficulty with conducting PEA at the early stages of design stems from
lack of appropriate representation formats and lack of design information. He seeks to better
harmonize the outputs of parametric geometric modeling with the input requirements of building
performance analysis. He develops DSOS as a software framework to do this. He describes how
DSOS seeks to integrate parametric geometry with performance analysis engines. It uses a non-
manifold geometric approach where surfaces and edges can be shared by more than two objects
allowing for their overlap in the definition of design spaces. The DSOS process uses Autodesk
Designscript, Open Studio and EnergyPlus scripts and files. It outputs results as color overlays on
parametric geometry. While Jabi performs parametric geometric modeling he does not perform
PEA. He also does not perform multi-disciplinary optimization nor weight his responses.

Qian and Lee (2014) sought to determine energy consumption in small commercial buildings,
studying building envelopes specifically, using mixed level factorial design (another form of DoE).
They used Trace 7000 for simulation and Minitab 1.7 for statistical analysis. They analyzed a small
commercial building at Morgan State University and found a potential saving of 16.6% total energy
consumption. Their study was not multi-objective or weighted, and it did not involve interactive
visualization or parametric geometry.

Pratt and Bosworth (2011)characterize energy efficient building design as wicked, after Rittel and
Webber. This is because the design space at the early stage is constrained, intermittently populated
and information poor. Parametric design solutions like Grasshopper can help alleviate this by
generating a wide range of variation within a defined parameter space. They propose combining
parametric methods with high throughput energy analysis methods.

5
 030717

They criticize restricted search based optimizations like genetic algorithms because larger datasets
allow for better understanding of variable dependencies, and the set of solutions which may satisfy
all design constraints may fall far from the quantitative optima. They develop sustainParametrics and
exportZones as Ruby plugins for Sketchup to create parametric models. They simulate these models
in EnergyPlus to produce building energy use metrics. Large simulation runs are performed on a 256
Intel cluster of cores. 34398 runs are simulated and an interactive visualization interface, including
parallel coordinate plots, is used to visualize the results. Their study, however, was not multi-
disciplinary, did not weight responses and used a large number of runs.

Chlela et al. acknowledge that parametric studies can help designers choose optimal solutions but
note that such studies can be complicated and time consuming due to a large number of runs. They
propose that DoE can simplify parametric studies by reducing significantly the required number of
experiments or simulations. They analyze a 540m2 three story prototype office building at three
locations in France using factor screening, DoE, metamodeling and testing. The DoE reduces a
177,147 run three factorial design to between 200 and 377 runs. The methods is not multi-
disciplinary or weighted and does not use interactive visualization or parametric geometry.

Magnier & Haghighat argue for the use of multi-objective optimization and propose the use of
evolutionary algorithms. They acknowledge that a shortcoming with genetic algorithms is the need
for thousands of evaluations to reach optimal solutions. They propose using response surface
methods with genetic algorithms to reduce the computational time while maintaining good accuracy.
They first use an ANN generated response surface to simulate the responses for the design space.
They then use a genetic algorithm to perform a search optimization but the responses are not
simulated they are obtained from the response surface. This reduces calculation time drastically.
The optimization took 7 minutes on an intel 1.66 GHz. However the initial training of the ANN
took 3 weeks. The authors estimate that it would have taken 10 years to perform a genetic algorithm
optimization without the response surface method.

It is seen that while many authors acknowledge the importance of PEA they are as well aware that it
is computationally demanding. Many seek to use DoE, in isolation or in combination with other
methods, to reduce the computational burden. This paper has similar motivations. By using DoE as
a filtering mechanism but still running a full factorial PEA of the reduced design space it avoids the
computational burden without necessarily sacrificing accuracy.

4. Background to the Design of Experiments

According to the US National Institute of Science and Technology (NIST) the design of
experiments is a systematic, rigorous approach to engineering problem-solving that uses statistical
methods to derive valid engineering conclusions under the constraint of minimal expenditure of
engineering resources (Croarkin & Tobias, 2017). There are four general problem areas in which
DoE is applied: comparative assessment of experimental outputs, screening for important factors,
modeling solutions and optimization of the problem space. DoE is widely applied to optimization
problems in several fields including building performance analysis.

DoE uses statistical methods to discover the optimum value from a large problem space of values
by methodically sampling the space and then interpolating between sampled values to obtain
estimates of non-sampled values. This estimated problem space can then be optimized, to a high
degree of accuracy, much faster than trying to experiment over the full problem space. The concept
6
 030717

can be illustrated by a simple example. In this example we use a least squares model for a two factor
experiment (Dunn, 2017). Suppose an experiment wishes to optimize an outcome O. Suppose that
there are two factors affecting this outcome A and B. A has two possible values A1 and A2 while B
has a range from 160 to 200, but can take on values in between. This is a two factor experiment with
two levels and a table of outcomes is shown (Table 2).

Table 2

Standard Order Run order A B Outcome (O)

1 2 -1 -1 52
2 4 +1 -1 74
3 1 -1 +1 62
4 3 +1 +1 80

From this sample of 4 experiments we can estimate any value of the outcome using the equation
= 67 + 10 + 4 . The equation is obtained by noting that the average of all outcomes is (52 +
74 + 62 + 80)/4 = 67, the average effect on factor A, going from 0 to +1, is (18 + 22)/4 = 10 and
the average effect on factor B, going from 0 to +1, is (6 + 10)/4 = 4. With this estimate we can use
software to quickly calculate an optimum value of the outcome, and the corresponding factor levels,
without performing any further experiments.

The general equation for this experiment, including interaction terms, can be written as

= 0 + + +

where xA is the coded value for the factor A, xB is the coded value for factor B, xAxB is the
interaction term and the bi are coefficients to be calculated. We can describe the experiment as
follows:

1 = 0 + + +
2 = 0 + + + + +
3 = 0 + + + + +
4 = 0 + + + + + + +

this reduces to the matrix equation:

1 1 1 1 +1 0
2 1 1
= 1 +1
3 1 1 +1 1
4 1 +1 +1 +1

or = . The solution is = ( ) , where is the inverse of the matrix and is

the transpose. This can be solved and optimized using statistical software.

The example can be extended to experiments with several factors and several levels, the statistical
calculations getting progressively more complex. In addition to least squares models there are also
other DoE methods like response surface models, half-factorial and full factorial models. In this
paper, however, it is acknowledged that architects are not statisticians. It is therefore recommended

7
 030717

that DoE be performed with tools that afford simplified user-interfaces, but with reasonable
defaults, that can still meet the goal of providing estimates for constructing a full factorial PEA
simulation. In the case study presented in the next section JMP (a version of the well-known SAS
statistical software) is introduced as such a tool.

5. Case Study - Sprout Space

Figure 3

Sprout Space (Figure 3) is a research initiative at Perkins+Will focused around developing a high
performance modular and portable classroom. The prototype is a 1000 sqft. pre-engineered and pre-
built design aimed in part at solving the problems of poor daylighting experienced in portable
classrooms. It was used for exploring the DoE based PEA workflow which involved the following
steps (Figure 4):

1. Develop design objectives

2. Establish factors and ideal ranges
3. Use DoE to filter the design space
4. Run a full factorial PEA on the reduced design space
5. Use a value function to optimize the responses
6. Use a parallel coordinates plot to visualize the results

The toolkit for this exploration included a custom Grasshopper definition for receiving inputs and
coordinating experimental runs with the simulation platforms; Rhinoceros for visualizing design
geometry; EnergyPlus for thermal simulation; Radiance for daylight simulation; JMP for DoE
analysis and a custom parallel coordinates plot interface for visualizing the high dimensional results
data of the experiment. The Grasshopper definition uses the HoneyBee plugin for thermal
simulation and the LadyBug plugin for daylight simulation (Figure 5). The steps of the workflow are
discussed.

8
 030717

Figure 4

Figure 5

9
 030717

1. Develop Design Objectives

Design objectives are developed by a team of stakeholders, designers and decision makers. They
include goals, indicators or responses, metrics and preferences or weights. In Sprout Space the goals
were identified as maximizing daylight while minimizing energy consumption. This make the
problem a multi-disciplinary (daylighting and thermal analysis) as well as multi-objective
optimization.
Table 3

GOAL INDICATOR METRIC WEIGHT

Maximize daylight Daylight Factor % 40%
Minimize energy consumption Energy Use kWh/m2/yr 60%

2. Establish factors and ideal ranges

Based on the defined objectives design variables relevant to the evaluation of these objectives are
identified. Ideal ranges for each of the variables are chosen together with the step-values needed for
a rigorous exploration of the design space. Some of these variables are geometric like the size of the
overhang while others may be performative. Some assumptions and constraints are involved in
choosing these factors and their ranges. Assumptions could be holding a factor constant while
constraints could be building geometry limits dictated by site restrictions. There are 43200
combinations of the factors which is computationally impractical on easily accessible resources.

Table 4

IDEAL NUMBER OF
FACTORS UNITS RANGE OPTIONS
Classroom Offset feet 4 - 20 5
Classroom Orientation degree 0 - 360 12
Front Window Width feet 10 - 30 5
Front Window Height feet 1-4 4
Roof Angle degree 1-9 9
Overhang Depth feet 0-3 4
43200

3. Use DoE to Filter the Design Space

A DoE is set up in JMP software to help reduce the size of the design space. The DoE is set up with
reasonable defaults to report estimated optimum values for each of the factors. A realistic reduced
design space will then be constructed centered around these estimates. The full factorial PEA will
then be run on this reduced space.

3.1
Input factors and step values into a DoE Custom Design in JMP. Define a response called Value
Function which will hold the sum of weighted response factors.

10
 030717

Figure 6

3.2
Define reasonable interaction and second order terms, and set the DoE to a reasonable number of
runs. Sprout Space used 32.

Figure 7

11
 030717

3.3
Use JMP to design the experiment. Note that the Value Function is not filled out yet. It will receive
the results from the 32 simulation runs performed in the Grasshopper definition.

Figure 8

3.4
Simulate the 32 DoE in Grasshopper and create a value function from the results to plug into JMP
(See Step 5 for Value Function discussion).

Figure 9

3.5
Run the DoE and obtain contour plots with the estimates of the optimum values of the factors.
These estimates will be used in constructing the full factorial design space.

12
 030717

Figure 10

4. Run a full factorial PEA on the reduced design space

Use the DoE results to create realistic ranges for the full factorial PEA with a computationally
feasible number of runs, 1152 in this case.

Table 5

NUMBER OF
FACTORS UNITS IDEAL RANGE TARGET RANGE OPTIONS
Classroom Offset feet 4 - 20 8,12,16,20 4
Classroom Orientation degree 0 - 360 180, 210, 240, 270, 300, 330 6
Front Window Width feet 10 - 30 15, 20, 25, 30 4
Front Window Height feet 1-4 3, 4 2
Roof Angle degree 1-9 1, 3, 6 3
Overhang Depth feet 0-3 2, 3 2
1152

5. Use a value function to optimize the responses

After generating the PEA results create a value function by standardizing the responses, weighting
them according to the preferences indicated when developing the objectives and inverting any
antagonistic objectives. For example, use the negative of the thermal performance value in the value
function because this has high value when the response value is low.

Table 6

RESPONSES STANDARDIZATION, WEIGHTING, INVERSION

Total
Thermal Cooling Heating Daylight Standardized - Standardized - Weighted Weighted Inverted Value
Energy Load Load Factor TTE DF - TTE -DF - TTE Function
6500 5000 850 2 -0.07 -0.46 -2.93 -13.88 2.93 5.95
5000 4000 950 1 -0.95 -0.92 -38.19 -27.77 38.19 -14.31

13
 030717

6. Use a parallel coordinates plot to visualize the results

Use the responses and value function in a parallel coordinates plot of the full factorial PEA (Figure
10). This will enable the architect to explore ranges of options and understand their impact on
individual responses as well as on the value function. The optimized outputs can also be visualized
in the context of the geometric model in Rhinoceros (Figure 11).

Figure 11

Figure 12

6. Conclusion

A workflow for using DoE with PEA has been described. The benefit of PEA for building
performance simulations was indicated and the limitation of a high computational burden
highlighted. A comparative literature review of sources that recognize the merits of DoE was

14
 030717

presented. However, limitations with many of the approaches was observed. The use of the DoE for
PEA workflow on the Sprout Space case study was also described. The key benefit of the method is
that it drastically reduces the time and resources needed for analysis. The method also allows for
multi-disciplinary, multi-objective optimization taking into account the weighting of responses in
constructing a value function. A tool, parallel coordinates plot, for visualizing high dimensional data
is prescribed. The workflow remains integrated with the parametric geometric model which can be
visualized in Rhinoceros software.

7. References

Chlela, F., Husaunndee, A., Inard, C., & Reiderer, P. (2009). A New Methodology for the Design of Low
Energy Buildings. Energy and Buildings, 41(9), 982-990.
Croarkin, C., & Tobias, P. (2017, March 7). Engineering Statistics Handbook. Retrieved from
NIST/SEMATECH e-Handbook of Statistical Methods:
http://www.itl.nist.gov/div898/handbook/index.htm
Dhariwal, J., & Banerjee, R. (2017). An approach for building design optimization using design of
experiments. Building Simulation, 10(3), 323-336.
Dunn, K. (2017, March 7). Process Improvement Using Data. Retrieved from https://learnche.org/pid/
Jabi, W. (2014). Parametric spatial models for energy analysis in the early design stages. Proceedings of the
Symposium on Simulation for Architecture & Urban Design (p. 16). Society for Computer Simulation
International.
Magnier, L., & Haghighat, F. (2010). Multiobjective optimization of building design using TRNSYS
simulations, genetic algorithms and Artificial Neural Network. Building and Environment, 45(3), 739-
746.
Naboni, E., Maccarini, A., Korolija, I., & Zhang, Y. (2013). Comparison of conventional, parametric and
evolutionary optimization approaches for the architectural design of nearly zero energy buildings.
Building Simulation.
Pratt, K., & Bosworth, D. (2011). A Method for the Design and Analysis of Parametric Building Energy
Models. 12th Conference of the International Building Performance Simulation Association, (pp. 2499 - 2506).
Sidney.
Qian, X., & Seong, L. (2014). The Design and Analysis of Energy Efficient Building Envelopes for the
Commercial Buildings by Mixed-level Factorial Design and Statistical Methods. Middle Atlantic Section
Proceedings. American Society for Engineering Education.
Ritter, F., Geyer, P., & Bormann, A. (2015). Simulation-based decision-making in Early Design Stages. 32nd
CIB W78 Conference, (pp. 27-29). Eindhoven.
Sadeghifam, A., Zahraee, S., Meynagh, M., & Kiani, I. (2015). Combined use of design of experiments and
dynamic building simulation in assessment of energy efficiency in tropical residential buildings. Energy
and Buildings, 86, 525-533.

15