VxRail System: Ensure Your

Cluster Is Firing on All Cylinders

with Under-the-Hood Performance
Tuning
Key takeaways

• The VxRail System architecture is designed to handle the performance and availability
requirements for the most demanding workload.

• VxRail System configuration options allow a system design that can satisfy the workload demands
and their SLOs at an optimal cost.

• Best practices provide predictable results.

2 © Copyright 2017 Dell Inc.

Agenda

• Basic vSAN architectural considerations

• Designing and sizing a VxRail System

• VxRail System performance and scalability

• Performance monitoring and optimization

• Q&A

3 © Copyright 2017 Dell Inc.

vSAN Architecture
Concepts
VMware vSAN
VM
• vSAN is a distributed object based storage system vCenter Server
• vSAN is a software defined storage (SDS) system
vSphere vSphere vSphere
for vSphere that sits on vSphere’s data path

• vSAN CPU-MEM overhead is estimated at 10% vSAN

vSAN distributed datastore
• vSAN’s datastore is an aggregation of disks from
multiple hosts
– The datastore is a logical construct
– It is a single storage pool shared by all VMs
– The data from a single VM can be distributed across
disks of multiple hosts …64
Hosts

• Management network
• vMotion network
• vSAN
• VM networks
5 © Copyright 2017 Dell Inc.
Structure of the vSAN datastore
vSAN • vSAN datastore consists of disk groups
datastore
• Disk group consists of one cache and from 1 to
VxRail 7 capacity disks
host
• Disk groups exhibit performance capabilities,
Disk groups 1..4 owns
capacity and endurance

• VxRail System hosts own from 1 – 4 disk

groups
1 Cache
Performance disk
capabilities Note
vSAN allows up to five DGs per host and seven capacity disks
1..6 Capacity
per DG, but each VxRail model has a different configuration
Capacity
disk maximum.

Endurance

6 © Copyright 2017 Dell Inc.

vSAN is a scale-out storage architecture
VM
• vSAN implements a scale-out storage architecture
– When capacity is added, it is immediately made vCenter Server
available to the entire cluster
vSphere vSphere vSphere
• The I/O performance of the vSAN cluster scales
almost linearly as new hosts are added to the vSAN vSAN
cluster (up to 64 nodes)
vSAN distributed datastore
• The I/O performance of an individual host also
DG DG DG DG DG 1..5 DG
scales as more disk groups are added to the host

…64
Hosts

Scales linearly

7 © Copyright 2017 Dell Inc.

Benefits of a scale-out architecture in the planning
phase
Scale-up
• Long-term forecasting
• Excess capacity bought to
satisfy future demand
• Larger capital acquisitions
8 © Copyright 2017 Dell Inc.
• Shorter-term forecasting
• Minimizes idle capacity
• Lower TCA
- It is simple and fast to adapt to new
workload demands after
implementation
Designing and Sizing
Considerations about Requirements and Component
Choices
Designing and sizing your VxRail system

• One of the core objectives of a sizing exercise

is to determine the pools of resources
needed to satisfy workload demands and
their SLOs, at an optimal cost
– What is the best VxRail configuration to satisfy
my needs?

• In this section, we will briefly discuss

– Determination of workload demands
– HW component options
– Considerations to ensure good system behavior
and VM performance

10 © Copyright 2017 Dell Inc.

Defining workload requirements

• In technology-refresh use cases, use data collected with tools such as Dell
DPACK
– https://dpack2.dell.com/register/loginpage

• When “real life” data is not available, use “reference workloads” to define the
requirements
• Login VSI provides reference loads for the sizing of virtual desktops
– Knowledge worker, power user, task user

• For general-purpose VMs, we recommend the use of reference workloads

available with the VxRail advanced sizing tool
– Reference loads are synthetic loads that attempt to represent real life loads such as
databases and online transactions or public benchmarks

• The VxRail advanced sizing tool, which is used by Dell EMC vArchitects,
provides reference loads
• Work together with your vArchitect on the sizing/modeling exercises

11 © Copyright 2017 Dell Inc.

VxRail 4.0 models
Purpose-built nodes for multiple use cases

G Series Nodes E Series Nodes V Series Nodes P Series Nodes S Series Nodes

General purpose Entry level VDI optimized Performance Storage dense

optimized

All-flash or hybrid All-flash or hybrid All-flash or hybrid All-flash or hybrid Hybrid only
Up to 19.2 TB/node Up to 30.7 TB/node Up to 46 TB/node Up to 46 TB/node Up to 48 TB/node

12 © Copyright 2017 Dell Inc.

CPU options from the Intel Broadwell family (Xeon E5 v4)
currently available for VxRail
GHz per GHz per Cache per Thermal
Processor model Cores Threads L3 Cache
• The Broadwell family provides a wide core processor thread design power

variety of processors E5-2603 v4 6 6 1.70 10.20 15MB 2.5 85W

E5-2609 v4 8 8 1.70 13.60 20MB 2.5 85W
– Varying in number of cores, clock speed and E5-2620 v4 8 16 2.10 16.80 20MB 1.25 85W
power,and L3 Cache sizes E5-2630 v4 10 20 2.20 22.00 25MB 1.25 85W
– Flexibility to satisfy the different CPU and E5-2637 v4 4 8 3.50 14.00 15MB 1.88 135W
E5-2640 v4 10 20 2.40 24.00 25MB 1.25 90W
memory needs
E5-2650 v4 12 24 2.20 26.40 30MB 1.25 105W
E5-2660 v4 14 28 2.00 28.00 35MB 1.25 105W
• Hybrid models can be configured with a E5-2667 v4 8 16 3.20 25.60 25MB 1.56 135W
single CPU socket E5-2680 v4 14 28 2.40 33.60 35MB 1.25 120W
E5-2683 v4 16 32 2.10 33.60 40MB 1.25 120W
E5-2687W v4 12 24 3.00 36.00 30MB 1.25 160W
• All-flash models require a dual socket E5-2690 v4 14 28 2.60 36.40 35MB 1.25 135W
E5-2695 v4 18 36 2.10 37.80 45MB 1.25 120W
• The E5-2637 and 2667 are good options E5-2697 v4 18 36 2.30 41.40 45MB 1.25 145W
for software/applications that demand a E5-2697A v4 16 32 2.60 41.60 40MB 1.25 145W
E5-2698 v4 20 40 2.20 44.00 50MB 1.25 135W
high CPU capacity but are licensed by E5-2699 v4 22 44 2.20 48.40 55MB 1.25 145W
number of cores (e.g. Oracle)
Table from 02/22/2017

13 © Copyright 2017 Dell Inc.

Considerations about CPU and over commitment
Our goal is to be as close
as possible to the target
• When sizing a VxRail cluster, we are
defining the capacity of a resource pool Accuracy
– vCPU to CPU ratios are very simple rules of Not accurate Accurate
thumb, and they vary from 4:1 to 10:1

Not Precise
– Use a more specific metric, such as GHz, to
define the CPU capacity pool
– A 10-20% variation in requirements can create a

Precision
significant difference after implementation Not-accurate and Accurate and
not-precise not-precise

Precise
Not-accurate and Accurate and
precise precise

14 © Copyright 2017 Dell Inc.

Memory options and DIMM placement

Models #of DIMM sizes Total allowed

DIMM (GB) MEM capacity
slots (GB)

G 16 16, 32, 64 512

E, P, S, V 24 16, 32, 64 1,536

• A wide variety of memory sizes are possible in • The way in which the DIMMs are populated on
the VxRail hosts the server board impacts performance

• Populating the memory channels equally allows

• The G series has a smaller number of slots and
CPU to leverage its multiple memory controllers
a smaller mem capacity limit
– CPU interleaves access between memory channels
• The other models can reach up to 1,536GB of • Populating all 4 memory channels per CPU will
capacity with 64GB DIMMs provide better performance
Frank Denneman’s blog is recommended for a deeper dive in
this topic
15 © Copyright 2017 Dell Inc.
Memory-sizing considerations

• We should be sure to allocate enough memory to hold the working set of the applications that will
run

• Collecting data from the existing environments is key for this determination

• The use of the host memory consumption as opposed to VM consumption is the

conservative and recommended approach

16 © Copyright 2017 Dell Inc.

Disk drive options and sizing considerations
Latency
VxRail Drive Rotational Form
Tier Endurance at low Capacities
model type speed factor
I/O rates

Cache All flash SSD 10 DWPD(*) n/a <1ms 2.5” 400, 800GB

Hybrid SSD 10 DWPD n/a <1ms 2.5” 400, 800, 1600GB

Capacity Flash SSD 1 DWPD n/a <1ms 2.5” 3.84TB, 1.92TB

Hybrid HDD n/a 10K 5ms 2.5” 1.2TB, 2TB

Hybrid HDD n/a 7.2K 7ms 3.5 4TB

• 10% cache-capacity ratio for hybrid models (*) DWPD = Number of times a drive can be written per day over 5 years
• For all Flash models we recommend
– 800GB cache drives when using 3.84TB capacity drives
– 400GB cache drives when using 1.92TB capacity drives

17 © Copyright 2017 Dell Inc.

Storage capacity recommendations
Normal rebalancing triggered when a
disk reaches a 80% space use

• A minimum of 20% of free

capacity is recommended Single disk
– Avoid rebalancing overhead capacity
@ 80%
– Ensure enough space for rebuild
or maintenance procedures
Excess rebalancing

Cluster
capacity
@ 80%

No spare capacity
means no capacity
for rebuild or
maintenance
18 © Copyright 2017 Dell Inc.
Network considerations

• The vSAN network can be configured with 1GbE or 10GbE NICs

– 10GbE is strongly recommended
– 1GbE is limited to E and G models and only eight nodes per cluster

• Consider the impact of network topology on bandwidth

– If the switches are oversubscribed at a 4:1 ratio, the 10GbE uplink used by vSAN can only achieve 2.5Gbps

• Consider host/disk group rebuild as part of your network sizing

– Time to rebuild
› It can take a long time to rebuild a host when using a 1Gbps NIC
– Performance effect during rebuild
› Hosts may request to send and receive traffic at wire speed to reduce rebuild time
› High subscriptions ratios can lead to congestion

19 © Copyright 2017 Dell Inc.

 OPTION A OPTION B

Sizing summary VxRail Model

VxRail Type
Rack Unit
E46F
All Flash
1U1N
P470F
All Flash
2U1N
Number of Nodes 20 19
Number of Appliances 20 19

CPU Model E5-2690 v4 E5-2695 v4

Clockspeed (GHz) 2.6 2.1
Number of cores per CPU 14 18
• VxRail provides great flexibility to customize your Number of CPUs per node 2 2
CPU per Node (GHz) 72.8 75.6
appliance means CPU per Cluster (GHz) 1456 1436.4

Memory per Processor (GB) 128 128

• There are lots of HW components, SW features Memory per Node (GB) 256 256
Memory DIMMs per Node 8 x 32 8 x 32
and configuration rules to be considered Memory per Cluster (GB) 5,120 4,864

No. of Disk Groups per node 2 3

• A sizing tool exists to ensure the following: No. of Cache SSDs per node 2 x 800GB 3 x 800GB

– Configuration rules, system overhead and the effect No. of Capacity Drives / node
Total Cluster raw capacity (TB)
7 x 3.84TB SSD
488.6
7 x 3.84TB SSD
464.2
of configuration options are taken in consideration Total Cluster usable capacity(TB) 246.0 253.4

– The correct system performance capabilities are

taken in consideration
– The many different options of hardware components
are evaluated
– The configuration produced is cost x performance
optimized

• Work with your vArchitect in the creation and

evaluation of configuration options

20 © Copyright 2017 Dell Inc.

VxRail I/O Performance and
Scalability
I/O performance

• A well planned vSAN cluster can support What expectations When should I
very demanding I/O workloads, scaling can I have about consider adding
as nodes are added service levels? nodes?

• In this section, we want to discuss the

performance differences related to
– Fault-protection methods
– Data-reduction features
– I/O-utilization levels

Which fault Is dedup a good

tolerance method option for my data
should I use? and workload?

22 © Copyright 2017 Dell Inc.

 System I/O throughput with different protection methods
OLTP 4K—70% Reads
Different fault tolerance methods
require a different number of back-
end I/Os, affecting the front-end I/O
IOPS per cluster throughput
300,000
255,127
250,000
202,000
200,000 181,152

150,000 123,178
100,000

50,000

-
Mirror-1 Mirror-2 RAID-5 RAID-6
Nodedup Nodedup nodedup nodedup

6 x nodes - VxRail P470F – V4.0.100 – vSAN 6.2 – OLTP 4K – 3 stripes

Each node with: 75GHz CPU – 768MB RAM – 2 DGs – 800GB Cache – 6 x 3.84TB SSDs
23 © Copyright 2017 Dell Inc.
Fault-tolerance method (FTM) and back-end I/O amplification
OLTP 4K
IOPS per
cluster

300,000
255,127
250,000
202,000
200,000 181,152

150,000 123,178

100,000
RAID-6 implications
50,000 • RAID-6 is more likely to reach the limit of the VSAN network
bandwidth (10Gb/s per node)
- • RAID-6 response times will be higher as a function of the extra
Mirror-1 Mirror-2 RAID-5 RAID-6 network and disk activity
Nodedup Nodedup nodedup nodedup

6 x nodes - VxRail P470F –V4.0.100 – vSAN 6.2 – OLTP 4K – 3 stripes

Each node with: 75GHz CPU – 768MB RAM - 2 DGs – 800GB Cache – 6 x
3.84TB SSDs

24 © Copyright 2017 Dell Inc.

Factors to consider when choosing an FTM

Throughput Space consumption Protection level FTM option

Highest Base = 100% Single RAID-1 FTT=1

High 200% Dual RAID-1 FTT=2
Moderate 33% Single RAID-5
Lower 50% Dual RAID-6
Each VM can have its own protection method

25 © Copyright 2017 Dell Inc.

System IO throughput for different
Protection methods with and without deduplication
OLTP 4K - 70% Reads
Effect of calculating hash keys and executing
IOPS per cluster lookups on metadata indexes at de-stage
300,000
255,127
250,000
202,000 207,700
200,000 181,152 188,000
153,106
150,000 123,178
100,000
Data not
50,000 available
-
-
Mirror-1 Mirror-2 RAID-5 RAID-6 Mirror-1 Mirror-2 RAID-5 RAID-6
Nodedup Nodedup nodedup nodedup Dedup Dedup Dedup Dedup

6 x nodes – VxRail P470F – V4.0.100 – vSAN 6.2 – OLTP 4K – 3 stripes

Each node with: 75GHz CPU – 768MB RAM – 2 DGs – 800GB Cache – 6 x 3.84TB SSDs
26 © Copyright 2017 Dell Inc.
Is dedup a good option for my data and workload?
Use the potential for data reduction and the write I/O activity in the system as a reference to
determine the value of enabling deduplication

Value = benefit @ cost

High Low-mid value Higher value

Potential for
data reduction
through
deduplication
Low Lower value Mid-high value

Lower value because depending High Low

on the load, nodes/disk groups Write intensity
may be added to support the
at system level
dedup I/O activity, reducing the
benefit of space reduction
27 © Copyright 2017 Dell Inc.
 System I/O throughput for different protection
methods with and without deduplication
OLTP 4K—70% reads
When sizing a new cluster, the choice to use deduplication should consider the
expected I/O utilization levels and protection methods.
IOPS per cluster
275,000 255,127
250,000
225,000 202,000 207,700
200,000 181,152 188,000
175,000 47% 153,106
150,000 Util. 65% 123,178 57% 80%
125,000
Goal: 120,000 IOPS of OLTP 4K
100,000
75,000
Data not
50,000 available
25,000 -
-
Mirror-1 Mirror-2 RAID-5 RAID-6 Mirror-1 Mirror-2 RAID-5 RAID-6
Nodedup Nodedup nodedup nodedup Dedup Dedup Dedup Dedup
6 x nodes – VxRail P470F – V4.0.100 – vSAN 6.2 – OLTP 4K – 3 stripes
Each node with: 75GHz CPU – 768MB RAM - 2 DGs – 800GB Cache – 6 x 3.84TB SSDs
• The term “utilization” is used in this presentation to describe how much of the “system capacities” are being consumed
• Resource capacity can be described in number of I/Os, CPU, memory, storage, …
28 © Copyright 2017 Dell Inc.
 Congestions and I/O utilization
“Congestions” can happen when utilization is higher than 80%

OLTP-4K (RAID-1)
max
500,000

450,000
90%

400,000
80%
70%
350,000

300,000
60%
50%
IOPS

250,000
40% • Congestions are an indication that the
200,000
30% back end of the system cannot keep up
150,000 with the front end.
20%
100,000 • When this happen vSAN throttles I/Os.
10%
50,000

0
13
19
25
31
37
43
49
55
61
67
73
79
85
91
97
1
7

103
109
115
121
127
133
139
145
151
157
163
169
175
181
187
193
199
205
211
217
223
229
235
241
247
253
259
265
271
277
283
289
295
301
307
12 x nodes – VxRail P470F – V4.0.132 – vSAN 6.3 – RDBMS – 3 stripes Intervals
Each node with: 75GHz CPU – 768MB RAM – 2 DGs – 800GB Cache – 6 x 3.84TB SSDs
• The term “utilization” is used in this presentation to describe how much of the “system capacities” are being consumed
29 © Copyright 2017 Dell Inc. • Resource capacity can be described in number of I/Os, CPU, memory, storage, …
Service levels and I/O utilization
Resp time (ms) VxRAIL V4 - Response time curves for OLTP4K using different fault tolerance options
7.0

5.8
6.0 5.6

5.0 4.6

4.0

2.9
3.0
2.3
70%
2.0 1.7 Better performance
1.5
1.3 1.4
1.2 predictability when operating
below 70% of the max that a
1.0
given system config can
produce for a workload
-
- 50,000 100,000 150,000 200,000 250,000 300,000 350,000 400,000 450,000 500,000
12 x nodes – VxRail P470F – V4.0.132 – vSAN 6.3 – RDBMS – 3 stripes IOPS
Each node with: 75GHz CPU – 768MB RAM – 2 DGs – 800GB Cache – 6 x 3.84TB SSDs
30 © Copyright 2017 Dell Inc.
Service levels and deduplication
IOPS vs. response time
OLTP 4K – Dedup versus non-dedup

Resp time (ms) VxRAIL V4 - Response time curves for OLTP4K using different fault tolerance options
7.00
207,705 , 5.4
6.00 5.4
Between 0.4 and 0.6ms added by 5.2
5.00 dedup, when operating below 50%
of max IOPS for configuration 4.0
4.00 255,127.75 , 4.95
3.1
3.00 2.6 3.60
2.1 2.1
1.8
2.00 1.6 2.58
1.97
1.00 1.60 1.52
1.30 1.40 1.28

-
- 50,000 100,000 150,000 200,000 250,000 300,000

IOPS
FLASH-OLTP4K-MIRROR-1-NODEDUP-6 nodes -2 DGs - 800GB cache - 3x3.84TB
FLASH-OLTP4K-MIRROR-1-DEDUP-6 nodes -2 DGs - 800GB cache - 3x3.84TB

31 © Copyright 2017 Dell Inc.

In large configurations, evaluate the option of
splitting the cluster to match workloads to features

vCenter Server

vSphere vSphere vSphere vSphere vSphere vSphere vSphere vSphere vSphere

vSAN vSAN vSAN
vSAN distributed datastore vSAN distributed datastore vSAN distributed datastore

General-purpose VMs Databases VDI

Mixed fault tolerance methods RAID-1, no-dedup RAID-5, dedup
dedup or no-dedup

32 © Copyright 2017 Dell Inc.

Cluster scalability
Resp time (ms) VxRAIL V4 - Response time curves for OLTP4K using different fault tolerance options
7.00 478,486 , 6.4
85%
6.00 282,334.0, 5.4
152,507 , 5.0
5.00

4.00

3.00

2.00

1.00

-
- 50,000 100,000 150,000 200,000 250,000 300,000 350,000 400,000 450,000 500,000
IOPS

V4.0.100 VSAN 6.2-FLASH-OLTP4K-MIRROR-1-NODEDUP-6 nodes -2 DGs - 800GB cache - 3x3.84TB

V4.0.132 VSAN 6.3-FLASH-OLTP4K-MIRROR-1-NODEDUP-12 nodes -2 DGs - 800GB cache - 3x3.84TB
V4.0.132 VSAN 6.3-FLASH-OLTP4K-MIRROR-1-NODEDUP-3 nodes -2 DGs - 800GB cache - 3x3.84TB

33 © Copyright 2017 Dell Inc.

Performance Monitoring
and Optimization
Periodically evaluate resource utilization

• Leverage vSphere Performance Monitor or other

available monitors to keep track of resource
consumption
• CPU utilization
– It should be below 80%

• Check memory usage as a % of the total configured

memory
– Verify if ballooning is being used to release memory pages

• Check storage capacity consumption

– vSAN starts to rebalance space when disk is more than
80% used

• vSAN consumption
– IOPS, throughput, latency, congestions
– Increases in latency and congestions should be evaluated
› Are they a effect of organic growth or result of a VM with an excess
consumption?

35 © Copyright 2017 Dell Inc.

Take advantage of QoS
Example of QoS use to protect a production workload from a heavy test & dev load running on the same host
❶ ❷ ❸ ❹
IOPS • Test load OLTP 60R/40W 32K is started 1st QoS limit applied to test load 2nd QoS limit applied Resp time(ms)
Production load running
on the same host. allowing production load to normalize to test load
100,000.0 in steady state 20.00
• Load runs at max rate and affects the its behavior
production load
18.00

10,000.0 16.00

14.00

1,000.0 12.00

10.00

100.0 8.00
5.25
6.00
3.82
10.0 2.76 4.00
2.79

2.00

1.0 -

Production VM IOPS Test&Dev VM IOPS Production VM - RespTime

P470F-6xnodes - Each node with 75GHz, 768GB and 2 Disk Groups. Each DG with 800GB cache, 3x3.84TB.

36 © Copyright 2017 Dell Inc.

Latency-sensitive applications
VM setting

• Set latency sensitivity to high

– This requires 100% memory reservation

• Consider reserving 100% of the CPU

– This guarantees exclusive PCPU access, which in turn
helps to reduce VCPU halt/wake-up cost

• Consider overprovisioning PCPUs

– This reduces the impact of sharing the last level cache
(LLC) and also helps to improve the performance of
latency-sensitive VMs that use VNICs for network I/O

• Consider disabling all power management in both

the BIOS and vSphere

37 © Copyright 2017 Dell Inc.

Conclusions/recap

• vSAN’s scale out architecture eliminates the need to add a large excess capacity at planning

• A good definition of workload demands improves the quality of the configuration and better satisfies SLOs

• Work with your vArchitect and evaluate different configuration options to optimize cost and performance

• If configured properly, a VxRail system can support the most demanding workloads

• Take advantage of policy-based management to optimize the use of resources and VMs performance

• Consider the effects of the fault tolerance methods and deduplication when making your policy decisions

• In large configurations, consider splitting the cluster to match workloads to policies

• Use vSAN’s or other datacenter monitors to continuously monitoring utilization of resources

• Establish utilization limits

• vSAN’s I/O performance scales almost linearly when adding nodes

38 © Copyright 2017 Dell Inc.

Questions
Realize your next steps

Attend Breakout Sessions Visit the CPSD Booth #872

VxRail Shields Up! Confidently Protecting Your Mission-Critical Workloads with
Data Services
• Join forces with the Xpert and see
if you can make it out of the
• Tuesday, May 9 @ 8:30 – 9:30
• Thursday, May 11 @ 8:30 – 9:30
Xscape Room
VxRail – Level-up with New Capabilities and Powers • Hear customer presentations in the
• Monday, May 8 @ 12 – 1, Singleton Room
theater and talk to Xperts
VxRail – A Customer’s Journey to Hyper-converged Awesome-Sauce
• Monday, May 8 @ 1:30 – 2:30
• Wednesday, May 10 @ 3:00 – 4:00 Get Hands-On
VxRack SDDC – VIP Access Deep Dive
• Visit the Hands-on Labs and test
• Tuesday, May 9 @ 3:00 – 4:00
drive VxRail for yourself
• Wednesday, May 10 @ 1:30 – 2:30
• Chance to demo VMware Cloud
Foundation in the HOL lobby too

40 © Copyright 2017 Dell Inc.