Memory Configuration Best Practices for

2S EP (E5-2600 based) UCS M3 Servers

(Version: 1.02, Nov. 26, 2012)

Ruchi Jain,
Cisco Systems,
ruchjain@cisco.com

This document is Cisco Confidential.

1
1 Purpose of this “Best Practice” document
The purpose of this document is to provide guidance on recommended
memory configurations for Intel Xeon E5-2600 (Sandy Bridge) based
servers.

The document gives specific details on populating DIMMs in various

memory channels/banks to obtain best performance. These
recommendations are based on in-house memory performance
benchmark testing.

This document also lists supported processor type, speed, and maximum
memory for currently released Intel Xeon E5-2600 based servers.

2 Introduction to Intel Xeon E5-2600 Processors

Intel has followed a tick tock model for CPU evolution, where the tick
generation uses advances in manufacturing technology, while the tock
generation provides new micro architecture advances. The Intel Xeon
processor E5-2600 Series (Sandy Bridge) represents a major micro-
architectural innovation (tock) with up to eight cores per single socket
and up to 8 instructions per CPU clock, and providing significant
improvement over previous-generation Intel Xeon processors.

This document is Cisco Confidential.

2
3 Currently Released Intel Xeon E5-2600 based UCS M3
Servers
Cisco UCS Intel Xeon E5-2600 based architecture are designated M3
servers. The following tables list currently released M3 Blade and Rack
servers.

Table 1: Currently Released UCS M3 Blade Server

Server Processor Sockets Memory Slots
B200 M3 E5-2600 2 24

Table 2: Currently Released UCS M3 Rack servers

Server Processor Sockets Memory Slots
C220 M3 E5-2600 2 16
C240 M3 E5-2600 2 24

This document is Cisco Confidential.

3
4 UCS 2-Sockets E5-2600 based Servers
4.1 Processor Memory Architecture

Please Note:
 C220 M3 Server has only two banks: Bank1 and Bank2.
 B200 M3 bank labels differ from picture above. These are: Bank0, Bank1,
and Bank2
 Please refer to Table 5: Memory Bank Labels/Colors/Slots in UCS M3
Servers for more details on CPU-memory architecture on a specific server.

This document is Cisco Confidential.

4
 4.2 Currently Supported Processors on UCS M3 Servers

Table 3: Currently Supported Processors in B200 M3, C220 M3 and

C240 M3

Highest Number Processor Threads SPEC_int

Intel Product ID DIMM of Cores Clock per Max QPI _rate_base
Processor (PID) Operation per Speed Processor Power Speed for
Speed Processor (GHz) (W) (GT/s) C220M3
(MHz)

E5-2690 UCS-CPU-E5-2690 1600 8 2.90 16 135 8 671

E5-2680 UCS-CPU-E5-2680 1600 8 2.70 16 130 8 640

E5-2670 UCS-CPU-E5-2670 1600 8 2.60 16 115 8 N/A*

E5-2667 UCS-CPU-E5-2667 1600 6 2.90 12 130 8 N/A*

E5-2665 UCS-CPU-E5-2665 1600 8 2.40 16 115 8 592

E5-2660 UCS-CPU-E5-2660 1600 8 2.20 16 95 8 575

E5-2650 UCS-CPU-E5-2650 1600 8 2.00 16 95 8 521

E5-2650L UCS-CPU-E5-2650L 1600 8 1.80 16 70 8 444

E5-2643 UCS-CPU-E5-2643 1600 4 3.30 8 130 8 364

E5-2640 UCS-CPU-E5-2640 1333 6 2.50 12 95 7.2 447

E5-2637* UCS-CPU-E5-2637 1600 2 3.00 4 95 7.2 N/A*

E5-2630 UCS-CPU-E5-2630 1333 6 2.30 12 95 7.2 419

E5-2630L UCS-CPU-E5-2630L 1333 6 2.00 12 60 7.2 379

E5-2620 UCS-CPU-E5-2620 1333 6 2.00 12 95 7.2 379

E5-2609 UCS-CPU-E5-2609 1066 4 2.40 4 80 6.4 218

This document is Cisco Confidential.

5
 Please Note:
 Processors marked with an * (asterisk) are not supported in Rack servers
(C220 M3, C240 M3) as of writing.
 Some processors support peak DIMM speed (1600 MHz). These are
highlighted in yellow.
 Intel’s hyper threading and turbo boost technologies are not
supported for processor E5-2609.
 SPEC_int_rate_base benchmark (the last column in table above)
shows the best results for 8GB dual rank 1600MHz DIMMs in 2DPC
(total 128GB) configuration as published on www.spec.org.
 N/A* - SPEC_int_rate_base results for some of the recently added
CPUs are not available as of writing. This result column will be
updated once these results are available.
 If the CPU doesn’t support the peak DIMM speed of 1600 MHz, then
the system memory speed will clock down to the speed supported
by CPU, even with 1600 labeled DIMMs. For example: In a server
with E5-2609 CPUs, the operating memory speed will be 1066 MHz.

This document is Cisco Confidential.

6
 4.3 Currently Supported DIMMs
4.3.1 Size, Voltage, and Ranks
The main variables that affect memory performance are type of
processors in the system, memory frequency, memory ranks, and memory
population across various memory channels and processors.

Table 4: Currently Supported DIMMs Size, Voltage, and Ranks

Capable Capable Highest Ranks
DIMM Operation Operation Operation DIMM per PID
Size Voltage Speed Speed Type DIMM
4GB 1.35V &1.5V 1600MHz 1600MHz R 1 UCS-MR-1X041RY-A
1.35V &1.5V 1333MHz 1333MHz R 1 UCS-MR-1X041RX-A*
8GB 1.35V &1.5V 1600MHz 1600MHz R 2 UCS-MR-1X082RY-A
1.35V &1.5V 1333MHz 1333MHz R 2 UCS-MR-1X082RX-A
16GB 1.35V &1.5V 1600MHz 1600MHz R 2 UCS-MR-1X162RY-A
1.35V &1.5V 1333MHz 1333MHz R 2 UCS-MR-1X162RX-A*
32GB 1.35V & 1.5V 1600MHz 1333MHz LR 4 UCS-ML-1X324RY-A

Please Note:
 DIMMs can operate in either “Power Saving Mode” or “Performance
mode”. Default mode is “Power Saving Mode (1.35V)”for rack servers
(C220 M3 and C240 M3). Default mode is “Performance Mode (1.5V)” for
blade server (B200 M3).
 DIMMs marked with * (asterisk) are not supported in B200 M3 servers.
 32 GB DIMMs are quad-rank, load-reduced (LR) DIMMs. These DIMMs
cannot be mixed with any other supported DIMMs.
 All other supported DIMMs (4GB, 8GB, and 16GB) are RDIMMs.
 For 32GB LR DIMMs, the maximum operating memory speed is 1333 MHz
at 1DPC and 2DPC even though their labeled maximum speed is 1600 MHz.
 All the DIMMs within the same server must have same speed for peak
performance.

This document is Cisco Confidential.

7
4.3.2 Type of DIMMs supported
4.3.2.1 RDIMM (Registered DIMM):

Registered DIMMs have a register on the DIMM which buffers the

address and command signals between the DRAM chip and the
memory controller. Data signals are not buffered in these types of
DIMMs. The 4GB, 8GB and 16GB DIMMs supported on 2-socket UCS
M3 Servers are registered DIMMs. These DIMMS are either single or
dual ranked.
4.3.2.2 LR-DIMM (Load Reduced DIMM):
Load reduced DIMMs have a “Memory buffer” on the DIMM, which
buffers the address, command and data signals between the DRAM chip
and the memory controller. These DIMMs have additional components
which have rank multiplication logic to logically reduce the number of
ranks visible to the memory controller. This special control logic make the
DIMM look like a single or dual rank DIMM to the memory controller. The
“Memory buffer” on these DIMMs reduces the electrical load on the data
bus and provides higher memory capacity. The 32GB DIMMs supported on
2-socket UCS M3 servers are load-reduced DIMMs. These DIMMs are quad-
ranked and cannot be mixed with RDIMMs in a server. LR DIMMs maximize
system memory capacity.

This document is Cisco Confidential.

8
4.4 Physical Layout of CPU, DIMM Channels and Banks

This document is Cisco Confidential.

9
This document is Cisco Confidential.

10
This document is Cisco Confidential.

11
Table 5: Memory Bank Labels/Colors/Slots in UCS M3 Servers
UCS Server 1st Bank 2nd Bank 3rd Bank

Label (Color) Label (Color) Label (Color)

CPU1: Slots CPU1: Slots CPU1: Slots
CPU2: Slots CPU2: Slots CPU2: Slots
B200 M3 Bank 0 (Blue) Bank 1 (Black) Bank 2 (White)

CPU1: A0, B0, C0, D0 CPU1: A1, B1, C1, D1 CPU1: A2, B2, C2, D2
CPU2: E0, F0, G0, H0 CPU2: E1, F1, G1, H1 CPU2: E2, F2, G2, H2
C220 M3 Bank 1 (Blue) Bank 2 (Black) Not applicable

CPU1: A1, B1, C1, D1 CPU1: A2, B2, C2, D2

CPU2: E1, F1, G1, H1 CPU2: E2, F2, G2, H2
C240 M3 Bank 1 (Blue) Bank 2 (Black) Bank 3 (Black)

CPU1: A1, B1, C1, D1 CPU1: A2, B2, C2, D2 CPU1: A3, B3, C3, D3
CPU2: E1, F1, G1, H1 CPU2: E2, F2, G2, H2 CPU2: E3, F3, G3, H3

This document is Cisco Confidential.

12
4.5 DIMMs Operational Speed and Voltage
Intel E5-2600 EP processor-memory architecture has 4 memory channels per
processor socket. It allows DIMMs with a maximum aggregation of 2 DIMMs per
channel to be able to operate at the labeled speed. (However, there is an
exception for 32GB LRDIMMs. Please see section 4.5.2. for more details.) When
the aggregated DIMM number in a channel is greater than 2, DIMMs will
operate at 1066MHz speed, regardless of manufacturer labeled speed.

4.5.1 4GB, 8GB, 16GB Operational Speed and Voltage

Table 6: DIMMs Speed and Voltage at 1DPC, 2DPC, and 3DPC

DIMMs Per Channel Power Saving Mode (Default ) Performance Mode
Voltage Speed Voltage Speed
1 DPC 1.35 V 1333 MHz 1.5 V Labeled
2 DPC 1.35 V 1333 MHz 1.5 V Labeled
3 DPC 1.5 V 1066 MHz 1.5 V 1066 MHz

Please Note:
 Performance mode is the default mode for B200 M3. Performance
mode forces the DIMMs to operate at 1.5 V and yields 1600-MHz
speed provided:
o The DIMMs are 1600-MHz devices and are R-DIMMs.
o The CPUs chosen support 1600-MHz operation. Please refer to Table
3: Currently Supported Processors in B200 M3, C220 M3 and C240 M3.
o There are less than 3 DIMMs per channel.
 In order to run the DIMMs at power-saving mode (1.35 V) in B200 M3,
you need to go into the BIOS and change the settings or set the
policy with UCSM (service profile).
 Power-saving mode (1.35 V) is the default mode for C220 M3 and
C240 M3 servers.
 1DPC and 2DPC can provide the maximum rated speed that the
CPU and DIMMs are labeled for. This is highlighted in yellow in table
above.
 3DPC causes the DIMMs to run at slower speed (1066MHz).
 With 3DPC, memory always runs at 1.5 V regardless if the BIOS
setting is power-saving mode (1.35 V) or performance mode (1.5 V).

This document is Cisco Confidential.

13
4.5.2 32 GB Operational Speed and Voltage
The newly supported 32GB DIMMs are quad-ranked, LR DIMMs. The
maximum memory operating speed in an E5-2600 based 2-socket server is
1333 MHz with 32GB LR DIMMs. Table7. lists the memory system operating
speed with 32GB LR DIMMs for E5-2600 based 2-socket M3 servers based
on the number of DIMMs installed in the server.

Table 7 . DIMMs Speed and Voltage at 1DPC, 2DPC, and 3DPC

DIMMs Per Channel Power Saving Mode Performance Mode (Default)
Voltage Speed Voltage Speed
1 DPC 1.35 V 1066 MHz 1.5 V 1333 MHz
2 DPC 1.35 V 1066 MHz 1.5 V 1333 MHz
3 DPC 1.35 V 1066 MHz 1.5 V 1066 MHz

Please Note:
 The system will run at a maximum of 1333 MHz at 1.5 V in 1DPC and
2DPC configurations with 32GB LRDIMMs even though their labeled
speed is 1600MHz.
 3DPC causes the DIMMs to run at slower speed (1066MHz).
 With 32GB DIMMs at 3DPC, memory runs at 1.5 V if the BIOS setting is
performance mode (regular voltage) and memory runs at 1.35 V
(low voltage) if the BIOS setting is power-saving mode.

This document is Cisco Confidential.

14
 4.6 Memory Configurations for Best Performance
The 2-socket based Intel E5-2600 allows many memory configuration
combinations. We encourage our field teams to configure memory sizes as
indicated below. These are created based on our internal memory
performance benchmark testing. Please note that for a total amount of
memory in a system, there are often many possible combinations. We
encourage our field teams to configure memory sizes in the sweet spot
configurations (marked in yellow) for optimum performance.

Table 8: Recommended Memory Configurations for C220 M3 and C240 M3

Total CPU-1 CPU-2

System
Memory Blue Black Black Blue Black Black DIMM Total
Size Slots Slots Slots Slots Slots Slots Max DIMMs
Relative
Oper. in the
Perf
Bank 1 Bank 2 Bank3 Bank 1 Bank 2 Bank 3 Speed system
(MHz) vs.
(A1, B1, (A2,B2, (A3,B3, (E1,F1, (E2, F2, (E3, F3,
C1, D1) C2,D2) C3,D3) G1,H1) G2, H2) G3,H3) Peak
Bandwidth

32GB 4x4GB - - 4x4GB - - 1600 8 0.90

2x8GB - - 2x8GB - - 1600 4 0.54
64GB 4x4GB 4x4GB - 4x4GB 4x4GB - 1600 16 0.98
4x8GB - - 4x8GB - - 1600 8 1.00
96GB 4x4GB 4x4GB 4x4GB 4x4GB 4x4GB 4x4GB 1066 24 0.68
4x8GB 2x8GB - 4x8GB 2x8GB - 1600 12 Unbalanced
Config*
3x16GB - - 3x16GB - - 1600 6 0.79
4x8GB 4x4GB - 4x8GB 4x4GB - 1600 16 1.00
128GB 4x8GB 4x8GB - 4x8GB 4x8GB - 1600 16 1.00
4x16GB - - 4x16GB - - 1600 8 0.99
4x8GB 4x8GB 4x8GB 4x8GB 4x8GB 4x8GB 1066 24 0.71
4x16GB 2x16GB - 4x16GB 2x16GB - 1600 12 Unbalanced
192GB
Config*
4x16GB 4x8GB - 4x16GB 4x8GB - 1600 16 0.98
4x16GB 4x16GB - 4x16GB 4x16GB - 1600 16 1.00
256GB 4x32GB - - 4x32GB - - 1333 8 0.83
384GB 4x16GB 4x16GB 4x16GB 4x16GB 4x16GB 4x16GB 1066 24 0.70
512GB 4x32GB 4x32GB - 4x32GB 4x32GB - 1333 16 0.79
768GB 4x32GB 4x32GB 4x32GB 4x32GB 4x32GB 4x32GB 1066 24 0.70

This document is Cisco Confidential.

15
 Please Note:
 Bank 3 does not exist in C220 M3 Servers.
 Since C220 M3 has only 16 DIMM slots (only two banks per CPU), “96GB
with 24x4GB” and “384GB with 24x16GB” memory combinations are not
applicable.
 Yellow Highlighted Cells represent Sweet Spot or Best
Recommended configurations for achieving optimum performance
in a system.
 Asterisk (*) represents unbalanced configurations in a single
channel. These configurations are possible but not recommended.
 For 128 GB, there are two possible sweet spot configurations:
o 8X16GB occupies 1 DPC, so there is a possibility of increasing
total memory size in the system (beyond 128 GB) by going to
2 DPC while still maintaining memory speed at 1600MHz.
o If there is no plan to increase total memory size in the system
(beyond 128 GB), then 16x8GB is more price effective at this
time.
 For 64 GB, there are two possible optimal configurations but we
recommend 8x8GB.
o 8X8GB occupies 1 DPC, so there is a possibility of increasing
total memory size in the system (beyond 64 GB) by going to 2
DPC while still maintaining memory speed at 1600MHz. Also
8x8GB consumes less power compared to the 16x4GB option.
Finally, 8GB is more price effective than 2x4GB DIMMs.

This document is Cisco Confidential.

16
 Table 9: Recommended Memory Configurations for B200 M3
Total CPU-1 CPU-2
System
Memory Blue Black White Blue Black White DIMM Total
Size Slots Slots Slots Slots Slots Slots Max DIMMs
Relative
Oper. in the
Perf
Bank 0 Bank 1 Bank2 Bank 0 Bank 1 Bank2 Speed system
(MHz) vs.
(A0, B0, (A1,B1, (A2,B2, (E0,F0, (E1,F1, (E2,F2,
C0, D0) C1,D1) C2,D2) G0,H0) G1,H1) G2,H2) Peak
Bandwidth

32GB 4x4GB - - 4x4GB - - 1600 8 0.90

2x8GB - - 2x8GB - - 1600 4 0.54
64GB 4x4GB 4x4GB - 4x4GB 4x4GB - 1600 16 0.98
4x8GB - - 4x8GB - - 1600 8 1.00
96GB 4x4GB 4x4GB 4x4GB 4x4GB 4x4GB 4x4GB 1066 24 0.68
4x8GB 2x8GB - 4x8GB 2x8GB - 1600 12 Unbalanced
Config*
3x16GB - - 3x16GB - - 1600 6 0.79
4x8GB 4x4GB - 4x8GB 4x4GB - 1600 16 1.00
128GB 4x8GB 4x8GB - 4x8GB 4x8GB - 1600 16 1.00
4x16GB - - 4x16GB - - 1600 8 0.99
4x8GB 4x8GB 4x8GB 4x8GB 4x8GB 4x8GB 1066 24 0.71
4x16GB 2x16GB - 4x16GB 2x16GB - 1600 12 Unbalanced
192GB
Config*
4x16GB 4x8GB - 4x16GB 4x8GB - 1600 16 0.98
4x16GB 4x16GB - 4x16GB 4x16GB - 1600 16 1.00
256GB 4x32GB - - 4x32GB - - 1333 8 0.83
384GB 4x16GB 4x16GB 4x16GB 4x16GB 4x16GB 4x16GB 1066 24 0.70
512GB 4x32GB 4x32GB - 4x32GB 4x32GB - 1333 16 0.79
768GB 4x32GB 4x32GB 4x32GB 4x32GB 4x32GB 4x32GB 1066 24 0.70

Please Note:
 Yellow Highlighted Cells represent Sweet Spot or Best
recommended configurations for achieving optimum performance
in a system.
 Asterisk (*) represents unbalanced configurations in a single
channel. These configurations are possible but not recommended.
This document is Cisco Confidential.

17
 For 128 GB, there are two possible sweet spot configurations:
o 8X16GB occupies 1 DPC, so there is a possibility of increasing
total memory size in the system (beyond 128 GB) by going to
2 DPC while still maintaining memory speed at 1600MHz.
o If there is no plan to increase total memory size in the system
(beyond 128 GB), then 16x8GB is more price effective at this
time.
 For 64 GB, there are two possible optimal configurations but we
recommend 8x8GB.
o 8X8GB occupies 1 DPC, so there is a possibility of increasing
total memory size in the system (beyond 64 GB) by going to 2
DPC while still maintaining memory speed at 1600MHz. Also
8x8GB consumes less power compared to the 16x4GB option.
Finally, 8GB is more price effective than 2x4GB DIMMs.
 These results are based on memory performance measurements
done for C240M3. B200M3 results are similar.

This document is Cisco Confidential.

18
4.7 Example of Memory Bandwidth as a function of DPC (DIMMs
per Channel)

Stream Bandwidth Triad (MB/s) for

8GB/1600MHz DIMMs
90000

80000

70000
Stream Triad (MB/s)

60000

50000

40000

30000

20000

10000

0
Speed 1600 MHz 1600 MHz 1066 MHz
DPC 1 DPC (8x8GB) 2 DPC (16x8GB) 3 DPC (24x8GB)
8GB 2Rx4 1600 MHz DIMMs on C240 M3

Summary:
 1 DPC and 2 DPC provide maximum memory performance.
 3 DPC causes the DIMMs to run at a slower speed.

4.8 Best Practice Rules for DIMM Population

 There are 4 memory channels per CPU. Each channel has three
DIMM slots (for example, channel A = slots A1, A2, and A3).

o Each channel can operate with one, two or three DIMMs

installed.
o If a channel has only one DIMM, populate slot 1 first (the blue
slot).
 When both CPUs are installed, populate the DIMM slots of each CPU
identically.

This document is Cisco Confidential.

19
 o Fill bank 1 blue slots in the channels first: slots A1, B1, C1, D1in
CPU1 and E1, F1, G1 and H1 in CPU2.

o Fill bank 2 black slots in the channels second: A2, B2, C2, D2 in
CPU1 and E2, F2, G2, H2 in CPU2.

o Fill bank 3 black slots (white in blade) in the channels last: A3,
B3, C3, D3 in CPU1 and E3, F3, G3, H3 in CPU2.

 For optimum performance, populate at least one DIMM per

memory channel per CPU.

 When populating DIMM slots, multiples of 8 DIMMs are best since

there are 4 memory channels per CPU socket and 2 CPUs must be
populated.

 At 1 DPC and 2 DPC, 1600MHz labeled R-DIMMS will run at 1600MHz

and 1333 MHz labeled R-DIMMs will run at 1333MHz provided CPU
supports 1600 MHz DIMMs.

 At 1 DPC and 2 DPC, 32GB LR-DIMMS will run at 1333MHz even

though their labeled maximum speed is 1600 MHz.

 DIMMs can be populated at 3DPC in B200M3 and C240 M3. But

there is a frequency downgrade penalty (1066 MHz) when
populating all 24 slots on the B200 M3 and C240 M3 servers.
 When mixing DIMMs , the following rules need to be followed:

o R-DIMMs (4GB, 8GB, 16GB) cannot be mixed with 32GB LR-

DIMMs.
o When mixing R-DIMMs of different densities (sizes), populate
DIMMs with the highest density first. For example, if you have
to mix 16GB with 8GB DIMMs, then populate 16GB DIMMs in
blue slots (or bank 1) and then 8 GB in black slots (or bank 2).
o When mixing R-DIMMs with different ranks, populate DIMMs
with the higher rank first. For example, populate 16GB 2R or
8GB 2R before 4GB 1R DIMMs.
o R-DIMMs of different sizes (4GB, 8GB, 16GB) can be mixed
across different channels, but the total memory size in a single
channel should be the same. For example, if you are mixing
4GB and 8GB DIMMs to achieve total 96GB (8x8GB+8x4GB)
for a server, then populate blue slots with 8GB and black slots
with 4GB and total memory size should be 12GB across all 4
channels for both CPUs.
o When mixing DIMMs, all the DIMMs in the server must be the
same speed.

This document is Cisco Confidential.

20
 5 Revision History

Date Revision Modifications Author

Number
5/7/2012 v1.0 Initial Version Ruchi
Jain
06/25/2012 v1.01 Added : Ruchi
1. Availability of E5-2660 and E5-2690 on Jain
B200 M3
2. Availability of E5-2660 on C220 M3 and
C240 M3
3. “Cisco confidential” footer
11/26/2012 V1.02 Added: Ruchi
1. Availability of E5-2670 and E5-2667, E5- Jain
2650L, E5-2637, E5-2630, E5-2630L and
E5-2609 on B200 M3.
2. Availability of E5-2667, E5-2670 on C220
M3 and C240 M3
3. Availability of 32GB LR DIMMs
4. Type of DIMMs : RDIMMs and LR DIMMs
5. Updated memory cheat sheet with
32GB memory configurations.

This document is Cisco Confidential.

21