1 Novascale Bullion - Hardware

Mesca-X Training - Hardware
C.Vangramberen – 2011 May

SUMMARY
- Overview - FRUs
- Rack
- Drawer 4S3U - Maintenance Procedures
- Boards - Documentation
- MXB
- ILB
- DSIB
- PDB
- FANs
- Disks
- LCP
- Power Supply
- PCIe boards
2 ©Bull, 2010 © Bull Confidentiel - Reproduction interdite sans accord préalable

Why Mesca (technical viewpoint) ?
- A technology disruption in Intel processors

Itanium Itanium
Ita- Ita- Ita- Ita- Next Next
or or
nium2 nium2 nium2 nium2 Bus Pt to pt Itanium Itanium
Xeon Xeon
(FSB) vs. Connections
(QPI)
Itanium Itanium
Next Next
Glue vs. Glueless or or
SNC Itanium Itanium
SMP Xeon Xeon
Frontside vs. Backside

memory
To more processors, To more processors,
memories & to I/Os memories & to I/Os
- With 2 objectives for Intel :
- Embrace state of the art : pt-to-pt links, backside memory, glueless SMP
- Ultimately converge Itanium and Xeon onto a common infrastructure
A new server platform architecture is required

Intel® Processor Overview
Last Level Cache L3 L3

Core-1 Core-2
24MB 6 MB 6 MB
Core-2
Core-4
Core-5
Core-6
Core-7
Core-8
Core-1
Core-3
L3 L3
Core-3 Core-4
6 MB 6 MB
Core/System Interface Switch
Mem Mem Mem Mem

Router Router
Cntrl Cntrl Cntrl Cntrl
QPI Links 4 SMI mem QPI Links 4 SMI mem

to CPU/I/O channels to CPU/I/O channels
Nehalem-EX CPU Tukwila-MC CPU

(a.k.a. NHM) (a.k.a. NHM)
Xeon Processor Itanium Processor

Family 2H10 Family 2H09
Binary compatible Binary compatible
within XPF family within IPF family

Intel® Nehalem-EX Overview

Nehalem-EX : Block Diagram

Nehalem-EX : New 3-level Cache Hierarchy
1 st level caches
- 32kB Instruction cache
- 32kB Data Cache
2 nd level Cache
- New cache introduced in Nehalem
- Unified (holds code and data)
- 256 kB per core
- Performance : Very low latency
- Scalability : As core count increases, reduce pressure on shared cache
3 rd level cache
- Shared across all cores
- Size depends on # of cores
- 8 cores: Up to 24MB
- Scalability:
- Built to vary size with varied core counts
- Built to easily increase L3 size in future parts
- Inclusive cache policy for best performance
- Data residing in L1/L2 must be present in 3 rd level cache

Xeon 7500 Processors
Not in catalog

Package Details: Nehalem-EX

BOARDS organization
BASE DRAWER :
- 4 Nehalem-EX (Expandable)
- 3U large configuration
- 32 DDR3 + 16 MillBrook
- QPI links @ 6.4 GT/s
- 2 IOH (Boxboro)
- 6 PCIe slots Gen2
- HDD SAS or SATA
. 4 or 8 SAS HDD
. 6 SATA HDD
- “BCS Ready”

4S3U Drawer
SIB board
Or dummy
Nehalem-EX (up to 4)
Hot swap fans

0 m m
74
4
Ethernet
ports
Disks (up to 8)
Hot swap
(2 x 4) SATA/SAS
45 Up to
0 135
m 32 Dimms
2 power supplies m mm
W e ig h t fu lly p o p u la te d : ~ 3 5 K g
6 PCI-e Gen2 slots P o w e r c o n s u m p t io n m a x lo a d : ~ 1 3 2 5 W
2-x16, 4-x8 P o w e r c o n s u m p t io n id le : ~ 7 4 0 W
BOARDS mechanics 4S3U
PDU DSIB
LCP NHM
Power Distribution Board
Local Control Pannel Dummy BCS Interconnect Board
HDX MXB ILB

Hard Disk Memory XEON Board IO Legacy Board
drive Board - Processors (4x NEHALEM) -I/O attachment (Boxboro IO
- 32 DDR3 Hub for PCIe slots – ICH10)
- Programmable controller for misc. - Programmable Contoller
functions (Reset, Power on/off, …)
- Fan Connexion and control - Connection to the BMC Board

Board Set for Base Drawer 4S3U
- Board (MXB)
MXB Memory Xeon Board Processors (4x Xeon
Nehalem chips)
- 32 DDR3 + 16 MillBrook
- Programmable controller for misc. functions (Reset, Power on/off, …)
- Board (ILB)
ILB IO Legacy Board
- I/O attachment (Boxboro IO Hub for PCIe slots –ICH10)
- IBMC : Based on nuvoTon BMC chip for PlatformManagement access over LA
- Board (PDBU)
PDBU for power distribution
- Local Control Panel (LCP)
- Board (FDB) Fan Distribution Board : Belongs to Fan Box
- Dummy SIB (DSIB)
DSIB to provide QPI connection in configuration without
BCS
- DSIBs will be used in place of SIB
- Board (SIB)
SIB for BCS and interconnect support
- 1 BCS chip
- Box (HDX)
HDX for HDD (4xSAS) connection

Blocks diagramm
Block A
32xDDR3
FAN
4xNHM FPGA
MXB
Block C: ILB_LEG
DSIB Block B2 PCI ES1000

4x
SATA
ILB_IOLE
QPI ICH10
VGA
TPM
LPC LCP
2xUSB
Block B1 FPGA ESI
ILB_IOE IOH IOH
1 PWR
0 USB LPC DVI Front
Panel PDB
Kawela 16x 8x 8x 16x 8x 8x Kawel
RMII 2xSerial
Block E: a
ILB_COM LAN
SuperIO
Pcie Pcie Pcie Pcie Pcie Pcie DVI Jtag DBG
2X
LAN LAN 16x 8x 8x 16x 8x 8x
LAN LAN
trans AC/DC
CLK
Jtag Test
iBMC
nuvoTon
Block F: Block D
ILB_CLK
ILB ILB_MNG
HDB HDB
optional

FANs
There are 4 x 2 redundant, hot-

pluggable FANs.
Status led

Servicing the Fan Box (summary)
- Place server in servicing position
- Remove all the Deflectors.
- Disconnect the four fan box
cables
from the MXB board
- Unlock the Fan Box
extract the fan box by pushing it

to the right and upward, out of its
location
QPI Links – without BCS
ILB
PCI KAWELA PCI KAWELA
IOH IOH
CSI0 CSI0
I0 I1
CSI1 CSI1
G3 G4
CPU CPU
DSIB DSIB
MTB
CPU CPU G1 G2
IOH IOH
Processor P0
Processor P2
Processor P1
Processor P3
CSI0 CSI1 CSI0 CSI1 CSI0 CSI1 CSI0 CSI1
CSI2 CSI3 CSI2 CSI3 CSI2 CSI3 CSI2 CSI3
For 4 socket-only systems CSI4 CSI5 CSI4 CSI5 CSI4 CSI5 CSI4 CSI5

Servicing the DSIB (summary)
Place the Server Drawer in the Servicing Position

Disconnect ILB and MXB cables
Loosen the two large screws(A) securing the DSIB to the rear of the server drawer
alternatively, beginning with the screwdriver and ending manually.
Carefully pull the board out of the server drawer.
Remove the DSIB.
Important The DSIB board is secured

to the rear of the server drawer by a
tenon-mortise device.
Take care not to damage connectors
when removing the board.

Block A :MXB - NHM processor+ MEM + FAN
Nehalem
socket
Nehalem
NHM NHM
socket
IO/BCS interface
FBDIMM ARRAY
Nehalem
NHM NHM
socket
DSIB
IOH IOH
Nehalem
socket
For 4 socket-only systems
QPI full width links

FBDIMM Channel SPI interface
SPI: Serial Protocol Interface

Block A : NHM processor + MEM + FAN
- Functions
- Processor NEHALEM
- DC/DC converters
- Debug interface (XDP)
- Interfaces (per NHM)
- 4 QPI links : High speed links (6.4 GT/s) for :
• interconnecting to other processors
• Connecting Node controller (BCS) or IOH (4.8 GT/s)
- 2x FBDIMM for each MemChannel
• GEN1 (533 -- 667 – 800 MHz)
- SPI FLASH (1 per socket) for boot firmware
- FAN connection

Servicing the MXB (summary)
Pull the drawer out of the cabinet
Remove all the deflectors
Remove the Fan Box.
Disconnect the cables
De-populate the MXB:
Remove all the processors and DIMMs
Remove the DSIB.
Remove the Prepositioner.
Remove the MXB
Install the new MXB
....
Update the FPGA Firmware.
(and North bios if needed)

Populating DIMM rules
- Each NEHALEM socket hosts 2 independant

memory controllers
- Each memory controller owns 2 FBD chanels
- The FBD chanels operates by pairs obligatorily
– Memory chanels must be populated by
pairs: chanels A and B, and/or chanels C
and D.
– Memory chanels can be populated with
0, 1 or 2 DIMM: DIMM0 and DIMM1 .
- Order for populating: (A0 and B0), then (C0 and D0),
then (A1 and B1), then (C1 and D1)
supported DIMMs: 2GB, 4GB, 8GB, 16GB

MillBrook converter : FBDIMM to DDR3
- 8 DDR3 / Socket
- 2 MillBrook attached by memory controller
- 2 DDR3 per MillBrook

Locating CPUs & DIMMs
CPU IDs P0 P1 P2 P3
MC1 MC0 MC1 MC0 MC1 MC0 MC1 MC0
CH CH CH CH CH CH CH CH CH CH CH CH CH CH CH CH
C D A B C D A B C D A B C D A B
MB MB MB MB MB MB MB MB
#C #B #C #B #C #B #C #B
RDIMM #1
RDIMM #0
RDIMM #0
RDIMM #1
RDIMM #1
RDIMM #0
RDIMM #1
RDIMM #1
RDIMM #0
RDIMM #0
RDIMM #1
RDIMM #1
RDIMM #0
RDIMM #0
RDIMM #1
RDIMM #0
RDIMM #0
RDIMM #0
RDIMM #1
RDIMM #1
RDIMM #0
RDIMM #1
RDIMM #1
RDIMM #1
RDIMM #0
RDIMM #0
RDIMM #0
RDIMM #0
RDIMM #1
RDIMM #1
RDIMM #1
RDIMM #0
MB MB MB MB MB MB MB MB
#D #A #D #A #D #A #D #A
DIMM IDs 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
DIMM Pair IDs 3 1 7 5 11 9 15 13

2 0 6 4 10 8 14 12
Populate CPUs in the following order: P0, P2, P1, P3
Populate DIMM pairs in the following order: P0 = { 0, 2, 1, 3 } P2 = { 8, 10, 9, 11 }
P1 = { 4, 6, 5, 7 } P3 = { 12, 14, 13, 15 }
Servicing a Processor assembly (remove)
The processor assemblies are numbered from 0 to 3 from left to right
Place the Server Drawer in the Servicing Position

Remove the required deflector
Remove the Processor:

With the T15 Torx screwdriver unscrew
the heat sink from the socket.

Servicing a Processor assembly (remove)
. Remove the heat sink.
The heat sink may be too much sticked to the processor due
to the thermal insulating material. In this case, carefully twist
the heat sink to free it from the processor.
. Fully open the lever arm (A) and open the load plate (B) by
pressing on the tail.
. Hold the processor on top and bottom

edges and pull it out of the socket.

Servicing a Processor assembly (install)
. Locate the foolproofing devices (A) to prevent you from incorrectly
installing the processor.
. Lower the processor onto the socket

vertically.

Servicing a Processor assembly (install)
. Close the load plate (A). Latch the lever arm
(B) by pressing down and into the assembly.
. Place the heat sink onto the processor

vertically. Firmly press it down to allow
the two screws (A) to fit into place.

Blocks B and C - I/Os
x16
PE1 PCIe x16
CSI0
x8
PCIe x8
QPI Full width
PE0 x8
IOH1 PCIe x8
Block B
CSI1
x4
LAN
kAWELA
RMII
LAN
x16
PE1 PCIe x16
CSI0 x8
QPI Full width PCIe x8
Legacy x8
PE0 PCIe x8
CSI1 IOH0
x4 LAN
kAWELA
RMII LAN
ESI
LPC TPM
Serial
LPC
to ILB_MNG
SIO Serial
Flash Spi
USB(2)
Block C
Sata DRAM
Sata ICH10
PCI VGA
ES1000 Video
USB
DVO

Block B - I/Os
- Functions
- IO Hubs (Boxboro for MESCA-X)
- ethernet controller (KAWELA)
- Interfaces
- PCIe slots (GEN2 = 5GT/s)
• 2 slots x16
• 4 slots x8
• Internal connections :

KAWELA Dual ethernet controller 10-100-1000
(connected on 1 x4 PCIe)

• 1 ESI (aka DMI) for ICH10 attachment
ESI:Enterprise South Bridge Interface DMI:Direct Media Interface

Blocks C – Legacy I/Os
- Functions
- Support ICH10
- Flash SPI for South bridge firmware
- controller VIDEO (ES1000) and its SDRAM (DDR2)
- Interface LPC to BMC for KCS/BT interface
- Interface USB to BMC for USB redirection over LAN
- Interface DVO to BMC for KVM over LAN
- Battery for RTC
- TPM
- Interfaces
- DMI/ESI with IOH
- SATA attachment (6 drives)
- 2xUSB
- VIDEO port of the drawer

Mesca PCI-e configuration
Embedded Slot Add-in
ESI
/PCi-e1 X Card
4X Kawela
4X
4X
Legacy HP 16X
4X
Boxboro-MC
IOH0 4X
4X
ConnectX HP 8X
4X
4X
8X
4X
Block B2
ESI 4X
/PCi-e1
4X Kawela
4X
4X
Boxboro-MC
16X
4X
IOH1
4X
4X
8X
4X
4X
8X
4X
Block B1
1x = 10Gb/s bidir

Chipset use in Mesca IO Board
Flash- 1 or 2 1 or 2 Full
• Dual Boxboro solution (/4 sockets)
ROM Full width width QPI
– PCI-e configurations :
QPI
• x16 + x8 + x8 + x4 + legacy SPI
(ICH10)
Boxboro I²C
• x16 + x8 + x8 (3U systems ICH10
x4 ESI
Boxboro (3U syst. only) JTAG
only)
– x16 slots for cluster interconnect,
storage and local/remote GPU (tbc)
– x8 for storage and 10G ethernet x4 x8 x8 x16 x4 x8 x8 x16
• Optional SAS RAID controller USB
PCI-
for embedded HDDs e
• Alternative : direct connection
of SATA HDDs to ICH10
– x4 for embedded Kawela for dual LPC Kawela
1Gb E-net
• Third party management chip Baseboard SMBUS
– nuvoTon selected Management
controller
1 GbE
RMII IB HCA
USB device Kawela
– Functions include redirection
ConnectX
• Management function (BMC)
Keyboard/
• USB device redirection Video/Mouse
• KVM redirection / KVM over over IP 1 GbE IB x4
IP
– Smart connection to Network adapter nuvoTon 10/100 Mb Eth
Video in
(DVI) rev3
South Bridge ICH10 Overview
• Features • Features
– 6 PCI Express x1 ports – SMBus 2.0 / SMLINK
Boxboro
– Integrated 10/100/1000 GbE – Up to 4 PCI 2.3 Masters
MAC GbE PHY – Flexible GPIO usage
• Optimized GbE & x4 ESI CLink – VT-d virtualized IO support
WLAN i/f LCI
– 6 SATA ports (AHCI, 3 Gb/s) GLCI
– 12 High-Speed USB 2.0 Ports
– High Definition Audio – System Management
– Serial Peripheral Interface PCIe x1 (6 Lanes)
(SPI) ICH10 – Storage e-SATA port multiplier
– Legacy BIOS support – Quiet System Technology
– Interrupt controllers SPI LPC • Fan Speed Control
– Power Management
SPI PCIe x1 • ACPI 3.0
Flash WLAN – Halogen free package chip
CLink
TPM1.2
Super I/O
BMC
CLink:controller Link

ICH10 connection in a Mesca Module
QPI Processor
Processor SPI
In a slave
module this IOH
In a multi-
North SPI FLash

is SLAVE
module partition,
an ICH10 in a
QPI BCS
BCS QPI
slave module
does NOT have IOH
IOH00 IOH
IOH11
the "legacy"
attribute XCSI
to remote modules
ESI PCIe PCIe
SPI
Enterprise
KAWELA
KAWELA Ethernet
ICH10 LAN
Packet routing for web access to local BMC
South SPI FLash
RMII RAM
USB
SATA
FLASH
LPC Private
VRAM
iBMC Manageme
Super nt LAN
SuperI/O
I/O
SATA
IPMB (I2C)
Disks Ethernet to remote BMC

ICH10 package

BOxboro package

ILB : Three configurations
IOH0 IOH1
x8 x8 x8 x8 x4 x8 x8 x8 x8 x4
data + cmd data + cmd data + cmd data + cmd data + cmd data + cmd data + cmd data + cmd
1a 1b
3a 3b
2a 2b
data
cmd
Con-
Kawela Kawela
nectX
Ext.
Ext.
Connector Connector
for cable for cable
Slot x16
Slot x16
to PCI-e boards to PCI-e boards
in x16 slot for in x16 slot for
bifurcable bifurcable
extension extension
Slot x8
Slot x8
Slot x8
Slot x8
to 2 times x8 to 2 times x8
- IOH0 : 3 possibilities - IOH1 :

- x16 bif. + x8 +x8 + Kawela - x16 bif. + x8 +x8 + Kawela
- x16 bif.+ x8 +ConnectX + Kawela
- x8 + x8 +x8 +ConnectX + Kawela
PCIe Interconnect Topology: ILB_A/B/C
Neither used nor
populated in config B
G2 G1
Used as x8 in config C
ILB-002
G4 G3
ICH10
Slot #2
Slot #5
QPI1 QPI1
QPI0
Slot #1
Slot #0
Slot #4
Slot #3
QPI0
IOH #1 IOH #0
Kawela Kawela
PCIe #1 #0 PCIe
Ch. B A ConnectX
Ch. B A IB
Slot #3
Slot #0
Slot #4
Slot #1
Slot #5
Slot #2
Ch. A B
Rear DIAG USB eth #3 #2 eth #1 #0 IB #0 #1

ETH MNG Absent in config A

ILB, C configuration
• ILB: C configuration, features :
• 1 ICH10
• 2 IOH
• 1 iBMC
• 6 slots PCIe (5 slots x8 and 1 slot x16)
• 1 internal connector for x4 SATA
• 1 internal connector for x2 SATA
• 2 IB ports
• 4 1GbE ports
• 1 ethernet ports 100Mbps reserved
for administration
• 2 USB ports
• 1 VGA port
Hot-plug

Servicing the ILB (summary)
Remove the MXB
De-populate the ILB
Remove the PCIe cards
Remove the ILB
Install the new ILB

....
Update the Firmware of the iBMC ( FPGA and Bios)
Restore Configuration Data

kiratool -l target@IP -u super -p pass cfg restore <filename>
<filename> is the backup file name from which the configuration

data is restored.
Or
ipmitool -H target@IP -U super -P pass bulloem restorecfg <foldername> config|pef|system
(use ipmitool with EMM 11.09 or above)
Rewrite some FRUs with Fructose: System Serial Number / Chassis serial number / Product name /Model
Check the QPI links speed (with the EMM hidden page)
Check the QPI links health (with the tool CheckLink)

SATA Connections
• SATA links are handled by the

ICH chip embeded on ILB.
6 SATA links are available, to
connect 6 SATA disks :
4 disks in the first HDX,
2 disks in the second HDX
( optional).
A SAS/SATA cable « SFF-8087 to
SFF-8484 » connects the SATA
connector of ILB to the
SAS/SATA connector on HDX
An EFI drivers exists for ICH chip, but without RAID support.
The ICH EFI driver is integrated in the system Bios.
SATA disks are usable for booting, but without the support of the RAID function.

Servicing the HDX (summary)
Unscrew the black M3 captive screw (A) to

the left of the HDX to be serviced
Allow the related HDX cables to extend

through the free opening at the front of
the server drawer.
Label Cable
A ILB-HDX SATA Cable
B PDB-HDX 5V Cable
C ILB-HDX Signal Cable
Connect the cables to the back of the HDX
Push the HDX inside the Server Drawer until

it stops.
Tighten the black M3 captive screw to
secure the HDX in position
Block D : Management (ILB/MNG)
DVO wpcm450 LEDs / buttons
JTAG global (I/O ports under BMC sw control)

USB2.0
vdram
LPC Other GPIOs
DVI receiver iBMC I2C (x5)
ethernet
DVI φ sdram flash mux
LPC SOL 2x switches Front panel

K100 connector
PCA9245a
LAN Super IO ARM ICE
DVI transmitter RJ4 LPC47B270 IDENT
(TMDS) 5 REAR DIAG LED
AD7414 (blue)
Debug -Intrusion
host K100 -ICMB
JTAG
ILB/MNG Serial Port 1 | Serial Port 2
-JTAG test
ext-test HDB0
(I2C)
RMII
(I2C) Service
PDB signals
SATA HDB1
LED
DVO
KAWELA 6 x I2C
FPGA
(SMBus) +5VSB
ES1000 toward ILBC
(I2C)
the rest of
+3.3VS
the
ILB/LEG ICH platform B

ILB/COM
ILB/IOLE ILB/TOP(*)
(*) +3.3V for TMDS and +12V for LCP screen are required from TOP toward MNG

PDBU – Power DISTRIBUTION
- Functions
- Distribute 12V and 3.3V Stand-By generated by
AC/DC block
- Provides power supply for Processors
- Handle UltraCapa module
- Interfaces
- 12V and 3v3STBY to ILB and MXB
- 12V to HDB

Power Distribution
PO W ER SU PPLY AR C H
H D B
12V
M TB 12V 12V
H D B
12V
3 .3 V S ta n d B y
1 2 V S IB P W R
PDB 12V
12V S IB 12V
1 2 V S IB P W R 12V
12V 12V
12V
IL B
G IG 3 .3 V S ta n d B y
3 .3 V S ta n d B y
12V
3 .3 V S ta n d B y
12V
12V
A C /D C A C /D C A C /D C

Servicing the PDB (summary)
The procedure consists in the replacement of the server drawer

(including the PDB, the anti-intrusion switch and internal cables)

Remove all the PSUs.
Remove all the deflectors
Disconnect all the cables
Remove the Fan Box.
Remove the DSIB.
Remove the MXB.
Remove the ILB.
Remove the LCP.
Remove the Hard Disk Box
Install the above compents in the new server drawer

in the reverse order

Power Supply Unit (PSU)
There are 1+1 Power Supply Units .
Power Supply 1600 W - 1U

The power supply is a two output power supply including 12V and
a 3.3V stand-by
A/C INPUT of 90 to 264V A/C.
The power supply is hot swappable, 1+1 redundant;
Active current sharing on main output
Over-voltage, over-current, over-temperature protection
Internal cooling fans

Servicing the PSUs
The PSU is Hot-pluggable
Locate the PSU to be serviced.
0 1 2
Loosen the captive screw (A) located on the left of
the PSU to be serviced.
Pull the PSU carefully out of its slot using the black
handle.

LCP
The LCD module displays
error/status messages.
This also provides the

power and reset
buttons for the system.
The front panel is
controlled by the BMC
Key Features:
LED identification
• LED power
• Button POWER
• LCD- screen
LCP block diagram

LCP con't
FRONT VIEW OF LCP

LCP con't
LED indicators on Front Panel Board:
UID LED : The UID LED indicator is blue.

This indicator is a unit identifier dedicated to the maintenance operations to
localize physically a drawer from the front side. This LED is turned on
simultaneously with the UID LED indicator for the backside.
The switching on/off of this indicator is driven by the maintenance operator
through the SHC or locally through the UID button of the LCP.
Power LED : The power LED indicator is green.

This indicator indicates the 12V main power presence within the drawer. This
indicator is managed by the BMC firmware.

Servicing the LCP (summary)
The LCP is not hot-replacable
Grasp either side of the LCP and carefully pull

forward to release it from its housing in the server
drawer front panel.
Disconnect the dedicated cable from the LCP.
Pass the dedicated cable through the opening in the

server drawer front panel

PCIe Boards
Fibre Channel PCIe boards Gigabit ethernet PCIe boards
Emulex LPe11000, LPe11002, LPe1150 PCI-E 10GbE 2ports

LPe11000, Lpe1150 PCI-E 10GbE 4ports
PCIe x4, low profile Intel PRO/1000 PT Single Port Gigabit PCI-e
Single port FC 4Gb/s Intel PRO/1000 PT Dual Port Gigabit PCI-e
Intel PRO/1000 PT Quad Port Gigabit PCI-e
LPe11002
PCIe x4, low profile
Ethernet 10Gb
Dual port FC 4Gb/s
Emulex LPe12002-M8 Myricom Dual ports 10G-PCIE2-8B2L-2S
PCIe x8, low profile
Dual port FC 8Gb/s SAS raid PCIe boards
Emulex LPe1250
PCIe x8, low profile LSI MegaRAID SAS 9261-8i
SAS RAID0,1,5,10,50,60
Single port FC 8Gb/s
controller 512 MB cache

Mesca 4S3U Module Drawer Rear View
Restore default (1) Reset BMC
IOH #1 - PCIe IOH #0 - PCIe
ConnectX
Slot #3 Slot #2
Kawela #0
Slot #4 Slot #1
Kawela #1
Slot #5 Slot #0
PSU#0
PSU#0 PSU#1
PSU#1
Connectors: VGA USB0 USB1 Mng1 Eth3 Eth2 Eth1 Eth0/Mng0
(1) :default password = admin ; Check QPI speed with the Hidden page in SHC
Rear Connections

Mesca Numbering : Depopulated Module
• When a module contains less than 4 processors,

some rules must be observed
- IOH#0, because of its attached ICH10, must be accessible
• IOH#0 is only connected to processors with physical numbers P0

and P1  P0 or P1 must be present
• IOH#1 is connected to processors with physical numbers P2 and
P3  P2 or P3 must be present if devices attached to IOH#1 are
needed in the configuration

Mesca Numbering : QPI Interconnected Units
CSI/QPI links Link bandwith CSI/QPI links Link bandwith

0 Processor 0 and 1 6 processor 0 and IOH 0 (or BCS)
4 Processor 1 and 3 0xA BCS and IOH 0 (if relevant)

5 Processor 2 and 3 0xB BCS and IOH 1 (if relevant)

QPI Topology 4S3U : NHM-EX QPI port #
N e h a le m -E X p r o c e s s o r L in k # P o rt #
0 2 3 0 2 3
P0 P3 P0 P3
2 0 2 0
1 3 1 1 2 1 3 1 1 2
4 4
6 0 9 6 0 5 9
5
3 0 3 0 3 0 3 0
2 3 1 2 3 1
P1 P2 P1 P2
7 8 1 7 8 2
1 2
0 1 2 3
1 0 1 0
IO H # 0 IO H # 1 IO H # 0 4 BCS 5 IO H # 1
0 1 0 A B 1
IC H 1 0 IC H 1 0
4 s o c k e t c o n fig u r a tio n > 4 s o c k e t c o n fig u r a tio n

No BCS W ith B C S
re v 2

Mesca-X 4S3U: Processor to Processor QPI Links
- W ith in e a c h M e s c a m o d u le ( 4 S 3 U d r a w e r )
- E a c h N e h a le m - E X p r o c e s s o r P i h a s 4 Q P I p o r t s n u m b e r e d P i –
Q P I0 /1 /2 /3
- E a c h B o x b o ro IO H i h a s 2 Q P I p o rts n u m b e re d IO H i – Q P I0 /1
- E a c h B C S h a s 6 Q P I p o rts n u m b e re d B C S – Q P I0 /1 /2 /3 /4 /5
L o g ic a l Q P I# P ro c e s s o r - Q P I # P ro c e s s o r - Q P I # C o m m e n ts
Q P0 P 0 – Q P I3 P 1 – Q P I3 L a n e re v e rs a l
Q P2 P 0 – Q P I0 P 3 – Q P I3
Q P3 P 1 – Q P I2 P 2 – Q P I1

Mesca-X 4S3U: Processor to IOH and BCS Links
- P r o c e s s o r - I O H c o n n e c t io n d e p e n d s o n B C S p r e s e n c e in t h e
m o d u le
L o g ic a l Q P I# IO H /B C S - Q P I # P ro c e s s o r - Q P I # C o m m e n ts
Q P6 IO H 0 – Q P I1 P 0 – Q P I1 D S IB , n o B C S – la n e r e v e r s a l
Q P7 IO H 0 – Q P I0 P 1 – Q P I1 D S IB , n o B C S
Q P8 IO H 1 – Q P I1 P 2 – Q P I2 D S IB , n o B C S
Q P9 IO H 1 – Q P I0 P 3 – Q P I2 D S IB , n o B C S – la n e r e v e r s a l
Q P6 B C S – Q P I0 P 0 – Q P I1 S IB w ith B C S – la n e r e v e r s a l
Q P7 B C S – Q P I1 P 1 – Q P I1 S IB w ith B C S
Q P8 B C S – Q P I2 P 2 – Q P I2 S IB w ith B C S
Q P9 B C S – Q P I3 P 3 – Q P I2 S IB w ith B C S – la n e r e v e r s a l
Q PA B C S – Q P I4 IO H 0 – Q P I0 S IB w ith B C S
Q PB B C S – Q P I5 IO H 1 – Q P I1 S IB w ith B C S
IO H 0 – Q P I1 N o t c o n n e c te d S IB w ith B C S
IO H 1 – Q P I0 N o t c o n n e c te d S IB w ith B C S

Spare Parts Number
Description Part number MI/PI Remarks
11001266-000 DSIB100-0000
11001629A-000 ILBA300-0000 ILB-A V4
11008109C-000
21005177-000 PSUS400-1540
Depopulated drawer + PDBU (Power Distribution Board) +

power/disks cables
MC000623-000
21005343-000 Myricom 10G-PCIE2-8B2-2S+E (specific Freudenberg)
21005373-000 Transceiver for 21005343-000
21005429-000 CKTU401-EX1M Intel PRO/1000 PT Single Port Gigabit Copper (PCI-Express) #868970
CKTU400-0060 Adaptec RAID SAS ADAPTER 5805-SLG (BBU not include with
spare)
F/C 4Gb/s 1 port LPe1150-F4 PCI-e HBA 21005431-000 CKTU301-EX4M Emulex PCIe LPe1150-F4
F/C 4Gb/s 2 port LPe 11002-M4 PCI-e HBA 21005435 -000 CKTU302-EX4D Emulex PCIe LPe11002-M4
21005071-000 CKTH304-EX8D Emulex LPe12002-M8
11016428-000
KITPROCX7560-000 CPKU410-7560 NEHALEM-EX X7560 24MB 8c 2.26GHz 6.4GT/s 130W KIT

CPU+Heatsink
KITPROCL7555-000 CPKU410-7555 NEHALEM-EX LV7555 24MB 8c 1.86GHz 5.86GT/s 95W KIT

CPU+Heatsink
KITPROCL7545-000 CPKU410-7545 NEHALEM-EX HC LV L7545(1.86GHz/18MB/5,86GT/s)+heatsink
KITPROCE7530-000 CPKU410-7530 NEHALEM-EX HC E7530 (1.86GHz/12MB/5,8GT/s)+heatsink
KITPROCE7520-000 CPKU410-7520 NEHALEM-EX QC E7520 (1.86GHz/18MB/4,8GT/s)+heatsink

Spare Parts Number (con't)
Description Part number MI/PI Remarks
KITPROCL7545-000 CPKU410-7545 NEHALEM-EX HC LV L7545(1.86GHz/18MB/5,86GT/s)
+heatsink
set of 2 DDR3 SDRAM 1066MHz 2GB Micron 22005327M12-000 CMMU600-0004 Ref Micron : MT18JSF25672PDZ-1G1F1
set of 2 DDR3 SDRAM 1066MHz 2GB Samsung 22005327S12-000 CMMU600-0004 Ref Samsung : M393B5673EH1-CF8
set of 2 DDR3 SDRAM 1066MHz 4GB Micron 22006326M12-000 CMMU600-0008 Ref Micron : MT36JSZF51272PY-1G1D2 or
MT36JSZF51272PZ-1G1F1
Set of 2 DDR3 SDRAM 1066MHz 8GB QR 21005415S11-000 CMMU700-0016 Ref Samsung : M393B1K73CH0-CF8
Samsung
21005050-000
11002062-000
2.5" 500GB SATA2 7.2Krpm 14.8mm 21005233-000 MSUU303-0072 Seagate Constellation ST9500530NS
2.5" 73GB SAS 15Krpm 21005411-000 MSUU410-0015 Seagate ST973452SS
11002016-000 DRWU700-0000
11016561-000
PDU (7 outlet x C13) - 2.5m cable w/ 32A PDUH006-M700 PDUH006-M700 ATOS 6901758
21005271-000 PDUH010-M160 APC AP7922
11028700-000 TYCO removal tool RapidLock for power cables
11015966-000 VGA Internal Cable Adapter for 4S3U
11019685-000 SERIAL Internal Cable Adapter

Documentation
- novascale bullion Installation Guide 86A126FE04

- novascale bullion Guide d'Installation 86F126FE04
- novascale bullion Service Guide 86A727FE03
- novascale bullion Hardware Console User's Guide 86A148FE03
- novascale bullion Guide Utilisateur de la Console Matérielle 86F148FE03
- Site Preparation Guide 86A140FA04
- iCare Console User Guide 86A171FA06

1 Novascale Bullion - Hardware

Caricato da

Informazioni sul documento

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

1 Novascale Bullion - Hardware

Caricato da

Copyright:

Formati disponibili

Mesca-X Training - Hardware

C.Vangramberen – 2011 May

2 ©Bull, 2010 © Bull Confidentiel - Reproduction interdite sans accord préalable

- A technology disruption in Intel processors

Frontside vs. Backside

A new server platform architecture is required

Last Level Cache L3 L3

Core/System Interface Switch

Mem Mem Mem Mem

QPI Links 4 SMI mem QPI Links 4 SMI mem

Nehalem-EX CPU Tukwila-MC CPU

Xeon Processor Itanium Processor

4 ©Bull, 2010 © Bull Confidentiel - Reproduction interdite sans accord préalable

5 ©Bull, 2010 © Bull Confidentiel - Reproduction interdite sans accord préalable

6 ©Bull, 2010 © Bull Confidentiel - Reproduction interdite sans accord préalable

7 ©Bull, 2010 © Bull Confidentiel - Reproduction interdite sans accord préalable

8 ©Bull, 2010 © Bull Confidentiel - Reproduction interdite sans accord préalable

9 ©Bull, 2010 © Bull Confidentiel - Reproduction interdite sans accord préalable

10 ©Bull, 2010 © Bull Confidentiel - Reproduction interdite sans accord préalable

Hot swap fans

HDX MXB ILB

12 ©Bull, 2010 © Bull Confidentiel - Reproduction interdite sans accord préalable

13 ©Bull, 2010 © Bull Confidentiel - Reproduction interdite sans accord préalable

DSIB Block B2 PCI ES1000

14 ©Bull, 2010 © Bull Confidentiel - Reproduction interdite sans accord préalable

There are 4 x 2 redundant, hot-

15 ©Bull, 2010 © Bull Confidentiel - Reproduction interdite sans accord préalable

extract the fan box by pushing it

PCI KAWELA PCI KAWELA

CSI2 CSI3 CSI2 CSI3 CSI2 CSI3 CSI2 CSI3

17 ©Bull, 2010 © Bull Confidentiel - Reproduction interdite sans accord préalable

Place the Server Drawer in the Servicing Position

Carefully pull the board out of the server drawer.

Remove the DSIB.

Important The DSIB board is secured

18 ©Bull, 2010 © Bull Confidentiel - Reproduction interdite sans accord préalable

QPI full width links

19 ©Bull, 2010 © Bull Confidentiel - Reproduction interdite sans accord préalable

20 ©Bull, 2010 © Bull Confidentiel - Reproduction interdite sans accord préalable

21 ©Bull, 2010 © Bull Confidentiel - Reproduction interdite sans accord préalable

- Each NEHALEM socket hosts 2 independant

supported DIMMs: 2GB, 4GB, 8GB, 16GB

22 ©Bull, 2010 © Bull Confidentiel - Reproduction interdite sans accord préalable

23 ©Bull, 2010 © Bull Confidentiel - Reproduction interdite sans accord préalable

DIMM Pair IDs 3 1 7 5 11 9 15 13

Place the Server Drawer in the Servicing Position

Remove the Processor:

25 ©Bull, 2010 © Bull Confidentiel - Reproduction interdite sans accord préalable

. Hold the processor on top and bottom

26 ©Bull, 2010 © Bull Confidentiel - Reproduction interdite sans accord préalable

. Lower the processor onto the socket

27 ©Bull, 2010 © Bull Confidentiel - Reproduction interdite sans accord préalable

. Place the heat sink onto the processor

28 ©Bull, 2010 © Bull Confidentiel - Reproduction interdite sans accord préalable

29 ©Bull, 2010 © Bull Confidentiel - Reproduction interdite sans accord préalable

• 1 ESI (aka DMI) for ICH10 attachment

ESI:Enterprise South Bridge Interface DMI:Direct Media Interface

30 ©Bull, 2010 © Bull Confidentiel - Reproduction interdite sans accord préalable

31 ©Bull, 2010 © Bull Confidentiel - Reproduction interdite sans accord préalable

32 ©Bull, 2010 © Bull Confidentiel - Reproduction interdite sans accord préalable

34 ©Bull, 2010 © Bull Confidentiel - Reproduction interdite sans accord préalable

North SPI FLash

35 ©Bull, 2010 © Bull Confidentiel - Reproduction interdite sans accord préalable