Linux on IBM System z9 and zSeries

System Hardware Overview

Linux on System z Workshop

New Technology Center
IBM France – Montpellier PSSC

© 2006 IBM Corporation IBM Systems

Press Conference

7th April 1964, Poughkeepsie NY

• A new generation of electronic computing equipment was introduced

today by International Business Machines Corporation. IBM Board
Chairman Thomas J. Watson Jr. called the event the most important
product announcement in the company's history.

• The new equipment is known as the IBM System/360.

• "System/360 represents a sharp departure from concepts of the past

in designing and building computers. It is the product of an
international effort in IBM's laboratories and plants and is the first
time IBM has redesigned the basic internal architecture of its
computers in a decade. The result will be more computer
productivity at lower cost than ever before. This is the beginning of a
new generation - - not only of computers - - but of their application in
business, science and government."

2 IBM System z Hardware 08/11/2007 IBM Systems

1
 1990 – The Mainframe is dead – or so they said

S/390 is dead or terminally ill
ƒthe aging user base is deserting the MVS ship and moving to UNIX or NT

S/390 is expensive to buy and expensive to run
ƒUNIX or Wintel is many times cheaper

S/390 requires a huge Datacenter/Glasshouse
ƒbig machines, water cooling and huge electricity bills

S/390 cannot run new eBusiness or ERP workloads
ƒit only runs Batch or Green-screen type work

Water Cooling equipment

as used 1970s to 1990s
9021-9X2 - 470 MIPS in 1994

.... but that was before IBM "Downsized the Mainframe"

3 IBM System z Hardware 08/11/2007 IBM Systems

The technology change …

CMOS significantly reduces cost, size & running costs

ƒSMP goes to 10-way, 12-way and 16-way systems
ƒParallel Sysplex clustering introduced in 1994 to enable shared data

1996 - fully integrated UNIX environment on MVS/ESA
ƒXPG4 and UNIX/95 branding - enables new workloads

Over time support is added for eBusiness workloads
ƒIEEE Floating point and Integrated Crypto engines - good performance
ƒOSA and Gb Ethernet support - full conectivity to Internet

1999 adds FICON channels - Fibre Channel stage-II
ƒhigher speed, longer distance - more D/R options, connection to 'Open Systems I/O'

2000 - z900 and z/OS - new platform support for eBusiness
ƒSharing systems resources in line with business goals

2000 introduces Linux on zSeries
ƒNew options for fast start eBusiness applications
ƒIntegration with z/OS via HiperSockets

Bipolar, Water Cooled Mainframes transition to smaller CMOS technology

4 IBM System z Hardware 08/11/2007 IBM Systems

2
 IBM System z9 and eServer zSeries
The incorporation of classical mainframe strengths

Reliability, Availability, Serviceability

Security (cryptographic HW and more ...)

Scalability:

Scale up on demand

Scale out on demand

Based on:
ƒ Architecture

64bit, QDIO, ...
ƒ Technology, Packaging

MultiChipModule, ...
ƒ Implementation

Internal and external bandwidth

'Storage Hierarchy': L1, L2,...
ƒ Virtualization: 'sharing physical resources'

CPU, memory, ...

I/O, network, (LAN)
► Most advanced On Demand Technologies

Redundant resources, hot-pluggable IO, …, OOCoD

5 IBM System z Hardware 08/11/2007 IBM Systems

Short history

1998-06-23 9672-nn6 G5

1999-05-03 9672-nn7 G6

2000-10-03 2064-1nn z900

2002-02-19 2066 z800
(Linux only model 2002-01-29)

2002-04-30 2064-2Cn z900 Turbo

2003-05-13 2084 z990

2004-04-07 2086 z890

2005-07-26 2094 System z9 EC

2006-04-27 2096 System z9 BC

6 IBM System z Hardware 08/11/2007 IBM Systems

3
 IBM System z9
IBM System z family IBM System z9
(z9 BC) (2096)
(z9 EC) (2094)

IBM eServer zSeries 890

IBM eServer zSeries 990 z890 (2086)
z990 (2084)

Announced 4/06 - Superscalar Server with 8

PUs

Announced 7/05 - Superscalar Server with
up to 64 PUs
2 models – Up to 4-way

5 models – Up to 54-way
High levels of Granularity available
► 73 Capacity Indicators

Granular Offerings for up to 8 CPs

Specialty Engines

Announced 4/04 – zSeries Superscalar
Specialty Engines ► CP, IFL, ICF, zAAP, zIIP

Announced 5/03 – first zSeries Superscalar Server with 5 PUs ► CP, IFL, ICF, zAAP, zIIP
Server with up to 48 PUs
On Demand Capabilities

1 model – Up to 4-way
On Demand Capabilities ► CUoD, CIU, CBU, On/Off CoD

4 models – Up to 32-way ► 28 capacity settings ► CUoD, CIU, CBU, On/Off CoD

Memory – up to 64 GB

Specialty Engines
Specialty Engines
Memory – up to 512 GB
CP, IFL, ICF, zAAP
Channels
► ► CP, IFL, ICF, zAAP
Channels ► Two LCSSs

On Demand Capabilities
On Demand Capabilities ► Four LCSSs ► Multiple Subchannel Sets
► CUoD, CIU, CBU, On/Off CoD ► CUoD, CIU, CBU, On/Off CoD ► Multiple Subchannel Sets ► MIDAW facility
► MIDAW facility 63.75 subchannels

Memory – up to 256 GB
Memory – up to 32 GB ►
► 63.75 subchannels ► Up to 420 ESCON channels

Channels
Channel ► Up to 1024 ESCON channels ► Up to 112 FICON channels
► Four LCSSs ► Two LCSSs ► Up to 336 FICON channels ► Enhanced FICON Express2 4 Gbps
► Up to 1024 ESCON® channels ► Up to 420 ESCON channels ► Enhanced FICON Express2 and 4 ► 10 GbE, GbE, 1000BASE-T
► Up to 240 FICON Express2 channels ► Up to 80 FICON Express2 channels ► 10 GbE, GbE, 1000BASE-T ► Coupling Links
► Token-Ring, GbE, 1000BASE-T Ethernet ► Networking Adapters (OSA) ► Coupling Links
Coupling Links
Configurable Crypto Express2
► ► Coupling Links
Configurable Crypto Express2

Crypto Express2
Parallel Sysplex clustering

Crypto Express2
Parallel Sysplex clustering

Parallel Sysplex clustering
Parallel Sysplex clustering
HiperSockets – up to 16

HiperSockets – up to 16

HiperSockets™ – up to 16
HiperSockets – up to 16
Up to 30 logical partitions

Up to 60 logical partitions

Up to 30 logical partitions
Up to 30 logical partitions
Enhanced Availability

Enhanced Availability

Operating Systems
Operating Systems
Operating Systems

Operating Systems ► z/OS, z/OS.ez/VM, VSE/ESA, z/VSE, TPF,
► z/OS, z/VM®, VSE/ESA™, z/VSE™, TPF, ► z/OS, z/OS.e, z/VM, VSE/ESA, z/VSE, ► z/OS, z/VM, VSE/ESA, z/VSE, TPF,
z/TPF, Linux® on zSeries z/TPF, Linux on System z9
TPF, z/TPF, Linux on zSeries z/TPF, Linux on System z9

7 IBM System z Hardware 08/11/2007 IBM Systems

Mainframes Continues to Scale with System z9 64 engines

54-way

Each new range is designed to deliver:
► New function
► Improved performance
► Improved availability and serviceability
► Field upgrades to help protect customer investment
► Better price/performance 48 engines
32-way

20 engines
16-way

Maximum ITRs z990 z9 EC

z900
Minimum ITRs z/OS 1.4 z/OS 1.4 z/OS 1.6
8 IBM System z Hardware 08/11/2007 IBM Systems

4
z9 BC – Delivering increased capacity and performance

Greater granularity and scalability
► Two models with one machine type (2096)

Improved I/O Performance
● 1 to 4-way high performance server standard engines ► 40% more FICON channels – up to 112
● Entry model with 1 to 3-way standard engines ► Up to 170% more bandwidth than z890
● Up to a 7-way with specialty engines
► Can improve FICON performance with
► 73 capacity settings for a 260% increase in flexibility over
Modified Indirect Data Address Word
z890
► Delivers up to 34% more capacity with the same low entry
(MIDAW) facility
point as the z890 ► Double the FICON concurrent I/O
► Up to 34% hardware performance improvement for Linux operations from 32 to 64 on FICON channel
(IFLs), Java (zAAPs) and Internal Coupling Facilities (ICFs) ► Multiple Subchannel Sets (MSS) for an
► New zIIP for data serving workloads increased number of logical volumes
► Double the memory – up to 64 GB per server
Processor Units

z9 BC
75% more

z800 z890

34% improvement
Capacity
9 IBM System z Hardware 08/11/2007 IBM Systems

z9 EC Product Positioning

z9 EC can provide increased capacity in a single footprint

► Faster uniprocessor performance than the z990
► Additionalengines
► Increased capacity on specialty engines for consolidation

On demand advancements for unplanned and planned outage reduction

May boost performance of new workloads and security

z9 EC

z990
z900

MIPS capacity in a single footprint

10 IBM System z Hardware 08/11/2007 IBM Systems

5
z9 BC Model R07

Machine Type
► 2096

Model
► R07

Processor Units (PUs)
►8 PUs per System
►1 SAP per book, standard
► No dedicated spares
► 7 PUs available for characterization
● 1 to 3 Central Processors (CPs)
● Integrated Facility for Linux (IFLs), Internal Coupling
Facility (ICFs), System z Application Assist
Processors (zAAPs), System z9 Integrated
Information Processors (zIIPs), optional System
Assist Processors (SAPs)
► Up to 15 LPARs

Memory
► Minimum of 8 GB
► Up to 64 GB per System
● 8 GB increments

I/O
► Up to 16 STIs per System @ 2.7 GB/s each
► Total system I/O bandwidth capability of 43.2 GB
► Up to 2 Logical Channel Subsystem (LCSS)
► Up to a maximum of 4 I/O Domains
● Up to 240 channels – dependent on Channel types

11 IBM System z Hardware 08/11/2007 IBM Systems

z9 BC Model S07

Machine Type
► 2097

Model
► S07

Processor Units (PUs)
►8 PUs per System
►1 SAP per book, standard
► No dedicated spares
► 7 PUs available for characterization
● 0 to 4 Central Processors (CPs)
● Integrated Facility for Linux (IFLs), Internal Coupling Facility
(ICFs), System z Application Assist Processors (zAAPs),
System z9 Integrated Information Processor (zIIP), optional
System Assist Processors (SAPs)
● Can have an IFL-only System
► Up to 30 LPARs

Memory
► Minimum of 8 GB
► Up to 64 GB per System
● 8 GB increments

I/O
► Up to 16 STIs per System @ 2.7 GB/s each
► Totalsystem I/O bandwidth capability of 43.2 GB
► Up to 2 Logical Channel Subsystems (LCSSs)
► Up to a maximum of 7 I/O Domains
● Up to 420 channels – dependent on Channel types

12 IBM System z Hardware 08/11/2007 IBM Systems

6
z9 EC Overview

Machine Type
► 2094

5 Models
► S08, S18, S28, S38 and S54

Processor Units (PUs)
► 12 (16 for Model S54) PUs per book
►2 SAPs per book, standard
► 2 spares per server
► 8, 18, 28, 38 or 54 PUs available for characterization
● Central Processors (CPs), Integrated Facility for Linux
(IFLs), Internal Coupling Facility (ICFs), System z
Application Assist Processors (zAAPs), System z9
Integrated Information Processor (zIIP), optional System
Assist Processors (SAPs)

Memory
► Minimum of 16 GB
► Up to 128 GB per book
► Up to 512 GB for System
● 16 GB increments

I/O
► Up to 16 STIs per book @ 2.7 GB/s each
► Total system I/O bandwidth capability of 172.8 GB*
► Up to 4 Logical Channel Subsystems (LCSSs)

* z9 EC exploits a subset of its designed I/O capability

13 IBM System z Hardware 08/11/2007 IBM Systems

z9 BC – Under the covers

A Frame

Internal
Batteries Single Processor
(optional) Book and Memory

Power
Supplies CEC Cage

STI Connectors

I/O Cage

Support Elements
(gate with Laptops
Fiber Quick swung open)
Connect Feature
(optional)
Front View

14 IBM System z Hardware 08/11/2007 IBM Systems

7
 z9 EC – Under the covers (Model S38 or S54)

Internal
Batteries
(optional)
Hybrid
Cooling

Power
Processor Books
Supplies
and Memory

CEC Cage

3x I/O STI cables

cages

Support
Fiber Quick Elements
Connect Feature
(optional) Front View

15 IBM System z Hardware 08/11/2007 IBM Systems

CMOS 10Ks0 chip Technology

z9 BC 8-way MCM ► PU, SC, SD and MSC chips
► Copper interconnections, 10 copper layers
►8 PU chips/MCM

Advanced 95mm x 95mm MCM
► 102 Glass Ceramic layers
●15.78 mm x 11.84 mm
●L1 cache/PU
► 16 chip sites, 217 capacitors
– 256 KB I-cache
► 0.545 km of internal wire
●121 million transistors/chip
– 256 KB D-cache
●0.7 ns Cycle Time
► 4 System Data (SD) cache chips/MCM
●15.66 mm x 15.40mm
●L2 cache per Book
MSC PU PU MSC – 660 million transistors/chip
– 40 MB
► One Storage Control (SC) chip
PU SD SD PU ●16.41mm x 16.41mm
●162 million transistors
●L2 cache crosspoint switch
PU SD SD PU ●L2 access rings to/from other MCMs
► Two Memory Storage Control (MSC) chips
●14.31 mm x 14.31 mm
CLK PU SC PU ●24 million transistors/chip
●Memory cards (L3) interface to L2
●L2 access to/from MBAs (off MCM)
► One Clock (CLK) chip - CMOS 8S
●Clock and ETR Receiver

16 IBM System z Hardware 08/11/2007 IBM Systems

8

CMOS 10Ks0 chip Technology
z9 EC 12-way MCM ► PU, SC, SD and MSC chips
► Copper interconnections, 10 copper layers
► 8 PU chips/MCM

Advanced 95mm x 95mm MCM ●121 million transistors/chip
► 102 Glass Ceramic layers ●L1 cache/PU
► 16 chip sites, 217 capacitors ●15.78 mm x 11.84 mm
► 0.545 km of internal wire – 256 KB I-cache
– 256 KB D-cache
●0.58 ns Cycle Time
► 4 System Data (SD) cache chips/MCM
●15.66 mm x 15.40mm
●L2 cache per Book
MSC PU PU MSC – 660 million transistors/chip
– 40 MB
► One Storage Control (SC) chip
PU SD SD PU ●16.41mm x 16.41mm
●162 million transistors
●L2 cache crosspoint switch
PU SD SD PU ●L2 access rings to/from other MCMs
► Two Memory Storage Control (MSC) chips
●14.31 mm x 14.31 mm
CLK PU SC PU ●24 million transistors/chip
●Memory cards (L3) interface to L2
●L2 access to/from MBAs (off MCM)
► One Clock (CLK) chip - CMOS 8S
●Clock and ETR Receiver

17 IBM System z Hardware 08/11/2007 IBM Systems

z9 BC Processor Book Layout

Up to 8 MCM
Hot pluggable MBA/STI
fanout cards

"F1" "D1" "D2"

MSC CP
Hitac hi CP
Hitac hi

"D3" "D4" "D5" SD SD SC

SD CP 2
SD

MSC CP CP

"D6" "D7" "D8"

Front View Memory Cards up to Side View

64 GB (z9BC)
Note: 1. Concept Illustration only - not to scale 128GB (z9EC)
2. 4 or 8 pluggable Memory Cards
3. Each MBA fanout card is hot-pluggable and has 2 STIs

18 IBM System z Hardware 08/11/2007 IBM Systems

9
z9 BC 8 PU Logical Structure

PU 4 always is SAP,
each PU is single core
PU0 PU1 PU2 PU3 PU4 PU5 PU6 PU7 and Crypto Assist
L1 L1 L1 L1 L1 L1 L1 L1

L2 contains Storage
Controller and Storage
Data Chips
1 Cache Control Chip + 2 Cache Data Chips
(40 MB Shared L2 Cache)

Book Package
contains PU, Storage
Controllers, L2 and L3
MBA MBA MBA MBA MBA MBA MBA MBA MSC 0 MSC 1 Caches, 2.7 GB/sec
Self-Timed Interfaces

Seven of 16 STIs can

STI- Links Up to 64 GB Memory
(L3 Cache) CLK attach to the I/O cage.
2.7 Gbyte/s
Rest can be used for
ICB-4s
Up to 28 I/O Adapters:
ESCON 16 Port
FICON Express2 4 port
FICON Express4 2 or 4
port
OSA-Express2 1 or 2 port ETR-Links
ICB-4 Links
Crypto Express2 2
Coprocessor
2 Gbyte/s Note: STI connections to the I/O cage are NOT
ISC-3 1-4 Port I/O Cage with 7 I/O domains representative of an actual configuration.

19 IBM System z Hardware 08/11/2007 IBM Systems

z9 EC Model S38 – Communication Ring Structure

CEC Board backplane

Book 0 Book 2

P
r MCM MCM MCM MCM
P P P
o Book 3
r r r
Book 1
c o o o
c c c
e e e e
s
s
s
s
s
s
s
s
The ring structure consists of two rings (one
Memory
Memory

Memory
Memory

o
o
r
o
r
o
r running clockwise, the other counterclockwise)
r C
a
C
a
C
a

In a two or three Book configuration, Jumper
C
r r r Book(s) (installed in the CEC cage) complete the
a
d d d
ring
r
► Jumper Books are not needed for a single-Book
Fanout
MBA
Fanout
MBA

Fanout
MBA
Fanout
MBA

d
configuration

Books may be able to be inserted into or removed
from the ring nondisruptively*
Slot 1 Slot 2 Slot 3 Slot 4 ► May allow Concurrent book add for model
Book 3 Book 0 Book 1 Book 2 upgrade
► Enhanced book availability to return a book after
removal for upgrade or repair
* Customer pre-planning required, may require acquisition of additional hardware resources

20 IBM System z Hardware 08/11/2007 IBM Systems

10
 Technology evolution with specialty engines

Building on a strong track record of

technology innovation with specialty
engines, IBM introduces the System
z9 Integrated Information Processor IBM System z9
Integrated Information
Processor (zIIP) 2006

Designed to help
IBM System z
improve resource
Application Assist
Processor (zAAP) optimization for
2004 eligible data
workloads within
Integrated
Designed to help the enterprise
Facility for Linux improve resource
(IFL) 2001 optimization for
z/OS Java™
Internal Coupling
Support for
technology-based
Facility (ICF) 1997 new workloads
workloads
and open

Centralized data standards
sharing across
mainframes
21 IBM System z Hardware 08/11/2007 IBM Systems

System z9 PU Characterization

Processor Units (PUs) that can be ordered on System z9

►Central Processor (CP)

● Provides processing capacity for z/Architecture™ and ESA/390 instruction sets
● Runs z/OS, z/VM, VSE/ESA, z/VSE, TPF4, z/TPF, Linux for System z and Linux under
z/VM or Coupling Facility
● z9 BC has Capacity Marker features NOT Unassigned CP features
►IBM System z Application Assist Processor (zAAP)
● Under z/OS, the Java Virtual Machine (JVM) assists with Java processing to a zAAP
►IBM System z9 Integrated Information Processor (zIIP) – when available
● Provides processing capacity for selected workloads e.g., DB2 for z/OS V8 workloads
executing in SRB mode
►Integrated Facility for Linux (IFL)
● Provides additional processing capacity for Linux workloads
►Internal Coupling Facility (ICF)
● Provides additional processing capacity for the execution of the Coupling Facility Control
Code (CFCC) in a CF LPAR
►Optional System Assist Processors (SAP)
● SAP manages the start and ending of I/O operations for all Logical Partitions and all
attached I/O

22 IBM System z Hardware 08/11/2007 IBM Systems

11
 Input/Output hardware
Device
I/O Device
ESCON Control Device
Channel Unit Device
I/O Device
Control
ESCON Unit
I/O
zSeries/z9 processor

Channel ESCON Control Device

Director Unit
ESCON I/O
Channel Control Device
I/O Unit
Control
Unit Device

Device
FICON Express2 I/O
Channel Control
I/O
FICON Device
Unit
Control
switch I/O Device
FICON Express2 Unit
Control Device
Channel Unit Device
FCP (FICON FCP
Storage Ctrl Unit SCSI Device
channel) switch
SAN Fabric SCSI Device
OSA
Express2
Adapter
LAN
Up to 1024 channels, total
Hipersocket bandwidth > 170GB/sec

Terms: Channel, ESCON, FICON, OSA, Hipersockets

23 IBM System z Hardware 08/11/2007 IBM Systems

Horizontal scaling & high availability within the data center

Coupling

Parallel Sysplex Facility

► Loosely coupled multiprocessing

► Requires: 12
12 11 1
11 1
10 2
10 2

► Hardware/software combination 9
8
7
6
5
3
4
9
8
7
6
5
4
3

● Data sharing IBM System z

Sysplex
Timers
● Locking zSeries
● Cross-system workload dispatching ESCON/FICON*
● Synchronization of time for logging, etc.
● High-speed system coupling
► Hardware:
Shared
● Coupling Facility data
– Integrated Cluster Bus and ISC to provide high-speed
links to CF
● Sysplex Timer – Time Of Day clock synchronization
► Implemented in z/OS* and subsystems
● Workload Manager in z/OS
● Compatibility and exploitation in software subsystems,
including IMS*, VSAM*, RACF*, VTAM*, JES2*, etc.

24 IBM System z Hardware 08/11/2007 IBM Systems

12
Hardware management/maintenance - Support Element and HMC

Two Support Elements (SE) are connected to two internal Ethernet LANs

There is a crossover capability between the LANs, so that both SEs have access to
both internal LANs

With a second Ethernet adapter, the SEs are also connected to an external Ethernet
LAN

The Hardware Management Console (HMC) is connected to the external LAN

FSP = Flexible Support Processor

Bulk power Bulk power
Primary FSP FSP
Support
Element
Ethernet “hub” 1

Ethernet “hub” 2

Alternate
Support
Element FSP FSP FSP
FSP FSP FSP FSP
FSP
DCA DCA DCA DCA
DCA DCA DCA DCA

HMC

I/O cage 2 I/O cage 3

book 0 book 1
external
LAN

25 IBM System z Hardware 08/11/2007 IBM Systems

Operating System

z/OS
► The flagship zSeries/z9 operating system
► Ultra-high reliability – over 50% of the OS is dedicated to error recovery!

z/VM
► Run any zSeries/z9 operating system as a “guest” – Large scale virtualization
► Sometimes used for application development

Linux for IBM System z and zSeries

► A native Unix like operating system for IBM System z and zSeries
► “zLinux” is already trademarked, so we can’t call it that…

z/VSE
► Small- to mid-size 390 customers
► Most often used on very small 390 hardware

z/TPF
► Primarily for very high transaction rate, specialized applications (airline scheduling)

26 IBM System z Hardware 08/11/2007 IBM Systems

13
Mixed workload - Virtualization - Consolidation
even beyond a single footprint, when using clustering
Legacy WebSphere
Database & Business Logic,
Transaction Web Serving, ... Native Linux
Processing DB, Business Logic
ERP Java Appl Web Serving, ...
Java Linux
CRM Appl. Java Virtual
Business Java Appl
Appl.
CICS Objects
Servers
SAPDB2 Business Objects
DB2MQ c++
UnixIMS CICS DB
IMS IMS Java
System Linux for
JVM
services zSeries Linux
Unix System Services
Linux Linux
Linux
for for
z/OS
z/OS z/VM z/VM zSeries zSeries

HiperSockets
PR/SM up to 60 partitions
for integration
100’s and 100’s !! of virtual linux servers – “virtual blades”

IBM System z9
Responsive - Variable - Resilient – Focused
Autonomic - Virtualized - Open - Integrated

27 IBM System z Hardware 08/11/2007 IBM Systems

Summary

From System/360 in 1964 to today’s System z9 and zSeries, we have

seen an evolution that has preserved customer investments in a unique
way

From OS/360 to MVS to OS/390 to z/OS, we have seen an evolution of
the operating system that is core to most corporate IT environments

From CP/67 as a research project and VM/370 as a migration tool, VM
has evolved to today’s z/VM as the core of IBM’s zSeries virtualization
technology

Virtualization is now considered “standard” in the industry and all
virtualization solutions owe much to the VM family

“Legacy systems are systems that work!”

28 IBM System z Hardware 08/11/2007 IBM Systems

14
29 IBM System z Hardware 08/11/2007 IBM Systems

15