Sa 400 RC

Sun Educational Services
Solaris System Performance Management

SA-400
April 2001
Copyright 2001 Sun Microsystems, Inc., 901 San Antonio Road, Palo Alto, California 94303, U.S.A. All rights reserved. This product or document is protected by copyright and distributed under licenses restricting its use, copying, distribution, and decompilation. No part of this product or document may be reproduced in any form by any means without prior written authorization of Sun and its licensors, if any. Third-party software, including font technology, is copyrighted and licensed from Sun suppliers. Parts of the product may be derived from Berkeley BSD systems, licensed from the University of California. UNIX is a registered trademark in the U.S. and other countries, exclusively licensed through X/Open Company, Ltd. Sun, Sun Microsystems, the Sun Logo, CacheFS, Forte, Open Boot, Solaris, Solaris Resource Manager, SunOS, Sun Enterprise SyMON, Sun Trunking, Sun Management Center, Sun Enterprise 10000, and Ultra are trademarks or registered trademarks of Sun Microsystems, Inc. in the U.S. and other countries. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. in the U.S. and other countries. Products bearing SPARC trademarks are based upon an architecture developed by Sun Microsystems, Inc. The OPEN LOOK and Sun Graphical User Interface was developed by Sun Microsystems, Inc. for its users and licensees. Sun acknowledges the pioneering efforts of Xerox in researching and developing the concept of visual or graphical user interfaces for the computer industry. Sun holds a non-exclusive license from Xerox to the Xerox Graphical User Interface, which license also covers Suns licensees who implement OPEN LOOK GUIs and otherwise comply with Suns written license agreements. U.S. Government approval required when exporting the product. RESTRICTED RIGHTS: Use, duplication, or disclosure by the U.S. Government is subject to restrictions of FAR 52.227-14(g) (2)(6/87) and FAR 52.227-19(6/87), or DFAR 252.227-7015 (b)(6/95) and DFAR 227.7202-3(a). DOCUMENTATION IS PROVIDED AS IS AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS, AND WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID.
About This Course
April 2001
Course Goal
This course provides you with the knowledge and skills to: Prepare to tune SunOS and hardware Use available tuning data Set the parameters for Sun SPARC systems This course is intended for experienced system administrators and system designers with a background in Solaris Operating Environment (OE) system administration.

Copyright 2001 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services April 2001, Revision C
About This Course, slide 2 of 15
Course Overview
Give an overview of the operation of various system hardware and software components Examine specific areas of hardware and software components Provide an understanding of what effect a certain tuning change will have and whether it is necessary

Course Map
Overview
Introducing to Performance Management
CPU
Processes and Threads CPU Scheduling
Memory
System Caches Memory Tuning
I/O
System Buses I/O Tuning UFS Tuning
Networks
Network Tuning
Summary
Performance Tuning Summary
Copyright 2001 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, April 2001, Revision C
Module-by-Module Overview
Preface About This Course Module 1 Introducing Performance Management Module 2 Processes and Threads Module 3 CPU Scheduling Module 4 System Caches Module 5 Memory Tuning

Module-by-Module Overview
Module 6 System Buses Module 7 I/O Tuning Module 8 UFS Tuning Module 9 Network Tuning Module 10 Performance Tuning Summary

Course Objectives
Describe the purpose and goals of tuning Describe the common characteristics of system hardware and software components Explain how the major system components operate Explain tuning report elements and how they relate to system performance Identify tuning parameters and understand when and how to set them Describe the different types of performance graphs

Course Objectives
Determine whether a problem exists in system hardware configurations Identify the primary memory problems and their solutions Identify network performance problems and their solutions Explain central processing unit (CPU) scheduling and thread functions in the Solaris OE

Guidelines for Module Pacing

Module About This Course "Introducing Performance Management" "Processes and Threads "CPU Scheduling "System Caches "Memory Tuning "System Buses "I/O Tuning "UFS Tuning "Network Tuning "Performance Tuning Summary Day 1 P.M. A.M./ P.M. A.M. A.M. A.M. A.M. P.M. A.M. P.M. A.M. P.M. Day 2 Day 3 Day 4 Day 5

Topics Not Covered

Capacity planning General system administration Network administration and management General Solaris OE problem resolution Data center operations Disk storage management

How Prepared Are You?

Experience with Solaris OE system administration? Understand the function of the system kernel? Have a basic understanding of software design and functionality? Understand the operation of I/O devices, storage arrays, and the UNIX file system (UFS)? Have a basic understanding of network protocols?

Introductions
Name Company affiliation Title, function, and job responsibility Performance tuning experience Experience with performance monitoring tools Reasons for enrolling in this course Expectations for this course

How to Use the Course Materials

Course Map Objectives Relevance Additional resources Overhead images Lecture Exercise Check your progress Think beyond
How to Use the Icons

Additional resources
Discussion
Caution Warning

Typographical Conventions and Symbols

Courier is used for the names of commands, files, and directories, as well as on-screen computer output. Courier bold is used for characters and numbers that you type. Courier italic is used for variables and commandline placeholders that are replaced with a real name or value. Palatino italic is used for book titles, new words or terms, or words that are emphasized.

Module 1
Introducing Performance Management
April 2001
Module Overview
Objectives Relevance

Module 1, slide 2 of 46
Objectives
Explain what performance tuning really means Characterize the type of work performed on a system to make meaningful measurements Explain performance measurement terminology Describe a typical performance tuning task Describe the purpose of baseline measurements

Objectives
Describe how performance data is collected in the system Describe how the monitoring tools retrieve the data List the data collection tools provided by the Solaris Operating Environment (OE) Explain how to view and set tuning parameters Describe the system accounting functions Describe the use of Sun Management Center software

What Is Tuning?
Improve resource usage Improve user perception of the system Make more efficient use of system resources Increase system capacity Adapt the system to your workload Lower costs

Why Should a System Be Tuned?

To meet service level agreement-related values To provide access to the appropriate resources to a nominated group of users To allow consolidation of services onto a specific host To provide sufficient capacity for expected peak loading

Basic Tuning Procedure

Get baseline measurements
Find a bottleneck Y Can any improvements be made? N Is performance satisfactory? Reduce or move it
Stop

Conceptual Model of Performance

Configuration changes
Workload
Computer system under analysis
Performance measurements
Random fluctuations and noise

Where to Tune First

Theory Realistic Area Operating system parameters Database Management System (DBMS) Application software Server and network hardware Example
ncsize
Shared memory Development methodology or software tools Disk layout

Where to Tune?
The closer to the application you tune, the more leverage you have. System administrators have little control at the application or Database Management System (DBMS) levels. For the operating system (OS) and hardware, general tuning often benefits all applications. You can tune the system specifically for one application if you accept the trade-offs. You need to understand the characteristics of applications to properly tune them.
Performance Measurement Terminology

Bandwidth Throughput Response time Service time Utilization Baseline

Performance Graphs
Throughput
Load level or user count Saturation point
Response time
Fast response degradation
Gradual response degradation
Load level or user count
Utilization
Load level or user count

Example Presentation Graph

Sample Tuning Task

Confirm what the problems appear to be Collect appropriate tuning data Locate the problems Generic system problems Application-specific problems Develop a plan to fix them Fix what is reasonable Do not fix what is not broken Verify that performance is acceptable
The Tuning Cycle Memory
CPU
I/O

SunOS Components
User level User applications and libraries System calls Process management File system Kernel mmap() M e m o r y m a n a g e m e n t S c h e d u l i n g
metadata cache segmap cache
I P C
Character
Block
Device drivers
Machine-dependent code Hardware

How Data Is Collected

Have the data collection tools read counters and other data structures Wait for the specified interval Reread the data structures Calculate and scale the counter differences Remember, the first reading uses zeros Ignore the first line in monitoring reports If the kernel routines do not collect the data, the data cannot be reported.

The kstat Kernel Data Structure

Is created by a system component Exists for everything the component wants to measure Has a specific format for each component Is read by the kstat-related system calls A copy is made into the requesting process Allows a whole chain of kstats to be read Is updated by a component as needed Usually, updates are made when the recorded event occurs
How the Tools Process the Data

Data is accumulated during each interval. The change from the previous interval is calculated. The information is written: To memory-based buffer space To the screen (stdout) As graphic objects (Sun Management Center) To a file for post-processing Post-processing can combine intervals. Be careful of too much smoothing.
Monitoring Tools Provided With the Solaris OE

sar vmstat iostat mpstat netstat nfsstat SUN Management Center Other utilities
The sar Command

Does general data collection and reporting Has 16 different reporting types covering: File system and system calls Physical memory usage, paging, swap, and kernel memory Block device activity and Terminal Identifier (TTY) activity Process table status System V Inter-Process Communication (IPC) activity and CPU usage
The vmstat Command

Reports primarily processor-related statistics Has five reports covering: Process, interrupt, CPU cache, and CPU activity Virtual memory and swapping activity Disk activity Provides information similar to sar

The iostat Command

Reports I/O device performance data Has 19 operands covering: CPU utilization and TTY activity Disk activity and device errors NFS mount activity Uses -M to display Mbytes per second instead of Kbytes Uses -n to display cXtXdX device names

The mpstat Command

Reports CPU-related activity Reports CPU-specific statistics Has one report: Page faults, interrupts, and cross-calls Locking, context switches, and CPU utilization Is usually not needed

The netstat Command

Reports network-related activity Has 14 different reports covering: Socket state, interface state, and Dynamic Host Configuration Protocol (DHCP) information arp and routing tables STREAMS statistics Gives you excellent visibility on the use of your network Can be complicated if you have many network interfaces

The nfsstat Command

Reports NFS-related activity Has five reports covering: Client-side information NFS-mounted file system information NFS and Remote Procedure Call (RPC) information Can help isolate the exact source of an NFS problem Can be useful with DBMS network problems as well Use RPCs to transfer requests and data

Sun Enterprise SyMON

Details from http://www.sun.com/symon/ Provides remote monitoring of: Performance data System failures System configuration Syslog information Uses Simple Network Management Protocol (SNMP) to transfer data Uses rules to signal exception conditions
Sun Management Center

http://www.sun.com/sunmanagementcenter/

Other Utilities
prex tnfxtract and tnfdump busstat cpustat and cputrack The /proc directory and /usr/proc/bin utilities Process Manager memtool System accounting

The prex Command

prex controls probes in a process or the kernel. prex can be linked to a process execution. Buffering of probe data can be turned on and off. Default buffer size is 384 Kbytes. Might experience problems when using 32-bit programs or interpreters in a 64-bit operating environment.

The tnfxtract and tnfdump Commands

tnfxtract reads the in-core buffer or can read a coredump file. tnfxtract produces a TNF binary output file. tnfdump converts a TNF binary file into ASCII text output. The output from tnfdump can be extensive.

The busstat Command

busstat provides access to bus related counters. Bus devices can be programmed to count events from a specified list. Though supplied as a Solaris 8 OE utility, busstat might not produce output depending on the hardware architecture. Use busstat -l to see if supported.

The cpustat and cputrack Commands

cpustat can be used to monitor overall performance of CPUs in the system. Processor cycle counts can be printed if supported on the system architecture. cpustat is restricted to use by the root user only. cputrack can be used to monitor individual processors.

The /usr/proc/bin Directory

This directory is still provided, for consistency, with the Solaris 8 OE. It contains 13 commands that format data from /proc. Commands include: ptree Displays a processs ancestors and children pmap Displays a processs virtual memory segments pfiles Displays details of a processs open files The files in this directory are symbolically linked to their equivalent files in /usr/bin.
The Process Manager

The memtool Package

Is not provided with the Solaris 8 OE Retrieve the latest version from ftp://playground.sun.com/pub/memtool Shows detailed information on memory usage Reports on physical memory and swap Provides real-time reports on: Process memory, file memory, and shared library usage

The SE Toolkit
Comprises three software packages that provide modifiable tools to monitor your system Provides graphic current and historical performance information Will offer suggestions on avoiding or resolving problems that it detects Looks at disks, networks, NFS, swap space, memory, CPUs, mutexes, and other areas Is available from http://www.sun.com/sun-on-net/ performance/se3 Is not supported on production systems
System Performance Monitoring With the SE Toolkit

System Accounting
Accounting is not enabled by default. Accounting enables you to track what is executed and how much resource it uses. It can be very useful in determining what is using your system. You must run the reports to reduce the data. It adds system overhead. If you have a varying environment, it is the easiest way to decide where to spend your time. You can correlate accounting data with tuning reports.
Viewing Tuning Parameters

Several utilities that provide details of the current settings of system variables are: sysdef New to the Solaris 8 OE mdb Prior to the Solaris 8 OE adb ndd nm

Setting Tuning Parameters

Parameters are usually set in /etc/system. The system must be rebooted for changes to take effect. Some changes can be made with mdb while the system is running. Be careful you could take the system down. Not all parameters can be changed with mdb. Network device parameters can be set and inspected with ndd. These changes can also be made in /etc/system.
The /etc/system File

This file can be used to modify kernel-tunable variables. The basic format is: set [modulename:]variablename = value You can also see /etc/system to force modules to be loaded at boot time, to specify a root device other than the default, and so on. The system must be rebooted for changes to take effect. Care should be used when updating /etc/system; this file modifies the operation of the kernel.
Check Your Progress

Explain what performance tuning really means Characterize the type of work performed on a system to make meaningful measurements Explain performance measurement terminology Describe a typical performance tuning task Describe the purpose of base line measurements

Check Your Progress

Describe how performance data is collected in the system Describe how the tools retrieve the data List the data collection tools provided by the Solaris OE Explain how to view and set tuning parameters Describe the system accounting functions Describe the use of the Sun Management Center system monitor

Think Beyond
Are there areas that will never cause a bottleneck? Why bother to tune a system at all? Are default system parameters geared toward desktop systems or servers? Why? How long should a tuning project take? How do you know? How can you determine a realistic and satisfactory performance goal ahead of time?

Think Beyond
What other tools might the Solaris OE need? Why do the tools seem to overlap so much? Can you write your own tools? Can an industry-standard set of tools be developed?

Module 2
Processes and Threads
April 2001
Module Overview

Objectives
Describe how a process commences its execution cycle Describe the life cycle of a process in a SunOS environment Explain multithreading and its benefits State the limits of the SunOS process scheduler Describe locking and its performance implications List the functions of the clock thread Describe how system interrupts affect process execution

Objectives
Use appropriate utilities to display process-and threadrelated information Describe the related tuning parameters

Process Lifetime
Parent Creation Child fork()
Parent is context switched into a sleep state
exec()
Execution
exit()
Termination
Status collection
wait()

fork With exec Example

Shell Print prompt Read from terminal Input line Parse line
Y EOF? N N fork() OK? Error Done
wait() on PID of child Parent
Which process? Child
exec() Input command

Process States
fork process
SRUN INIT
(creation) idle (on CPU) kernel mode
system call or interrupt return from call or interrupt
ZOMBIE
(defunct)
exit
URUN RUN
(ready)
wake on signal
(on CPU) user mode
sleep
SLEEP
(waiting)

Process Performance Issues

The default maximum number of processes allowed is 30,000. Executables that use shared libraries take fewer system resources (disk space, memory, I/O, and so on). Opening program threads is more efficient than forking multiple subprocesses. Zombie processes cause no performance problems but each use up one slot in the process table. The Solaris 8 OE introduces the pidmax parameter. This parameter determines the highest possible PID value.

Process Lifetime Performance Data

The sar -v and -c options provide data related to the process lifetime. scall/s The total number of system calls made by all processes in the system, both user and system fork/s The total number of fork system calls exec/s The total number of exec system calls proc-sz The ratio of the total number of allowed processes that are in use

Process-Related Tunable Parameters

These are scaled from maxusers or maxpid
Name Default Number of Mbytes of real memory - 10 (16 x maxusers) + 10 Min 8 266 133 213 48 48 226 226 Max 2048 (4096)
maxusers max_nprocs maxuprc ndquot npty pt_cnt ncsize
maxpid
29995 50480 3000+
max_nprocs - 5
(maxusers x 10) + max_nprocs 48 48 4 x (maxusers + max_nprocs) + 320
maxpid
34906 34906
ufs_ninode 4 x (maxusers + max_nprocs) + 320

Tunable Parameters Specic to the Solaris 8 OE

The following parameters have been introduced or modified in the Solaris 8 OE release
Name Description Species the value of the largest possible process ID You set the value in the /etc/system le the value range allowed is 266 to 999 999 Derives its value from pidmax at system boot time When set, the value of pidmax is ignored
pidmax
maxpid
max_nprocs* Will be assigned a value based on the value of pidmax or the formula (16 x maxusers) + 10, whichever is the higher
* The max_nprocs parameter existed in pre-Solaris 8 versions but is managed differently in the Solaris 8 release.

Multithreading
A thread is a logical sequence of program instructions. The kernel is multithreaded. Multiple tasks may be running in the kernel concurrently and independently. A user process can have many application threads that execute independently of each other. There is still only one executable in the process. Fewer system resources are used than with multiple processes. Special programming techniques are required.
Application Threads
proc Threads
LWP
Each thread has its own stack. Threads share the process address space. Threads execute independently (and concurrently). Threads are completely invisible outside fo the process. Threads cannot be controlled from the command line.
Process Execution
87 62
CPU
45
15
CPU

Thread Execution
Process System
73 44
LWP
87 62
CPU
12
45
LWP
15
CPU

Process Thread Examples

User layer proc1 proc2 proc3 proc4 proc5
Kernel layer
CPU assignment
Hardware layer
= Application thread
= LWP
= CPU
= Kernel thread

Performance Issues
Multithreading an application allows it to: Be broken into separate tasks that can be scheduled and executed independently Take advantage of multiprocessors with less overhead than multiple processes Share memory without going through the overhead and complexity of IPC mechanisms Use a cleaner programming model for certain types of applications Extend the program more easily
Thread and Process Performance

Creation Primitives Unbound thread create Bound thread create fork() Nanoseconds 150 370 1700 Ratio 2.4:1 11.3:1
Using fork can create memory management issues. Timings were obtained on a SPARCstation 5 system.

Locking
Use locks to synchronize threads by serialization. Locks protect critical data from simultaneous write access. You must use locks when threads share writable data. SunOS provides four types of locks. Which type is used depends on the requirements. A bad locking design can cause performance problems. Locking problems usually require significant reprogramming.

Locking Problems
Lock contention due to inappropriate lock type Deadlock Lost locks Race conditions Incomplete implementation

The lockstat Command

Lock use in the kernel is identified. Unidentified delays might be caused by lock contention. Excessive counts might indicate a problem.
# lockstat lpstat Adaptive mutex block: 2 events Count indv cuml rcnt nsec Lock Caller --------------------------------------------------------------------------1 50% 50% 1.00 87500 0xf5ae43d0 esp_poll_loop+0xcc 1 50% 100% 1.00 151000 0xf5ae3ab8 esp_poll_loop+0x8c ---------------------------------------------------------------------------

Interrupt Levels
15 Nonmaskable interrupt Real-time CPU clock Serial communications 10 clock() Video vertical interrupt, graphics Tape, Ethernet 3 Lower priority SCSI and FC disks Soft interrupts 0 All interrupts enabled
Higher priority

The clock Routine

The clock executes at interrupt level 10. Most system timing is run off this clock. Each time the clock routine executes is a tick. For most processors, there are 100 ticks per second. Ticks per second can be set to 1000 for real-time processing. This is the limit of normal timing resolution. Serial communication interrupts can cause the time-of-day clock to run slow.
Clock Tick Processing

Every clock tick the system checks for: Timer time-outs Sleep and wait times I/O timeouts Network timeouts Profiling timers Active threads that have used their time slice Processes that have exceeded their CPU resource limit

Process Monitoring Using the ps Command

You need to identify active processes before determining: Which process is causing a delay Which resource is creating a bottleneck for the process The ps command enables you to check process status. The ps command helps determine how to set process priorities. The BSD version, /usr/ucb/ps -aux, provides the best performance-related data.
Process Monitoring Using System Accounting

Short-lived processes cannot be observed using ps System accounting keeps a record of all processes Terminating processes are logged to /var/adm/pacct The acctcom and lastcomm commands report on /var/adm/pacct Summary scripts /usr/lib/acct/runacct and monacct create daily and monthly summary reports.

Check Your Progress

Describe how a process commences its execution cycle Describe the life cycle of a process in the SunOS environment Explain multithreading and its benefits State the limits of the SunOS process scheduler Describe locking and its performance implications List the functions of the clock thread

Check Your Progress

Describe how system interrupts affect process execution Use appropriate utilities to display process - and thread related information Describe the related tuning parameters

Think Beyond
Why does the Solaris OE need threads when processes have worked for so long? Why are LWPs used in the thread model? Are they really needed? Why are locks difficult for programmers to work with?

Module 3
CPU Scheduling
April 2001
Module Overview

Objectives
Describe the Solaris OE scheduling classes Explain the use of dispatch priorities Describe the time-sharing scheduling class Explain Solaris OE real-time processing List the commands that manage scheduling Describe the purpose and use of processor sets Introduce Solaris Resource Manager

Scheduling States
RUNNING (on CPU)
BLOCKED
SCHEDULED
READY TO RUN (queued)

Wake on signal
SLEEPING
sleep or wait on I/O

Scheduling Classes
Four scheduling classes are provided by SunOS: Timesharing TS Interactive IA System SYS Real-time RT The classes are loaded dynamically as they are used. Scheduling algorithms are table driven.

Dispatch Priorities
Dispatch queues 169 160 159 Dispatch queues Interrupt thread priorities
Real-time priorities 109 100 99 60 59 Timesharing/interactive priorities 0 0 Interrupt thread priorities Kernel priorities 60 59 Timesharing/interactive priorities 100 99 Kernel priorities
Without real time
With real time

Dispatch Parameter Table Issues

For timesharing class processes: Reducing time quanta favors interactive processes. Raising time quanta favors compute-bound and large processes Using the ts_maxwait and ts_lwait fields controls CPU starvation. Slightly raising the values of ts_tqexp causes the priority of compute-bound processes to drop more slowly. Changing the table can be done to fit your workload.
Timesharing/Interactive Dispatch Parameter Table

# Time Sharing Dispatcher Configuration RES=1000 # ts_quantum 200 200 200 200 200 200 200 200 200 200 160 160 160 160 160 160 160 160 160 160 120 ts_tqexp 0 0 0 0 0 0 0 0 0 0 0 1 2 3 4 5 6 7 8 9 10 ts_slpret 50 50 50 50 50 50 50 50 50 50 51 51 51 51 51 51 51 51 51 51 52 ts_maxwait ts_lwait 0 50 0 50 0 50 0 50 0 50 0 50 0 50 0 50 0 50 0 50 0 51 0 51 0 51 0 51 0 51 0 51 0 51 0 51 0 51 0 51 0 52 PRIORITY # # # # # # # # # # # # # # # # # # # # # LEVEL 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20


# ts_quantum 120 120 120 120 120 120 120 120 120 80 80 80 80 80 80 80 80 80 80 40 ts_tqexp 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 ts_slpret 52 52 52 52 52 52 52 52 52 53 53 53 53 53 54 54 54 54 54 55 ts_maxwait ts_lwait 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 52 52 52 52 52 52 52 52 52 53 53 53 53 53 54 54 54 54 54 55 PRIORITY LEVEL # # # # # # # # # # # # # # # # # # # # 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40


# ts_quantum 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 40 20 ts_tqexp 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 ts_slpret 55 55 55 55 56 57 58 58 58 58 58 58 58 58 58 58 58 58 59 ts_maxwait ts_lwait 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 32000 55 55 55 55 56 57 58 58 59 59 59 59 59 59 59 59 59 59 59 PRIORITY LEVEL # # # # # # # # # # # # # # # # # # # 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59

Timesharing Dispatch Parameter Table (Batch)

globpri quantm 100 0 100 1 100 2 100 3 100 4 100 5 100 6 100 7 100 8 100 9 80 10 80 11 80 12 80 13 80 14 80 15 80 16 80 17 80 18 80 19 tqexp slpret mxwait 5 10 0 5 11 0 5 12 1 5 13 1 5 14 2 5 15 2 5 16 3 5 17 3 5 18 4 5 19 4 5 20 5 5 21 5 5 22 6 5 23 6 5 24 7 5 25 7 5 26 8 5 27 8 5 28 9 5 29 9 lwait 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

The dispadmin Command

Displays or changes scheduler parameters Uses options: -l Lists available scheduling classes -c class Specifies the class whose parameters are to be displayed or changed -g Displays configured parameters Provides a simple way of formatting control file -s file Sets parameters from a file
dispadmin Example
# dispadmin -c TS -g #Time Sharing Dispatcher Configuration RES=1000 ts_maxwai t 0 0 0 0 0 0 0 0 0 0 0 0 32000
#ts_quantum 200 200 200 200 200 200 200 ... 40 40 40 40 40 20
ts_tqexp 0 0 0 0 0 0 0 44 45 46 47 48 49
ts_slpret 50 50 50 50 50 50 50 58 58 58 58 58 59
ts_lwait 50 50 50 50 50 50 50 59 59 59 59 59 59
PRIORITY LEVEL # # # # # # # # # # # # # 0 1 2 3 4 5 6 54 55 56 57 58 59

The Interactive Scheduling Class

Is used to enhance interactive performance Is the default scheduling class for processes in Common Desktop Environment (CDE) sessions Uses most of the timesharing class facilities Boosts the priority of the task in the active window by 10 points The priority is reset when it is no longer the active window. Does not boost processes changed using nice or other commands
Real-Time Scheduling
Real-time systems have rigid response time or dispatch latency requirements. Failure to meet the time constraints can cause problems. General-purpose systems have problems with real time. Page faults have an unknown time to resolve, so: All pages are locked in memory. Locked pages include shared libraries and shared memory. A preemptable kernel is used.

Real-Time Issues
Real-time processes can hurt the system if they: Use lots of memory All pages are locked in memory Are compute-intensive Other tasks might be locked out and system daemons might suffer fork other processes Children inherit real-time Get stuck in a loop Might be difficult to stop Make sure that you really need real time before using it. Consider using a real-time thread instead of a process (priocntl(2)) Set hires_tick to 1 to set clock interrupts to 1000
Real-Time Dispatch Parameter Table

globpri quantm 100 100 100 101 100 102 100 103 100 104 100 105 100 106 100 107 100 108 100 109 80 110 80 111 80 112 80 113 80 114 80 115 80 116 80 117 80 118 80 119 60 120 60 121 60 122 60 123 60 124 60 125 60 126 60 127 60 128 60 129 globpri quantm 40 130 40 131 40 132 40 133 40 134 40 135 40 136 40 137 40 138 40 139 20 140 20 141 20 142 20 143 20 144 20 145 20 146 20 147 20 148 20 149 10 150 10 151 10 152 10 153 10 154 10 155 10 156 10 157 10 158 10 159

The priocntl Command

Works only at the process level Uses options: -l List currently loaded scheduling classes -d Display a processs scheduling parameters -s Set a processs scheduling parameters -e [-c class] Create a new process Makes the priocntl(2) system call available to programs Works on threads and LWPs
The priocntl Command

Example
# priocntl -e -c RT -p 50 -t 500 command args
Run command with operand args Use the real-time scheduling class Set real-time class relative priority to 50 Global priority is 150 (50 + base real-time class priority). Use a time quantum of 500 milliseconds.

Kernel Thread Dispatching

Switch request Scheduling class routine
Finished with time slice? No Yes
Queue front
Queue new
Pick new thread
Set it up Pick a CPU
Run it
Old thread
New thread

Processor Sets
Allows exclusive use of groups of processors by certain processes Also known as CPU fencing Is very different from the pbind (1M) Is managed by the psrset(1M) command Is controlled by the root user Has system-defined processor sets, which can be used by any user Forces Dynamic Reconfiguration (DR) to release bindings if necessary
Using Processor Sets

You must leave one processor for the system. Make sure you leave enough CPU capacity for non-fenced work. A CPU may be in only one processor set. The system will not adjust dynamically. The tools do not show processor set queues. Queuing on a processor set could be missed. You can change CPU membership dynamically. You need to watch fenced processes carefully.
The Run Queue

A count of kernel threads waiting to run is kept. It contains a total of all system dispatch queues. It is not scaled by the number of CPUs. A depth of 35 per CPU is usually OK. This depends on the type of work being run. There is no way to tell what is waiting or how long it has been waiting.

CPU Activity
User The user process is running. System The kernel is running. This includes system thread time and user system calls. Wait I/O The CPUs are idle, but a disk device is active. Idle The system is waiting. Some reports add Wait I/O and Idle. This is usually reported as Idle time. You can check the tools man page.
CPU Performance Statistics

Activity The time CPU is executing in user mode The time CPU is executing in kernel mode The time CPU is waiting for I/O The time CPU is idle The average number of threads blocked on I/O The average runnable threads over the interval The percentage of time the run queue was occupied The average number of thread (context) switches Origin cpu.sysinfo[CPU_USER] cpu.sysinfo[CPU_KERNEL] cpu.sysinfo[CPU_WAIT] cpu.sysinfo[CPU_IDLE] sysinfo.waiting sysinfo.runque sysinfo.runocc b (number of) r (number of) runq-sz (number of) %runocc (%) vmstat (Unit) us (%) sy (%) Summed as id (%) sar (Unit) %usr (%) %sys (%) %wio (%) %idle (%)
cpu_sysinfo.pswitch
cs (number per second)
pswch/s (number per second)

Available CPU Performance Data

cpu_sysinfo.cpu[CPU_USER] User mode CPU cpu_sysinfo.cpu[CPU_KERNEL] Kernel mode CPU cpu_sysinfo.cpu[CPU_WAIT] CPU idle, block I/O active cpu_sysinfo.cpu[CPU_IDLE] CPU idle cpu_sysinfo.pswitch The number of thread switches sysinfo.runque The average number of runnable threads sysinfo.runocc The percentage of time a thread was on any dispatch queue
CPU Control and Monitoring

Processor control and information is reported by: mpstat Displays CPU usage statistics psradm Enables and disables individual CPU psrinfo Determines which CPUs are enabled prtconf Shows device configuration prtdiag Prints system configuration and diagnostic information Process Manager Shows current activity psrset Manages processor sets
The mpstat Command

Displays per-processor performance statistics Is similar to vmstat in operation Is used to check per-CPU loads and load balancing Is usually not needed in normal tuning Can be useful with erratic performance Uses lockstat to resolve locking problems

mpstat Data
xcal The number of interprocessor cross calls (cpu_sysinfo.xcalls) intr The number of interrupts received on each CPU (cpu_sysinfo.intr) migr The number of threads that migrated from another CPU (cpu_sysinfo.cpumigrate) smtx The number of failed adaptive mutex_enter() calls (cpu_sysinfo.mutex_adenters) srw The number of times readers and writer locks were not acquired on the first try (cpu_sysinfo.rw_wrfails, cpu_sysinfo.rw_rdfails)
Solaris Resource Manager

Is Suns software tool for allocating shares of system resources to users, groups, and enterprise applications Offers the ability to control and allocate: CPU time Logins and processes Virtual memory Connect time Allows information technology (IT) managers to consolidate applications on a single UNIX server and achieve service level agreements
Solaris Resource Manager Hierarchical Shares Example

Server
A BC A Financial database: Sixty shares CPU Twenty Gbytes virtual memory Three hundred processes
Web server: B Ten shares CPU Thirty Gbytes virtual memory One hundred processes Kelvin Six shares CPU Ten Gbytes virtual memory Forty processes Susan Five shares CPU Group-limited virtual memory Sixty Processes Brian Nineteen shares CPU Group-limited virtual memory Seventy processes
Operations database: C Thirty shares CPU Twenty-ve Gbytes virtual memory Two hundred processes
Benet:
Efcient utilization of resources Improved service level to end users

Solaris Resource Manager Limit Node

Defines a resource control policy for users or groups of users rather than a system-wide policy Has as a fundamental building block in Solaris Resource Manager, the lnode (limit node) Has an lnode for each user and group of users Uses an lnode that contains attributes that correspond to a resource control policy Has the kernel access lnodes to determine a users resource limits and resource usage

Check Your Progress

Describe the Solaris OE scheduling classes Explain the use of dispatch priorities Describe the time-sharing scheduling class Explain Solaris OE real-time processing List the commands that manage scheduling Describe the purpose and use of processor sets Introduce Solaris Resource Manager

Think Beyond
Why are there four scheduling classes? What was the point of adding the interactive scheduling class? Why would you want to change a dispatch parameter table? When might you want to use the SunOS real-time capability? What types of work might you use processor sets for? Why might you use Solaris Resource Manager instead of processor sets?

Module 4
System Caches
April 2001
Module Overview

Objectives
Describe the purpose of a cache Describe the basic operation of a cache List the various types of caches implemented in Sun systems Identify the important performance issues for a cache Identify the tuning points and parameters for a cache Understand the memory systems hierarchy of caches and their relative performances Use standard tools to display cache-related statistics

What Is a Cache?
Keeps accessed data near its user Must provide high-speed data access Holds a small subset of the available data Is used by hardware and software Has many different types around the system Is critical for performance Can be managed in many different ways

Hardware Caches
There are hardware caches all over the system. CPU Level 1 and Level Two Translation Lookaside Buffer (TLB)/Dynamic Lookaside Table (DLAT)/Page Descriptor Cache (PDC) Main memory I/O components Each system has different caches with different characteristics.

Static RAM (SRAM) and Dynamic Random Access Memory (DRAM)

Hardware caches are usually SRAM. SRAM does not need a refresh. DRAM data fades after about 10 cycles. Refresh rewrites data, but this delays access. SRAM does not fade, so no refresh is needed. SRAM takes four times as many transistors as DRAM. SRAM is always more costly and takes more power.
CPU Caches
There are usually two levels of cache in a CPU: Level 1 (internal) Usually 440 Kbytes in size Level 2 (external) Usually 0.58 Mbytes in size A Level-1 cache operates at CPU speeds. Data must be in the Level 1 cache before the CPU can use it.

Cache Operation
The caches user requests data. If found, this is a cache hit. The data is returned. If not found, this is a cache miss. Data is requested from the next level up. The requester has to wait until the data arrives.

Cache Miss Processing

Caches are usually kept 100 percent full. On a cache miss, new data must be put in the cache. This process is called a move in. A location for the incoming data must be chosen. Almost always, the least recently used (LRU) algorithm is used. The oldest data is always replaced.

Cache Data Replacement

Data to be replaced might have to be copied back. Changes are not always reflected up. A move out or ush copies modified data to the next level up. When the move out is complete, the move in starts. The amount of data moved is a cache line. A hardware cache line is usually 64 bytes.

Cache Operation Flow

Cache hit
Request
Hit
Cache miss
M
Next level cache Request

is
Request
Move out Move in

Cache Hit Rate

The performance of a cache depends on its hit rate. The cache hit rate is how often requested data is available. The cache hit rate depends on: The size of the cache The fetch rate The locality of data references The cache structure

Cache Hit Rate

Misses are very expensive. For example: The hit cost is 20 units, miss cost is 120 units. At 100 percent hit rate, cost = 100 x 20 = 2000. A 99 percent hit cost = (99 x 20) + (1 x 120) = 1980 + 120 = 2100. A 1 percent miss cost is 100 2000 or 5 percent degradation. The exact numbers depend on the system.

Effects of CPU Cache Misses

Performance Level
Cache Hit Rate (Percent)
There are many ways to improve the hit rate.

Cache Characteristics
There are several pairs of mutually exclusive cache characteristics. Virtual Write through Direct mapped Fully associative Harvard Snooping cache Physical Write back Set associative Unied

Virtual Address Cache

Stack
Hole
Physical address CPU
Cache
Virtual address
MMU
Data
Text
Physical memory

Physical Address Cache

Stack
Hole Physical address Data CPU
MMU
Virtual address
Cache
Text
Physical memory

Direct-Mapped and Set-Associative Caches

Direct-mapped cache One possible location for the data Set-associative cache Multiple possible locations All possible locations are checked in parallel. The number of locations is referred to as ways; for example, four-way. Set associative caches are more complex, but provide better contention management. Harvard cache Separate instructions and data

Write-Through and Write-Back Caches

Write-through cache Always updates the next level up when its data changes. Write-back cache Usually updates only when the line is flushed. Write-back caches assume a line modified once will be modified again. This may allow write cancellation. Unnecessary updates are avoided, saving bus capacity.

CPU Cache Snooping

Write-back caches are great for CPU performance. In a multiprocessor system, other CPUs might need modified data from a different CPUs cache. To avoid write-though caching, you must have the system use cache snooping. All memory requests are monitored by the cache controllers. When a request is made for modified data in its cache, the controller provides it.

CPU Cache Snooping

Module 0 CPU Snooping cache Write buffers Module 1 CPU Snooping cache Write buffers Main memory
MMU
MMU
Bus
I/O interfaces

Cache Thrashing
Picking the wrong memory stride essentially turns off the cache. Accessing data at a regular address interval can map to the same cache location every time. This can cause 100-percent cache miss. The performance degradation is easily noticeable. Diagnosing this problem can be hard; everything is working correctly. Inspecting the application and comparing it with the cache characteristics is required.
Cache Thrashing
Physical pages Virtual address 1 (array_in[])
MMU page table Cache
Virtual address 2 (array_out[])

System Cache Hierarchies
CPU
Registers
L1 cache
L2 cache
Main memory
Disk
Network
People

System Cache Hierarchies

Cache Registers L1 CPU L2 CPU Main memory Disk Network Size 1 Kbyte ~32 Kbytes 0.58 Mbytes 16 to 64 Gbytes 1 Gbyte to 11 Tbytes N/A Access Time 3 ns 10 ns 60 ns 450 ns 20 ms 200 ms Managed by Compiler Hardware Hardware Software Software, people Software, people

Relative Access Times

If one nanosecond was one second, then: Level CPU cycle time Level-1 cache Level-2 cache Main memory Disk Network Nanoseconds 3 10 60 450 20,000,000 200,000,000 Seconds 3 10 1 minute 7.5 minutes 7.7 months 6.5 years

Cache Performance Issues

Cache and locality of reference are important. Remember, linked lists are bad for locality. Use Forte Developer Workshop analysis to locate problems. Cache thrashing can occur. Check algorithms for power of two sizes. Run on a different type of system. vmstat -c reports cache flushes (but not hits).

Items You Can Tune

Application Locality of reference Memory strides Algorithms Compiler optimization Main memory management

Cache Monitoring Command Summary

prtdiag Determines size of Level-1 and Level-2 prtconf Uses the -v and -p options vmstat Views Directors Name Looking Cache (DNLC) and re (reclaim) statistics sar Views many cache related statistics netstat Uses -k to view inode cache statistics prtmem Views current size of page/file cache sysdef Views configured size of buffer cache
Check Your Progress

Describe the purpose of a cache Describe the basic operation of a cache List the various types of caches implemented in Sun systems Identify the important performance issues for a cache Identify the tuning points and parameters for a cache Understand the memory systems hierarchy of caches and their relative performances Use standard tools to display cache-related statistics

Think Beyond
How could you prove that you were having hardware caching problems? Are there workloads that are insensitive to caches? How big is too big for a cache? How many caches might be too many to coordinate among CPUs? Why? Can you find software equivalents for each of the hardware cache types?

Module 5
Memory Tuning
April 2001
Module Overview

Objectives
Describe the structure of virtual memory Describe how the contents of main memory are managed by the page daemon Understand why the swapper runs Interpret the measurement information from these components Understand the tuning mechanisms for these components

Virtual Memory
Is what the programs see as addressable memory Resides in main memory or on disk (or both) Is created and deleted by the program as necessary Is managed by the OS SunOS acts as the main memory cache manager. Is contained in address spaces (processes) Is managed in segments of related, contiguous bytes

Process Address Space

32-bit sun4u Kernel
0xFFBEC000
64-bit sun4u Kernel base

0xFFFFFFFF.7FFFC000
Kernel Stack
Stack
Shared libraries
Shared libraries
Heap
Heap
Data Text
0x00010000
Executable le
0x00000001.00000000
Data Text

Types of Segments
Device space Kernel stacks Buffer cache Kernel data Kernel text Stack segkmem segkp segmap segkmem segkmem segvn
Kernel base
Device Frame buffer device
segdev
Data Text
segvn segvn

Virtual Address Translation

Stack
Hole Physical address CPU Data Virtual address Physical memory MMU
Text
Translations are performed by the memory management unit (MMU) using a three-stage table lookup process.

Page Descriptor Cache

Is also called Translation Lookaside Buffer (TLB) or Dynamic Lookaside Table (DLAT) Caches virtual to real address translations Holds entries created by memory management unit (MMU) table lookups Is usually between the Central Processing Unit (CPU) and the Level-1 one cache Has a limited size; usually 64 pages Is fully associative Has one per CPU Entries are based on the CPUs activity.
Virtual Address Lookup

Request A In PDC? No (PDC miss) CPU waits for MMU to do translation. Is translation valid? No (PDC hit) To L1 cache
Hardware
Yes
Put in PDC.
(Page fault) (Minor page fault) Yes Validate page.
Is the page anywhere in main memory?
Software
No (Major page fault) Get free page and do I/O. Suspend LWP. Yes I/O successful? No Error

The Memory-Free Queue

Main memory is a fully associative disk cache managed by the OS. Move ins and move outs occur as in any cache. For memory, this is called paging. Moves involving disk are very expensive. A move out followed by a move in is too costly. You always need to have enough pages on the free queue. Solaris 8 OE maintains a memory free list to help reduce page scanning.
The Paging Mechanism

Frequent checks are made to see if the amount of free memory is less than the quantity specified by the lotsfree parameter. If it is, the page daemon is run to replenish the free memory queue. Pages are stolen from their users if they have not been used recently. Pages are scanned to determine which can be stolen. The more pages needed, the faster the scan rate. If the page daemon cannot keep up, swapping may occur.
Page Daemon Scanning

Start
Locate next page
Reset reference bit
Yes
Was page referenced? No Was page changed? Yes No
Put page on free queue
Yes
At maxpgio limit?
No
Write page to disk

Page Scanner Processing

fastscan
Scan rate
Swapping occurs
slowscan
Deficit of RAM Page scanner required
RAM is available Page scanner is not required
minfree
desfree
lotsfree
Free memory

Default Paging Parameters

Variable
lotsfree
Purpose
Starts scanning at slowscan rate
Default
1/64 of main store; minimum 512 Kbytes. Suggested maximum is no more than 30 percent of main store 1/2 lotsfree; minimum 256 Kbytes 1/2 desfree; minimum 128 Kbytes or 1/256th main store Suggested maximum is no more than 7.5 percent of main store
desfree minfree
Swapping control Scans at fastscan rate
throttlefree pageout_reserve
Blocking memory requests put to sleep
minfree
The number of pages reserved exclu- throttlefree/2 sively for pageout or scheduler threads Fastest scan rate Slowest scanning rate Limits on pages to write per second Alternates tuning mechanism 1/2 available memory; maximum 64 Mbytes 100 40; can be in range 1 to 1024
fastscan slowscan maxpgio handspreadpages
fastscan

The Clock Algorithm

Pages 1 Backhand 0 0 0 0 1 0 0 0 Fronthand 1 0
handspreadpages
Reference bits

The maxpgio Parameter

Modified pages need to be written back. If every page was modified, there would be periodic I/O spikes. This would not help performance. The system cannot take only unmodified pages. Remember, some modified pages must be taken. Determine how many are safe. You must adjust maxpgio to a realistic value to set the limit (in the range 1 to 1024).

File System Caching

Process User process Kernel Buffer cache Inodes and meta data Logical read (le, offset)
File system
Page cache (le contents)
Disk Physical read (device, sector #)

The Buffer Cache

Caches only inodes and metadata Is dynamically sized; usually too large Is tuned using the bufhwm parameter in /etc/system Uses a sizing formula for UFS file systems: 300 bytes per inode + 1 Mbyte for every 2 Gbytes of concurrently accessed files For its cache hit rate, is monitored using sar -b Works best with a read hit rate of 100 percent on file system with few large files and 90 percent on file system with many small files
The Page Cache

The page cache grows to consume all available memory. There is no equivalent parameter to bufhwm for sizing the page cache. The page scan rate is proportional to the rate at which the file system is reading or writing pages to disk. Heavy file system usage results in high page scan rates. With heavy file system usage, a high page scan rate does not necessarily indicate a memory shortage.

Available Paging Statistics

Activity Pages scanned Scanner decit Pagein requests Pages paged in Pageout requests Pages paged out Pages freed Page reclaimed from free list Average amount of free memory Minor page faults Pages becoming shared Source cpu_vminfo scan deficit pgin pgpgin pgout pgpgout dfree pgfrec freemem hat_fault po (Kbytes) fr (Kbytes) re (pages) free (Kbytes) mf (pages) atch/s (pages) freemem (pages) pi (Kbytes) vmstat (Unit) sr (pages) de (pages) pgin/s ppgin/s (pages) pgout/s ppgouts (pages) pgfree/s (pages) sar (Unit) pgscan/s (pages)

Additional Paging Statistics

The Solaris 7 OE and later releases provide an extended set of paging counters. Counters can be displayed using memstat command. memstat can be downloaded from ftp://playground.sun.com/pub/rmc/memstat The memstat command shows the three types of paging broken out as executable, application, and file. memstat verifies that by enabling priority paging, the scanner frees file pages before freeing executable and anonymous pages.

Swapping
Swapping is a last resort (desperation swapping). If the page daemon consistently cannot keep up with the demand for memory, memory use must be cut. The number of swaps is not important; the fact that there are swaps is important. A process is swapped when its last LWP is swapped out. Its memory is not freed until then. Do not try to tune swaps; try to eliminate them.

When Is the Swapper Desperate?

The swapper is considered desperate for memory when all of the following are true: There is more than one runnable thread. avefree and avefree30 are both less than desfree. The paging rate is excessive or avefree is less than minfree.

What Is Not Swappable

A thread is not swappable if it is: In the system (SYS) or real-time (RT) scheduling class A process that has a fork in progress In execution or stopped by a signal Exiting or a zombie A system thread Blocking a higher priority thread A priority is calculated for each eligible thread left.

Swapping Priorities
The higher the calculated priority, the more likely the LWP will be swapped. The priority calculation is:
epri = swapin_time - rss (maxpgio 2) - pri

where swapin_time How long since the thread was last swapped rss Amount of memory used by the threads process pri The threads dispatching priority
Swap In
When the pressure is off the memory subsystem, swapped-out LWPs are brought back in, when they are ready to run. A swapping priority is calculated again. The lowest priority swapped-out LWP is brought back in. The process continues until all LWPs are back. Memory pressure might cause new swaps.

Swap Space
Virtual memory includes most of main memory. Reservable virtual memory is disk swap space plus physical memory, minus the number of kernel pages and a safety factor. Systems with a large amount of physical memory might not require much disk space for swap. You should allocate a primary swap partition large enough to hold a panic dump. Swap space is not allocated until the page is written out. If disk space is not available, the page stays in memory.
The tmpfs File System

It allows the creation of a virtual memory file system, such as /tmp. It uses virtual memory like any other user. It uses real memory and swap space. If used heavily, /tmp can fill main memory. You can limit the use of /tmp. The size option in the vfstab entry is used. You may move /tmp to a hard drive partition.

Available Swapping Statistics

Activity Average amount of free swap space Average amount of free swap space Process swap outs Pages swapped out Average number of swapped-out LWPs Percentage of the interval any LWP was swapped out LWP swap ins Pages swapped in Source swap_avail vmstat (Unit) swap (amount of unreserved swap; Kbytes) freeswap (amount of unallocated swap; sectors) so (Kbytes/sec) swpout/s bswout/s (sectors) w swpq-sz swpocc% sar (Unit)
swap_free swapout pgswapout swpque swpocc
swapin pgswapin
si (Kbytes/sec)
swpin/s bswin/s (sectors)

Shared Libraries
Shared libraries have a longer startup time when a routine is used; however: When the routine is in memory, the system can avoid subsequent I/Os. There is a huge savings in both main memory and disk space. Tools, such as Analyze from SPARCworks, can be used to improve locality of reference.

Paging Review
Kernel pages are fixed in memory. Kernel Stack Library data Library code Memorymapped files Device space Heap /bin/cp Program data Program code Address space segments Program data Program code Library code Program code is paged in from the file system /usr/lib/libc.so Library data Dynamicallylinked libraries are paged in in from the file system /home/me/myfile File data File data is paged in and out from its file /dev/rdsk/c0t0d0s1 Swap space Anonymous memory is paged out to swap space

Memory Utilization
Memory utilization must be observed before application memory requirements can be determined. You need to answer the questions: How much physical memory is available? How much memory is being used for file buffering? How much memory is being used for the kernel? How much memory are the applications using? How much memory is free?

Total Physical Memory

# prtconf System Configuration: Sun Microsystems Memory size: 128 Megabytes System Peripherals (Software Nodes): SUNW,SPARCstation-5 packages (driver not attached) disk-label (driver not attached) deblocker (driver not attached) obp-tftp (driver not attached) options, instance #0 aliases (driver not attached) openprom (driver not attached) ... ... # sun4m

File Buffering Memory

The page cache uses available free memory to buffer files on the file system. A summary of the page cache memory is displayed by the MemTool prtmem command.
# prtmem Total memory: Kernel Memory: Application memory: Executable memory: Buffercache memory: Free memory: # 56 14 19 17 1 4 Megabytes Megabytes Megabytes Megabytes Megabytes Megabytes
Page cache is displayed as Buffercache memory.

Kernel Memory
Use the sar command with the -k option (kernel memory allocation statistics). Total all of the alloc columns to determine the amount of kernel memory. Remember the output of the sar -k command is in bytes.
# sar -k 1 SunOS earth 5.8 Generic sun4m 22:21:54 sml_mem alloc 22:21:55 4657344 3853080 11/14/00
fail lg_mem alloc fail ovsz_alloc fail 0 30806016 28656468 0 2752512 0#
The kernel has grabbed 35,262,060 bytes of memory.

Identifying an Applications Memory Requirements

To predict memory requirements for users use the pmap command or Memtools pmem. Use pmap to show the amount of real memory each process is using. Use pmap to show the amount of real memory that is shared with other processes on the system, through shared libraries and executables. Use pmap to show the amount of private (non-shared) memory a process is using.

Free Memory
Free memory can be measured with the vmstat command. The first line of output from vmstat is an average since boot. The real free memory figure is available on the second line. The output is in Kbytes.
# vmstat 2 procs memory r b w swap free 0 0 0 210528 2808 0 0 0 199664 1904 0 0 0 199696 1056 0 0 0 199736 1144 ^C# re 0 0 7 0 mf 19 4 0 0 page disk pi po fr de sr f0 s0 s6 -49 38 73 0 24 0 5 0 0 12 0 0 0 0 0 2 0 0 0 628 620 0 20 0 0 0 0 0 0 0 0 0 0 0 0 0 faults in sy 247 1049 224 936 401 1212 165 800 cs 239 261 412 188 cpu us sy 4 2 7 2 27 9 0 0 id 94 92 64 100

Identifying a Memory Shortage

Determine if applications are paging excessively. Determine if the system could benefit by making more memory available for file buffering. Use vmstat to see if the system is paging: If there is no paging, there is no memory shortage. Constant nonzero values in the scan-rate (sr) and page-out (po) columns are characteristic of excessive paging. Look at the swap device for I/Os queued to it. Use MemTool to measure the distribution of memory.
Items You Can Tune

Page daemon parameters maxpagio lotsfree, minfree, desfree fastscan, slowscan The amount of physical memory The swap space placement /tmp usage The use of shared libraries

Check Your Progress

Describe the structure of virtual memory Describe how the contents of main memory are managed by the page daemon Understand why the swapper runs Interpret the measurement information from these components Understand the tuning mechanisms for these components

Think Beyond
Why is the use of main memory modeled so closely after a cache? What is the benefit of managing virtual memory in segments? If the memory subsystem is performing poorly, where else would you see performance problems? Why? Why are tuning parameters other than the scan rate generally not very useful?

Module 6
System Buses
April 2001
Module Overview

Objectives
Define a bus Explain general bus operation Describe the processor buses: UltraSparc Port Architecture (UPA), Gigaplane, and Gigaplane XB Describe the data buses: SBus and Peripheral Component Interconnect (PCI) Explain how to calculate bus loads Describe how to recognize problems, and configure a system to avoid bus problems List Dynamic Reconfiguration considerations
What Is a Bus?
It is a mechanism to transfer data, just like a network. It usually connects more than two components. It runs at a fixed speed. It is usually bidirectional and broadcasts messages. It is like a highway: The speed limit is a maximum, not the required speed. It has lower speed for on-and-off ramps. Higher speeds mean more traffic capacity.
General Bus Characteristics

Circuit switched (Mbus and SBus): Operates like a telephone Keeps the bus busy for the whole operation Packet switched (Gigaplane and PCI buses): Operates like a network Sends a request and then frees the bus for the next request Most buses are packet switched.

System Buses
Are used to connect sections of the system to each other: They include CPUs, peripheral controllers, and memory. Different types of system buses are produced, depending on cost and technology needs. Are simpler on smaller systems: Cost issues are important. For large systems, throughput outweighs the cost.

System Buses: MBus and XDbus

MBus For small desktop multiprocessors (sun4m) Is circuit switched Is a separate I/O bus from CPU and memory Looks like a current Pentium system design XDbus SS1000, SC2000, and CS6400 (sun4d): Is packet switched Has one on 1000, two on 2000, and four on 6400 Is easy to overload with a full configuration

System Buses: UPA and Gigaplane Bus

UPA Used by the Ultra systems (sun4u) Is a crossbar which connects CPUs, memory, and I/O Is very efficient and very fast Gigaplane Used in the Ultra servers Connects multiple UPA buses (on system boards) Is configurable while you are running the system Is what Dynamic Reconfiguration (DR) does

Gigaplane XB Bus
Is only on the Sun Enterprise 10000 Is based on the Gigaplane bus Has point-to-point connections to all 16 boards Allows addition and removal of whole boards from domains Contains multiple parallel sub-buses Can run without most of them Runs at 100 MHz Can do two operations per cycle
The Gigaplane XB Bus

System board 12
rd 1
oa mb
10
tem
boa
rd 1
d ar
te
Sys
Sys
bo
m te
Sy
Sy
st
em
bo
ar
14

Sys
Sys
System board 8
te
tem
mb oa
boa
rd 9 rd 7
Sys
te
oa mb
rd 1
System board 0
Sys
tem
boa
rd 1
Sy st e m
Sys
Sy st em
b oa r d 6
rd 5
Sys
bo
System board 4
ar
te mb oa
tem boa rd 3
d 2
Peripheral Buses
Connect peripherals to the system buses Are usually much slower than the system buses Are circuit or packet switched Can easily be the (hidden) limiting factor in I/O performance Are provided as either PCI or SBus by Sun You can mix PCI and SBus in Enterprise servers. Most new systems use PCI bus.

SBus Compared to PCI Bus

SBus runs at 2025MHz, 32-and 64-bits wide. Its peak bandwidth is 200 Mbytes/sec. PCI bus runs at 33 or 66 MHz and 32-or 64-bits wide. Peak bandwidth is 528 Mbytes/sec. It is provided on all new Sun hardware. There are many 100-Mbyte interface cards today. SOC+, SCI, ATM OC-12, Gigabit Ethernet, and so on are available.
PIO and DVMA

PIO cards require the CPU to perform all data transfer. The CPU can only move 8 or 64 bytes at a time. They consume much CPU resource and overhead. They are used for lower performance or slow cards. DVMA bypasses the CPU and goes directly to memory. It is a much more complicated setup. The break-even point with PIO is at about 2048 bytes. Many cards can do both. The driver chooses which is used (usually silently).
The prtdiag Utility

Is provided for Suns sun4d and sun4u systems Shows the hardware configuration of the system It locates configuration problems. Requires knowledge of system part identification Use the Field Engineers Handbook. Is similar to what prtconf does, but is easier to read

prtdiag CPU Section

# /usr/platform/sun4u1/sbin/prtdiag System Configuration: Sun Microsystems System clock frequency: 83 MHz Memory size: 1024 Megabytes sun4u SUNW,Ultra-Enterprise-10000
========================= CPUs ========================= Run MHz ----250 250 250 250 250 250 250 250 Ecache MB -----1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 CPU Impl. -----US-II US-II US-II US-II US-II US-II US-II US-II CPU Mask ---1.1 1.1 1.1 1.1 1.1 1.1 1.1 1.1
Brd --0 0 0 0 1 1 1 1
CPU --0 1 2 3 4 5 6 7
Module ------0 1 2 3 0 1 2 3

prtdiag Memory Section

========================= Memory ========================= Memory Units: Size 0: MB 1: MB 2: MB ------------256 256 0 256 256 0 3: MB ----0 0
Board 0 Board 1
You can determine memory interleaving from this. Proper interleaving can provide a 10 percent or more performance boost. Systems configure this automatically and silently. Some systems can do up to 16-way interleaving.

prtdiag I/O Section

========================= IO Cards ========================= Bus Type ---SBus SBus SBus SBus SBus SBus Freq MHz ---25 25 25 25 25 25 Brd --0 0 0 1 1 1 Slot ---0 0 1 0 0 1 Name -------------------------------qec/qe (network) SUNW,soc/SUNW,pln QLGC,isp/sd (block) qec/qe (network) SUNW,soc QLGC,isp/sd (block) Model ---------------------SUNW,595-3198 501-2069 QLGC,ISP1000 SUNW,595-3198 501-2069 QLGC,ISP1000
You can calculate bus loading based upon this output. You can move controllers if a bus is overloaded.

Bus Limits
A bus is like a freeway: More traffic means more congestion. Too much congestion causes traffic queues to form. When queues are large, you have major problems. Going faster takes more skill and money Bus saturation causes performance problems. You do not get any specific messages.

Bus Bandwidth
If I have checks, I must have money. If I have slots, I must have bandwidth. Both assignments might be wrong. Symptoms of this are: Bad service time Dropped network packets Intermittent hardware problems Irregular but constant problems
Diagnosing Bus Problems

If after every board has been replaced, the problem is still there: Check your configuration using prtdiag. Correlate failures with system usage. Trust the specifications, not the sales person.

Avoiding Bus Problems

Remember, you will obtain 60 percent of advertised bandwidth (at best). Watch for old devices and cards. They can silently slow a bus down. Spread things out as much as possible. Usage patterns must be considered. Remember, configuration symmetry is often a bad idea. Appearance is not important.

Avoiding Bus Problems

Know how your system works. There are many informative white papers. Be prepared to leave empty slots. Watch for unexpected problems. Understand the basics: Everything is either a bus or a cache. Be sure to tune and monitor it accordingly. Smarter is better than bigger.
SBus Overload Examples

Enterprise XX00 server Gigaplane bus
UPA
SBus A (200 Mbytes/sec)
SBus B (200 Mbytes/sec)
12 100T
20 FW SCSI
? Slot 1
? Slot 2
? Slot 3
25
25
Fibre Channel

System Bus Overload Examples

SPARCcenter 2000E 20 300 Mbytes/sec buses CPU 2030 Mbytes/sec
10 Memory 5 Gbytes in 20 256-Mbyte banks
SBus
50 Mbytes/sec

CPU Load on the System Bus

SPARCcenter 2000E 20 CPU L2 Cache 2030 Mbytes/sec
300 Mbytes/sec buses
10 Memory 5 Gbytes in 20 256-Mbyte banks
SBus
50 Mbytes/sec

Dynamic Reconguration Considerations

Watch for problems that occur when you add boards, not when you remove them Make sure you have bandwidth when you add components Be careful replacing slower cards or components Consider reorganizing to spread the load Be careful of controller and component renumbering Plan ahead

Tuning Reports
Not much is available in this area. Vigilance is the best prevention. When you add new cards, be sure to check prtdiag. Remember that empty slots do not mean available bandwidth. Strange behavior in tuning reports might be related to bus problems. It usually occurs when the system is most heavily loaded.

Bus Monitoring Commands

prtdiag Shows system bus clock speed plus number and type of I/O buses on sun4[du] systems busstat Solaris 8 OE bus-related monitor prtconf Use v and p options for details of hardware OBP Use OpenBoot PROM firmware commands Sun Management Center Graphical tool

Check Your Progress

Define a bus Explain general bus operation Describe the processor buses: UPA, Gigaplane, and Gigaplane XB Describe the data buses: SBus and PCI Explain how to calculate bus loads Describe how to recognize problems, and configure a system to avoid bus problems List Dynamic Reconfiguration considerations

Think Beyond
Why are there so many different types of buses? Why does not every new system from Sun use the Gigaplane XB bus? Why do some systems use several different buses to connect their components? How do you tell if a device is using Programming I/O (PIO) or Direct Virtual Memory Access (DVMA)? How could you confirm whether you have a hardware or a configuration problem?

Module 7
I/O Tuning
April 2001
Module Overview

Objectives
Demonstrate how I/O buses operate Describe the configuration and operation of a Small Computer System Interface (SCSI) bus Explain the configuration and operation of a Fibre Channel bus Demonstrate how disk and tape operations proceed Explain how a storage array works List the Redundant Array of Independent Disks (RAID) levels and their uses Describe the available I/O tuning data
SCSI Bus Overview

Was originally shipped in 1982; SCSI-1 in 1986 Is standardized now at SCSI-3 Upward compatible evolution Supports disk, tape, network devices, media changers, CD-ROMs, and so on Is the most common computer peripheral connection Defines message traffic and bus operation Uses principles that apply to most I/O buses

SCSI Bus Characteristics

There are several types of SCSI buses. They are distinguished by their mix of characteristics: Speed How fast the signals move Width How much data moves at a time Length How long the bus can be Buses can support several different sets of characteristics concurrently. Device drivers negotiate a devices behavior on the bus. It behaves like a packet-switched bus.
SCSI Speeds
Name Asynchronous Synchronous Fast Ultra 20/40/80 Maximum Speed 4 Mbytes/sec 5 Mbytes/sec 10 Mbytes/sec 20, 40, and 80 Mbytes/sec
The proceding values relate to parallel SCSI. Using Fibre Channel, Ultra 320 can transfer up to 320 Mbytes/sec.

SCSI Widths
Name Narrow Wide Width 1 byte 2 bytes in parallel Cable 50-conductor cable 68-conductor cable
Speed and width together determine the bandwidth. Multiply speed by width

SCSI Lengths
Name Single-ended Differential Limit 20 feet, 6 m Sync 66 feet, 20 m Fast 82 feet, 25 m Symbol
Subtract 18 inches or 0.5 meters (m) for each device on the bus Do not mix differential and single-ended devices on the same bus. You could damage the equipment.

SCSI Properties Summary

Protocol Speed Async Fast Sync Fast-20 Width Narrow Wide Length Single-ended Differential
Each device runs with one combination. Choose one from each column Multiple combinations are supported on one bus. Do not mix single-ended and differential devices.

SCSI Interface Addressing

A SCSI target is a device controller, not a device. There are eight unique targets on a narrow bus, 16 on a wide. Each target supports eight LUNS, or devices. Embedded SCSI has one LUN per target. The tXdX scheme identifies these targets and LUNS. SCSI commands are issued by initiators. The most common initiator is the host adapter. Host adapters do not have LUNS.
SCSI Bus Addressing

ESP Host adapter target 7 Disk, target 0, LUN 0 Disk, target 1, LUN 0 Disk, target 2, LUN 0 Disk, target 3, LUN 0
SBE/S
CD-ROM, target 6, LUN 0
Tape, target 5, LUN 0
Tape, target 4, LUN 0

SCSI Target Priorities

The SCSI bus needs an arbitration scheme. This determines which device goes first. Higher priority devices get better service. On a narrow bus, the priorities run 70. On a wide bus, they run 70 then 158. The host adapter is usually 7. Multi-initiator SCSI allows multiple initiators. The host adapters are numbered 7 and 6.

Fibre Channel
Is a reliable, scalable, gigabit interconnect technology Combines the best of channels and networks into a single I/O interface Operates on both fiber-optic cable and copper wire Supports a variety of popular protocols: SCSI IP ATM IEEE 802.5 - Token Ring
Fibre Channel
High-speed HA clusters
FC-AL disk farm
FC-AL loops FC switch creates fabric FC hub Point-to-point ber connection
FC hub
Individual localized FC-AL loops
SSA-SCSI disk arrays

Disk I/O Time Components

Access to the I/O bus Other queued disk requests Bus transfer time Seek time (to find correct track in cylinder) Rotation time (to find correct sector in track) Internal throughput rate (ITR) transfer time Reconnection time Interrupt time

Disk I/O Time Calculation

For a 9-ms, 7200-RPM disk device Queue time OS ? Cmd Data transfer Arb. trans. to cache 1.5 ms 1 ms (or less) Seek 9 ms Internal Rotation transfer 4 ms 2 ms Arb. status 1.5 ms ? Interrupt driver
Bus
Device
Service time Response time

Disk Drive Features

Multiple zone recording Drive caching and fast writes Mode pages Tagged queueing Properties from prtconf and scsiinfo

Multiple Zone Recording

Zone table Sector 0 350,000 685,000 Density Cylinder 75 70 65 60 55 ... 0 88 176
Zone 0 Zone 1 Zone 2 Zone 3

Drive Caching and Fast Writes

Disk drives today have from 0.5 Mbyte to 8+ Mbytes of cache. Cache is used with tagged queueing to manage requests. It also buffers the drives ITR to bus speed. Fast writes signal completion when the data reaches the cache. You should have the drive on an uninterruptable power supply (UPS) to use fast writes. It can be set in arrays or for individual drives.
Tagged Queueing
Adds a special ID field to the device command Allows the drive to handle multiple simultaneous requests Sometimes executes requests out of order Takes advantage of ordered requests to limit arm motion Is negotiated with the device by the driver Not all devices support it Can provide noticeable performance improvements

Mode Pages
Are provided for all SCSI devices Identified by number Describe device properties and configuration Have several standard, required pages Have additional device-specific pages Can include vendor-specific pages Can be viewed using format -e Have header files in /usr/include/sys/scsi

Device Properties From prtconf

prtconf -v reports some device information.
name <target1-TQ> length <4> value <0x00000001>. name <target1-wide> length <4> value <0x00000001>. name <target1-sync-speed> length <4> value <0x00004e20>.
What is reported depends on the device. Device properties are listed under the controller. This device is using a fast-wide SCSI bus and tagged queueing.
Device Properties From scsiinfo

scsiinfo can read mode page properties and device characteristics. It is not part of SunOS. Available from ftp://ftp.cs.toronto.edu/pub/jdd/scsiinfo/ Provided in source code scsiinfo is the easiest way to see how your SCSI devices are operating.

I/O Performance Planning

Disk Elite 3 Elite 9 SSA RSM drive Baracuda 4 Baracuda 9 Sequential Reads Mbytes/Sec 4.5 5.5 5.5 6.5 7.5 Random Reads Operations/Sec 60 60 60 75 75
Controller ISP SOC

Sequential Reads Mbytes/Sec 14 18
Random Reads Operations/Sec 1700 2400

Tape Drive Operations

Bus speed is a speed limit, not a mandatory speed. Tapes can seriously hurt your bus performance. They transfer very slowly. Other devices have to wait until the tape is done. Tapes should be on a separate bus from production disks. Backups can seriously degrade bus performance.

Storage Arrays and JBODs

Just a bunch of disks (JBOD, A5000, MultiPack) Looks like a bus in a box Has no controller and no special control operations Might have special power and reliability support Storage arrays (SPARCstorage Array, A7000) Has a controller and cache in front of multiple buses Must be slower than plain disks if the cache fills Provides many additional features

Storage Array Architecture

Host interfaces
Cache
CPU
Controller
SCSI bus controllers Drives

RAID
Stands for redundant array of independent disks Combines multiple disk drives into a single logical unit Overcomes inherent limitations of a single disk drive Is designed to enhance both reliability and performance Six RAID levels were defined in the original UCBerkeley papers Has different levels for different environments

RAID-0
RAID-0 is striping (spreading data cross multiple drives). RAID-0 offers no data protection or redundancy. It is effective in both sequential access and random access environments. You can distribute member disks across multiple controllers.

RAID-0 Stripe Sizing

You should tune the stripe size according to the data access pattern. You must keep in mind SCSI overhead when selecting stripe size. Smaller stripe size over many controllers/disks favors multiple, random read/write operations (Web servers, NFS servers). Larger stripe size over fewer disks favors single, intensive read/write operations (databases). Match the stripe size to the application read/write size.

RAID-1
RAID-1 is mirroring (copying data to multiple locations). RAID-1 provides data redundancy. Multiple controllers are necessary for performance. Read workload can be distributed for better performance. Writes are serviced in parallel. Typical writes to mirrors take only about 15 to 20 percent longer than writes to individual disks. The main drawback is the cost of extra disks for mirrors.
RAID-0+1
RAID-0+1 is mirrored stripes. RAID-0+1 provides the performance of RAID-0 and the redundancy of RAID-1. It is often implemented in hardware using smart mirroring controllers. Sequential reads improve through striping. Random reads improve through the use of multiple disks. Writes are about 30 percent slower than RAID-0 writes.

RAID-3
RAID-3 is striping with a dedicated parity disk. Parity bits or bytes are computed from the data. The parity information is stored on a separate disk. Data can be reconstructed from the parity information. It performs well when servicing large quantities of single-stream sequential I/O. It does not perform well for random access I/O because of the physical organization of the RAID-3 stripe. Its reads perform as well as RAID-0 reads because parity data does not have to be accessed.
RAID-5
RAID-5 is also striping with parity. Parity information is striped across all member disks. It eliminates the single parity disk bottleneck of RAID-3. It is optimized for random access I/O patterns. Its writes use a two-phase commit protocol for data integrity. Nonvolatile memory is frequently used to accelerate the write operations.

Tuning the I/O Subsystem

Remember, The fastest I/O you can do is the one you do not do. Main memory problems can cause I/O problems. Start by tuning main memory and UFS caches. Spread activity across boards, buses, and devices. Ensure devices are operating as they are supposed to. Calculate good I/O times. Eliminate hot spots. Use target and device priorities to place data.
Available Tuning Data

iostat sar scsiinfo and prtconf

Available I/O Statistics

Activity The sample time disk was active The queue wait time The device service time The number of reads / writes The number of blocks read and written per second The percentage of time the system is idle waiting for I/O from disk The percentage of time the system is idle waiting for I/O from any disk
Source (calculated)
iostat (unit) %b
sar (unit) %busy avwait (ms)
rtime reads writes nread nwrite
svc_t(ms) r/s w/s Kr/s (Kbytes) Kw/s (Kbytes) %w
avserv (ms) r+w/s blks/s (sectors)
%wio
Available I/O Data

ks_disks.nread The number of bytes a device has read ks_disks.nwritten The number of bytes a device has written ks_disks.reads The number of read operations ks_disks.rlentime The average length of time a transfer request takes to run (actual device service time) ks_disks.rtime The amount of time the device is active ks_disks.wlentime The average length of time a transfer request has to spend waiting to be serviced (pre-interface time) ks_disks.writes The number of write operations
Check Your Progress

Demonstrate how the I/O buses operate Describe the configuration and operation of the SCSI bus Explain the configuration and operation of a Fibre Channel bus Demonstrate how disk and tape operations proceed Explain how a storage array works List the RAID levels and their uses Describe the available I/O tuning data

Think Beyond
Why are there only eight LUNS on a SCSI target? What would happen if the SCSI initiator was not the highest priority device on the bus? How could you tell? What devices besides a disk drive might tagged queueing benefit? What special considerations are there in looking at tuning reports from a storage array?

Module 8
UFS Tuning
April 2001
Module Overview

Objectives
Describe the vnode interface layer to the file system Explain the layout of a UNIX File System (UFS) partition Describe the contents of an inode and a directory entry Describe the function of the superblock and cylinder group blocks Understand how allocation is performed in the UFS List three differences between the UFS and Veritas File Systems
Objectives
List the UFS-related caches Tune the UFS caches Describe how to use the UFS tuning parameters

The vnode Interface

System calls
vnode layer
/proc PCFS
HSFS
UFS swapfs TMPFS NFS cachefs
Process CD-ROM address Diskette space
Disk
Anonymous memory

The Local File System

A disk drive must be divided into one or more partitions or slices. Each partition can contain only one file system. Files are accessed in fixed-length blocks on the disk. Four-Kbyte and 8-Kbyte blocks are available Four-Kbyte blocks are not supported for Ultra systems Information about a file is kept in an inode. A directory points to an inode for every named object in a partition.
Fast File System Layout

Boot block Superblock BU superblock Cylinder group block inodes Label
Cylinder group 0
Data
Data BU superblock Cylinder group block inodes
Cylinder group 1
Data

The Directory
inode number Entry length Name length Name inode number Entry length Name length Name 0 Entry length Name length Name Deleted entry

Directory Name Lookup Cache (DNLC)

Searching for a name in a directory is expensive. DNLC is a cache of the most recently referenced directory entry names and their associated vnodes. It is used by UFS and NFS. ncsize determines the size of DNLC. vmstat -s and sar display DNLC statistics.

The inode Cache

Contains an entry for every open inode Holds as many closed inodes as possible Locates entries through the DNLC Grows in size to hold all open files Can exceed your setting If it does exceed your setting, closed (unused) inodes are flushed When closed inodes are cached, reuses their data pages If flushed (removed from cache), loses the pages
The inode Cache

inode cache size is set by ufs_ninode. You can use netstat -k to display inode cache statistics. maxsize is the variable equal to ufs_ninode. A maxsize_reached value higher than maxsize indicates the number of active inodes has exceeded cache size. In this case, you should increase ufs_ninode. You can add line to /etc/system. set ufs_ninode=50000
DNLC and inode Cache Management

DNLC and inode caches should be the same size. Each DNLC entry has an inode entry. Default sizes are 4 x (maxusers + max_nprocs) + 320. inode cache size is set by ufs_ninode. DNLC cache size is set by ncsize. Set them high (50,000 or more) on NFS servers Try to keep the DNLC hit rate above 90 percent

The inode Structure

inode type and permissions Number of hard links Owner and group ID Size in bytes Time file last accessed Time file last modified Time inode last modified Twelve direct block pointers Three indirect block pointers Number of sectors Shadow inode location

The inode Block Pointers

12 direct data Block pointers Data block Data Data Single indirect Double indirect Triple indirect Data Data Addresses 2048 2048 Addresses Addresses 2048 2048 2048 Data 2048 Data Addresses 2048 Data Data Data Data
addresses
2048
Block
Addresses
2048
The pointers are sector addresses.

UFS Buffer Cache

Holds inodes, cylinder group blocks, and indirect blocks only Has bufhwm as the only parameter specified in Kbytes A default of 0 allows up to 2 percent of main memory to be used. This might be too much on very large systems. You can usually reduce it to a few Mbytes. Uses sysdef to determine the current setting

Hard Links
Directory entry
inode number
Directory entry
inode number
Name
Name inode ic_nlink = 2
Work only in the same partition Have no limit to the number Have no back pointers Use find -inum to locate the other names

Symbolic Links
Regular symbolic link
inode number Entry length Name length Name
Fast symbolic link /usr directory entry

inode number
12 4 t p m \0
inode Link count 1033 0
/usr/tmps inode Link count 543 . . / v a r / t m p
Data block 1033 /preposterously_long_directory_name /extremely_long_filename Data block 543 ../var/tmp

Allocating Directories and inodes

d d i b b d i i b bb i d i ii b b b d d i i i bb b b d d d i i i bb b b d i i bb b
UFS tries to provide even performance regardless of partition utilization. Directories are allocated in a cylinder group (cg) containing a larger than average number of free inodes. Ties are broken by the fewest directories. File inodes are allocated to the same cg as their directory. Data blocks are allocated in the same cg as their inode.
The Veritas File System (VxFS)

Is based on the allocation of extents Is biased towards the storage of larger files Extents are referenced by a pointer and size-offset value
VxFS Inode Extent Identifier Pointer 0 / Size Pointer 1 / Size Pointer 2 / Size Pointer 3 / Size Pointer 4 / Size Pointer 5 / Size Pointer 6 / Size Pointer 7 / Size Pointer 8 / Size Pointer 9 / Size Single Indirect Double Indirect
DiskBlock DiskBlock DiskBlock DiskBlock Extent-1 DiskBlock DiskBlock Extent-2 DiskBlock DiskBlock

Comparison of UFS and VxFS

Each file system has specific benefits Apply the file system depending on the storage requirement
UFS Better suited to the storage of large numbers of small les in many directories Data blocks are stored across cylinder groups in the entire partition Logging is available Slow boot time due to fsck activity over many cylinder groups VxFS Better suited to the storage of small numbers of large les in few directories Data blocks gradually ll up the partition in contiguous extents Uses the intent log as standard Fast boot time due to availability of the le system log

Allocation Performance
(For illustrative purposes only) VxFS
Access performance
UFS
Compacted UFS Faster 0 Partition percentage full 100

Allocating Data Blocks

Data block allocation is a ve-step process. 1. Request the ideal block. 2. Take a block in the cylinder group. 3. Perform a quadratic rehash. 4. Look at all remaining cylinder groups. 5. Panic the system.

Quadratic Rehash
Tries to avoid clumps of full cylinder groups Searches twice as far each time another cg is found full Stops quickly, which is important for performance Reserves free space to try to stop quickly This is the 10 percent reserved seen in df. The formula is (64 Mbytes file system size) x 100. It runs from 10 percent (disk storage <= 640 Mbytes) to 1 percent (disk storage>= 6.4 Gbytes).

Fragments
A full 8-Kbyte block is a waste for small files. Fragments use 1/8 block increments to extend a file. Fragments can be allocated by best-fit or full block. Fragment rules are: Only the last block of the file can be fragmented. Files larger than 96 Kbytes cannot use fragments. Fragments must be contiguous within their block. Fragments can be moved to remain contiguous.

Logging File Systems

Allocation requires lots of I/O. Cylinder group block, inode, and superblock updates are needed for one data block allocation. Logging file systems write to the log; other writes are done later. This takes advantage of write cancellation. It has faster recovery because all changes are logged. The log is usually a disk hot spot. Some examples are VxFS and SunOS 5.8 UFS with log.
Application I/O
Application I/O usually goes through the segmap cache. The segmap cache is a kernel memory segment. The segmap cache is from 4 to over 90 Mbytes in size. It is a fully associative, write-back cache. The segmap cache is at the same kernel virtual address for every process. The segmap cache contents differ depending on the processs file usage pattern. Blocks in the cache might not be in memory.
Application I/O
segmap cache bread bwrit
Disk
Kernel User
read() write()
There is one system call and (at least) one copy per request.

Working With the segmap Cache

You can bypass the cache by using direct I/O or mmap. I/O goes directly to and from a disk. There are restrictions. You can make the cache write-through using O_SYNC and O_DSYNC on the open call. You can continue to execute during I/O operations using async I/O. This allows non-blocking application I/O.

File System Performance Statistics

From sar -a, -g, and -v: namei/s DNLC path name component requests iget/s DNLC cache misses dirbk/s Directory block reads per second ufs_ipf Flushed inodes: percentage of (closed) UFS inodes flushed from the cache that had reusable file pages still in memory file_sz The total number of open files in the system inode_sz The current percentage of inode cache in use
The fsflush Daemon

Remember that the segmap cache is a write-back cache. Data could stay there unwritten indefinitely, which is not good in case of a system crash. fsflush writes modified data back to disk on a regular basis for UFS. There are two tuning parameters: tune_t_fsflushr How often to wake up autoup How long a cycle should take autoup is the longest that unmodified data will remain unwritten.
Direct I/O
Allows I/O to bypass the segmap cache Works best when files are much larger than memory or data is already buffered Allows regular file system use in place of raw partitions Is enabled by: Application use of directio call forcedirectio mount option Has restrictions on its use

Accessing Data Directly With mmap

Process Stack
paddr
Mapping len
off
Data Text
File (or file-like object)

Using the madvise Call

Controls more precisely an applications use of main storage. It can be used for most virtual memory areas. MADV_NORMAL Normal (read ahead only) access. MADV_SEQUENTIAL Sequential access (read ahead and free behind). MADV_RANDOM Random access; no read ahead. MADV_WILLNEED Virtual memory that the application intends to use soon. The data will be read in by read ahead. MADV_DONTNEED Unneeded virtual memory. Any associated physical memory will be freed.
File System Performance

The workload characteristics of the application must be understood before configuring a file system. Data-intensive applications access large files without creating or deleting files. Attribute-intensive applications create and delete many small files. Decision support or data warehousing has sequential access I/O patterns. Transaction processing has random access I/O patterns.

Cylinder Groups
The size of the cylinder group is dependent on the type of application running on the file system. The goal is to keep files in a single cylinder group. The default of 16 cylinders per group is sufficient for attribute-intensive applications. You should consider increasing the number of cylinders per group to 256 (maximum) for dataintensive applications. The size of a cylinder group varies depending on the size of the disk partition.

Sequential Access Workloads

This typically occurs when moving large amounts of data. The truss command can be used to investigate the nature of an application. Absence of lseek system calls between each read indicate that the application is reading sequentially. The first read system call requires a physical disk read. The first file system block, 8 Kbytes, is read into RAM. Subsequent read calls get data directly from memory. A read-ahead algorithm improves the performance of sequential reads.
UFS File System Read Ahead

The UFS file system determines when to implement read ahead. Several criteria must be met before engaging read ahead. Memory-mapped files do not use UFS read ahead. Cluster size (maxcontig parameter) determines number of blocks to read ahead. maxcontig defaults to 16 8-Kbyte blocks (128 Kbytes). Default value is often too low to allow optimal read rates. maxcontig can be viewed by fstyp -v command.
Cluster Sizes for RAID Volumes

Cluster sizes larger than 128 Kbytes require the maxphys parameter to be set in /etc/system. The guidelines for cluster sizes on RAID devices are as follows: RAID0 striping The cluster size is equal to the number of stripe members multiplied by the stripe size. RAID1 mirroring The cluster size is the same as for a single disk. RAID0+1 striping + mirroring Cluster size is equal to the number of stripe members per mirror multiplied by the stripe size.
Commands to Set the Cluster Size

Use newfs -C # to set the cluster size (where # is the number of 8-Kbyte blocks) newfs -C 112 /dev/rdsk/xxx creates a file system with a cluster size of 896 Kbytes. Use tunefs -a # to modify the cluster size for an existing file system (where # is the number of 8-Kbyte blocks) tunefs -a 112 /dev/rdsk/xxx changes the cluster size to 896 Kbytes. Remember, for RAID devices, maxcontig should be set so that the cluster size is the same as the stripe width.
File System Write Behind

Delayed asynchronous writes is the UNIX file systems default method for writing data blocks. This allows the application to continue to process without having to wait for each I/O. This also allows the operating system to delay the writes long enough to group together adjacent writes. The maxcontig parameter controls how many writes are grouped together before a physical write is performed. Guidelines used for read ahead should also be applied to write behind.
UFS Write Throttle

Prevents the file system from flooding system memory with outstanding I/O requests Uses three parameters to control the throttling mechanism: When ufs_WRITES is set to 1 (the default), the UFS write throttle is enabled. Writes are suspended when the number of outstanding writes is larger than ufs_HW and are resumed when the number of writes falls below ufs_LW. Uses default parameters to prevent full sequential write performance of most disks
Random Access Workloads

Typically this occurs when small amounts of data are being moved. Online Transaction Process (OLTP), NFS servers with many small files, and ISP servers typically have random access workloads. You can use the truss command to investigate the nature of an application. The presence of a lseek system call between each read indicates the application is reading randomly. The third argument of read system call shows block size.
Tuning for Random I/O

Try to match the I/O size and the file system block size Choose an appropriately large file system cache Disable prefetch and read ahead, or limit read ahead to the size of each I/O Disable file system caching when the application does its own caching When configuring a RAID device for random access workloads, use many small disks Set the stripe size of a RAID device small to reduce latency and the number of physical blocks transferred
Post-Creation Tuning Parameters

Parameters on tunefs (and newfs/mkfs) are: Optimization for time or space: time Allocates a new block for each fragment section space Allocates fragments best-fit minfree Reserves space to cut quadratic rehash short: Formula is: (64 Mbytes file system size) x 100 Default is 10 percent maxbpg - Helps keep large files more contiguous
Application I/O Performance Statistics

From sar -b, you can get segmap and buffer cache behavior. bread/s, bwrit/s segmap cache I/O operations. lread/s, lwrit/s Buffer cache I/O operations. %rcache, %wcache segmap cache hit ratio. pread/s, pwrit/s phsyio() transfers. physio() is used when accessing raw devices.

Check Your Progress

Describe the vnode interface layer to the file system Explain the layout of a UFS partition Describe the contents of an inode and a directory entry Describe the function of the superblock and cylinder group blocks Understand how allocation is performed in the UFS List three diffences between the UFS and Veritas File Systems.

Check Your Progress

List the UFS-related caches Tune the UFS caches Describe how to use the UFS tuning parameters

Think Beyond
What will the effect on UFS be if there is not enough main storage? Why does the UltraSPARC not support the 4-Kbyte UFS block size? When would the use of a logging file system be critical to performance? What are the most expensive UFS operations? Why must they always be used?

Module 9
Network Tuning
April 2001
Module Overview

Objectives
Describe the different types of network hardware List the performance characteristics of networks Describe the use of Internet Protocol (IP) trunking Discuss the NFS performance issues Briefly describe the functions, features, and benefits of the CacheFS file system Discuss how to isolate network performance problems Determine what can be tuned
Networks
Networks are generally packet-switched buses. There are several places to tune: The application The OS The hardware The network itself You tune differently depending on your workload.

Network Bandwidth
Network Connection 28800 baud modem ISDN 1 BRI ISDN 2 BRI T1 E1 Ethernet T3 FastEthernet GigabitEthernet Full Duplex FastEthernet FDDI ATM (OC3) ATM (OC12) Bandwidth 28.8 Kbits/sec 64 Kbits/sec 128 Kbits/sec 1.544 Mbits/sec 2.048 Mbits/sec 10 Mbits/sec 45 Mbits/sec 100 Mbits/sec 1000 Mbits/sec 100+100 Mbits/sec 100 Mbits/sec 155+155 Mbits/sec 622+622 Mbits/sec

Full Duplex 100 BASET Ethernet

Fast Ethernet adapter autosenses the network characteristics. It automatically determines how the network is operating. Sometimes the adapter makes mistakes. Speed is usually correct, duplexing is sometimes wrong. You can use ndd to inspect and change the adapters operation.

IP Trunking
Is like network striping Data is striped over multiple network interfaces. Allows multiple interfaces to look like one Works only with qfe cards The 100 BASET Quad Fast Ethernet card Can combine two or all four interfaces Usually goes to an Ethernet switch Requires Sun Trunking software

TCP Connections
Transmission Control Protocol (TCP) connections are expensive. TCP is optimized for reliable data on long-lived connections. Making a connection uses a lot more CPU resources than moving data. Connection setup protocol involves several round-trip delays. Pending connections can cause listen queue issues. Each new connection goes through a slow start as reliability is determined.
TCP Stack Issues

Warn if retransmit rate is over 15 percent. It is a problem if rate is over 25 percent. Warn if incoming duplicate packets are more than 15 percent. It is a problem if duplicate packets are at 25 percent; the remote system is retransmitting to you. Warn if listen drops are seen. It is a problem if drops are more than 0.5 per second. Warn if outgoing connection fails over 2 per second.
TCP Tuning Parameters

tcp_time_wait_interval The default 240,000 ms is reduced to 60,000 for SPECweb96 runs. tcp_conn_req_max_q0 Incomplete listen connection limit The default is 1024; no need to tune unless tcpListenDropQ0 is nonzero.

TCP Tuning Parameters

tcp_conn_req_max_q Pending completed connection limit Default is 128; no need to tune it unless tcpListenDrop is seen. tcp_slow_start_initial The number of initial packets to send The default is 1, 2, and 4 for Solaris 2.6, Solaris 7, and Solaris 8 OEs respectively. tcp_xmit_hiwat and tcp_recv_hiwat Window size The default is 8192; try setting it to 32768 if using fast Wide Area Network (WAN) links.
Using the ndd Command

Network driver parameters can be listed and changed with ndd. ndd can be run by anyone, but changes can be made only by the root user. Changes last until reboot. Permanent changes can be made through /etc/system. To see available parameters for a driver use: ndd /dev/tcp \? Any network driver can be specified.
Network Hardware Performance

Networks are packet-switched buses. They have identical performance issues. Access to the wire is important, so low utilization is essential. Collisions imply there is not enough available access. You can cut traffic or increase the network speed. You might have a lot of unnecessary traffic. NFS attribute messages may be unnecessary.

NFS
Tune the host before attempting to tune NFS, itself Remove any non-NFS-specific performance issues Increase the size of the DNLC and inode cache Solve server I/O and memory performance issues Resolve network performance issues Make sure actimeo is not too low Use cachefs Use NFS version 3; some protocol efficiencies were added
NFS Client Retransmissions

If NFS server cannot respond to client request, the client retransmits. The retransmission rate is the major factor affecting NFS performance. nfsstat -c displays client retransmissions (retrans). If you have a high retransmission rate, check for: Overloaded servers Ethernet interface dropping packets Network congestion

NFS Server Daemon Threads

I/O requests are handled by server system threads. The default number of threads is 16 This is specified in /etc/init.d/nfs.server: It must be adjusted based on server configuration. You might need several hundred threads. A common cause of slow NFS servers is too few I/O daemon threads.

The Cache File System

Use cachefs to reduce the volume of NFS server access
NFS server
Client requests data Server transmits data
NFS client
Client caches data on local storage

CacheFS Characteristics
File data is cached. File attributes are cached. Directory contents are cached. Symbolic links are cached. Local disk resources are reclaimed on a least recently used basis. Cache content consistency is periodically verified.

CacheFS Benets and Limitations

Benefits include: Improved client performance Increased server capacity Increased network capacity Limitations include: No benefit for write mostly file systems No benefit for read once file systems No benefit if data is frequently modified on the server
Monitoring Network Performance

Commands available for monitoring network performance include: ping Verifies connectivity to hosts on network spray Tests the reliability of packet sizes snoop Captures packets and traces calls netstat Displays network status nfsstat Displays NFS server and client statistics traceroute Displays the route packets take to a network host

Isolating Problems
Check the four sources of problems: Client Server Application Network Ensure that the individual pieces are tuned, and then look at the network and application specifically.

Tuning Reports
netstat: Provides detailed physical traffic and error information Shows detailed IP protocol-level message traffic nfsstat: Provides detailed data transfer information Gives detailed information for all NFS messages Report meanings can change based on the network topology and protocols used.

Check Your Progress

Describe the different types of network hardware List the performance characteristics of networks Describe the use of IP trunking Discuss the NFS performance issues Briefly describe the functions, features, and benefits of the CacheFS file system Discuss how to isolate network performance problems Determine what can be tuned
Think Beyond
What other similarities are there between system buses, the SCSI bus, and networks? Why does the TCP/IP stack have so many tuning parameters? Why is tuning the NFS client and server more likely to improve performance than tuning the network? How can you be sure that you had a network problem and not a system issue?

Module 10
Performance Tuning Summary
April 2001
Module Overview

Objectives
List four major performance bottlenecks Describe the characteristics and solutions for each Describe several different ways to detect a performance problem Identify the type of performance problem that you have, and what tools are available to resolve it

Some General Guidelines

Tune one item at a time Spend the most time tuning the things which have the largest effect; expect trade offs Realize one size does not fit all What works on a small system might not work on a larger system. Many parameters have to be scaled or adjusted. Do not spend money first

What You Can Do

Use accounting to see what is being run Start accounting during boot Use adm crontab to process it Collect performance data regularly with the sys crontab entry Generate special purpose measurements with sar, vmstat, and so on Graph the data to see exceptions and trends Consider using three-dimensional graphs

Application Source Code Tuning

It is usually much more productive to tune applications than to change OS parameters. (But do both.) Application performance is affected by: Programming model Algorithms Choice of programming language Effective use of provided system functions Compiler and optimizations

Compiler Optimization
Can provide performance improvements of several hundred percent Is usually not done by the programmer Makes programs a bit harder to debug Provides even better optimization for specific CPUs Uses the basic compile flags -xO3 -xdepend Compiles will run longer Uses optimized libraries, if available

Application Executable Tuning Tips

Use accounting to see what is using resources. Trace system calls to see what is going on. Use truss, apptrace, or prex to view application activity at a system level. Use cachefs to increase the speed of NFS reads. Set the proper values for your basic tuning parameters.

Performance Bottlenecks
Memory bottleneck I/O bottleneck CPU bottleneck Network bottleneck

Memory Bottlenecks
Memory bottleneck detection
Symptom There is steady pageout activity. Scan rate is nonzero (page daemon is active). Swapper is active.

Memory Bottleneck Solutions

Adjust paging parameters such as lotsfree Use priority paging Use shared libraries Use madvise to use memory more efficiently Analyze locality of reference in applications Set memory usage limits with ulimit, limits, and setrlimit(2) Modify the process load, if possible Add more physical memory
I/O Bottlenecks
I/O bottleneck detection
Symptoms There is an uneven workload. Many threads are blocked waiting on I/O. The disk utilization rate is high. You have active disks with no free space.

I/O Bottleneck Solutions

Adjust the DNLC and inode cache sizes Set appropriate UFS partition parameters Balance your disk load Use RAID-0 (striping) and proper disk/bus balance Use mmap or direct I/O instead of plain read and write Use shared libraries and compiler optimization Organize I/O requests to be more contiguous Move slow devices off of production I/O buses Add more or faster disks or buses
CPU Bottlenecks
CPU bottleneck detection
Symptoms CPU idle time is low. Threads are waiting on run queue. There is lower response time or interactive performance.

CPU Bottleneck Solutions

Modify applications to use system calls more efficiently Use mmap instead of read and write Use compiler optimization Use processor sets Use priocntl and nice to modify process priorities Modify the dispatch parameter tables Check old custom device drivers for PIO usage Modify or limit the process load Add more or faster CPUs
Ten Tuning Tips

1. Remember the interrelationships in the system. The source of a problem is not always obvious. 2. Sustained high scan rates can indicate a memory shortage. 3. Ignore free memory numbers; they should be about lotsfree. 4. Ignore paging I/O statistics. 5. Always check for disk bottlenecks.

Ten Tuning Tips

6. Use nfsstat -c on slow NFS clients to find the source of the problem. 7. Be prepared to alter the workload characteristics. 8. Graph your data to see exceptions quickly. 9. Base your tuning on the characteristics of the component (bus or cache). 10. Things change: check patches and new release documentation.

Check Your Progress

List four major performance bottlenecks Describe the characteristics and solutions for each Describe several different ways to detect a performance problem Identify the type of performance problem that you have, and what tools are available to resolve it

Think Beyond
What will you look at first when a performance problem is reported? Why? What type of problems do large applications pose for system tuning? How do you know when your system is well tuned? What types of performance problems might your systems have that require special monitors or tuning approaches?

Copyright 2001 Sun Microsystems Inc., 901 San Antonio Road, Palo Alto, Californie 94303, U.S.A. Tous droits rservs. Ce produit ou document est protg par un copyright et distribu avec des licences qui en restreignent lutilisation, la copie, la distribution, et la dcompilation. Aucune partie de ce produit ou de sa documentation associe ne peut tre reproduite sous aucune forme, par quelque moyen que ce soit, sans lautorisation pralable et crite de Sun et de ses bailleurs de licence, sil y en a. Des parties de ce produit pourront tre drives du systme UNIX licenci par Novell, Inc. et du systme Berkeley 4.3 BSD licenci par lUniversit de Californie. UNIX est une marque enregistre aux Etats-Unis et dans dautres pays et licencie exclusivement par X/Open Company Ltd. Le logiciel dtenu par des tiers, et qui comprend la technologie relative aux polices de caractres, est protg par un copyright et licenci par des fournisseurs de Sun. Sun, Sun Microsystems, le logo Sun, CacheFS, Forte, Open Boot, Solaris, Solaris Resource Manager, SunOS, Sun Enterprise SyMON, Sun Trunking, Sun Management Center, Sun Enterprise 10000, et Ultra sont des marques dposes ou enregistres de Sun Microsystems, Inc. aux Etats-Unis et dans dautres pays. Toutes les marques SPARC, utilises sous licence, sont des marques dposes ou enregistres de SPARC International, Inc. aux Etats-Unis et dans dautres pays. Les produits portant les marques SPARC sont bass sur une architecture dveloppe par Sun Microsystems, Inc. Les interfaces dutilisation graphique OPEN LOOK et Sun ont t dveloppes par Sun Microsystems, Inc. pour ses utilisateurs et licencis. Sun reconnat les efforts de pionniers de Xerox pour la recherche et le dveloppement du concept des interfaces dutilisation visuelle ou graphique pour lindustrie de linformatique. Sun dtient une licence non exclusive de Xerox sur linterface dutilisation graphique Xerox, cette licence couvrant aussi les licencis de Sun qui mettent en place linterface dutilisation graphique OPEN LOOK et qui en outre se conforment aux licences crites de Sun. Le systme X Window est un produit de X Consortium, Inc. CETTE PUBLICATION EST FOURNIE EN LETAT SANS GARANTIE DAUCUNE SORTE, NI EXPRESSE NI IMPLICITE, Y COMPRIS, ET SANS QUE CETTE LISTE NE SOIT LIMITATIVE, DES GARANTIES CONCERNANT LA VALEUR MARCHANDE, LAPTITUDE DES PRODUITS A RPONDRE A UNE UTILISATION PARTICULIERE, OU LE FAIT QUILS NE SOIENT PAS CONTREFAISANTS DE PRODUITS DE TIERS.
Course Contents
About This Course ..................................................................................................................... 1
Course Goal .................................................................................................................................................2 Course Overview ........................................................................................................................................3 Course Map ..................................................................................................................................................4 Module-by-Module Overview ..................................................................................................................5 Course Objectives ........................................................................................................................................7 Guidelines for Module Pacing ..................................................................................................................9 Topics Not Covered ..................................................................................................................................10 How Prepared Are You? ..........................................................................................................................11 Introductions .............................................................................................................................................12 How to Use the Course Materials ...........................................................................................................13 How to Use the Icons ................................................................................................................................14 Typographical Conventions and Symbols ............................................................................................15
Introducing Performance Management ...............................................................................1-1

Module Overview ................................................................................................................................... 1-2 Objectives ................................................................................................................................................. 1-3 Objectives ................................................................................................................................................. 1-4 What Is Tuning? ...................................................................................................................................... 1-5 Why Should a System Be Tuned? ......................................................................................................... 1-6 Basic Tuning Procedure ......................................................................................................................... 1-7 Conceptual Model of Performance ....................................................................................................... 1-8 Where to Tune First ................................................................................................................................ 1-9 Where to Tune? ..................................................................................................................................... 1-10 Performance Measurement Terminology .......................................................................................... 1-11 Performance Graphs ............................................................................................................................. 1-12


Example Presentation Graph .............................................................................................................. 1-13 Sample Tuning Task ............................................................................................................................. 1-14 The Tuning Cycle .................................................................................................................................. 1-15 SunOS Components .............................................................................................................................. 1-16 How Data Is Collected .......................................................................................................................... 1-17 The kstat Kernel Data Structure ....................................................................................................... 1-18 How the Tools Process the Data ......................................................................................................... 1-19 Monitoring Tools Provided With the Solaris OE ............................................................................. 1-20 The sar Command ............................................................................................................................... 1-21 The vmstat Command ......................................................................................................................... 1-22 The iostat Command ......................................................................................................................... 1-23 The mpstat Command ......................................................................................................................... 1-24 The netstat Command ....................................................................................................................... 1-25 The nfsstat Command ....................................................................................................................... 1-26 Sun Enterprise SyMON .................................................................................................................... 1-27 Sun Management Center ..................................................................................................................... 1-28 Other Utilities ........................................................................................................................................ 1-29 The prex Command ............................................................................................................................. 1-30 The tnfxtract and tnfdump Commands ........................................................................................ 1-31 The busstat Command ....................................................................................................................... 1-32 The cpustat and cputrack Commands .......................................................................................... 1-33 The /usr/proc/bin Directory ........................................................................................................... 1-34 The Process Manager ............................................................................................................................ 1-35 The memtool Package ........................................................................................................................... 1-36 The SE Toolkit ........................................................................................................................................ 1-37 System Performance Monitoring With the SE Toolkit ................................................................... 1-38 System Accounting ............................................................................................................................... 1-39 Viewing Tuning Parameters ................................................................................................................ 1-40 Setting Tuning Parameters .................................................................................................................. 1-41 The /etc/system File .......................................................................................................................... 1-42 Check Your Progress ............................................................................................................................ 1-43 Solaris System Performance Management
ii

Check Your Progress ............................................................................................................................ 1-44 Think Beyond ........................................................................................................................................ 1-45 Think Beyond ........................................................................................................................................ 1-46
Processes and Threads ...........................................................................................................2-1

Module Overview ................................................................................................................................... 2-2 Objectives ................................................................................................................................................. 2-3 Objectives ................................................................................................................................................. 2-4 Process Lifetime ...................................................................................................................................... 2-5 fork With exec Example ....................................................................................................................... 2-6 Process States ........................................................................................................................................... 2-7 Process Performance Issues ................................................................................................................... 2-8 Process Lifetime Performance Data ...................................................................................................... 2-9 Process-Related Tunable Parameters ................................................................................................. 2-10 Tunable Parameters Specific to the Solaris 8 OE .............................................................................. 2-11 Multithreading ...................................................................................................................................... 2-12 Application Threads ............................................................................................................................. 2-13 Process Execution .................................................................................................................................. 2-14 Thread Execution .................................................................................................................................. 2-15 Process Thread Examples .................................................................................................................... 2-16 Performance Issues ............................................................................................................................... 2-17 Thread and Process Performance ....................................................................................................... 2-18 Locking ................................................................................................................................................... 2-19 Locking Problems ................................................................................................................................. 2-20 The lockstat Command .................................................................................................................... 2-21 Interrupt Levels ..................................................................................................................................... 2-22 The clock Routine ................................................................................................................................ 2-23 Clock Tick Processing ........................................................................................................................... 2-24 Process Monitoring Using the ps Command .................................................................................... 2-25 Process Monitoring Using System Accounting ................................................................................ 2-26

iii

Check Your Progress ............................................................................................................................ 2-27 Check Your Progress ............................................................................................................................ 2-28 Think Beyond ........................................................................................................................................ 2-29
CPU Scheduling ......................................................................................................................3-1

Module Overview ................................................................................................................................... 3-2 Objectives ................................................................................................................................................. 3-3 Scheduling States .................................................................................................................................... 3-4 Scheduling Classes .................................................................................................................................. 3-5 Dispatch Priorities ................................................................................................................................... 3-6 Dispatch Parameter Table Issues .......................................................................................................... 3-7 Timesharing/Interactive Dispatch Parameter Table ......................................................................... 3-8 Timesharing Dispatch Parameter Table (Batch) ............................................................................... 3-11 The dispadmin Command .................................................................................................................. 3-12 dispadmin Example ............................................................................................................................. 3-13 The Interactive Scheduling Class ........................................................................................................ 3-14 Real-Time Scheduling .......................................................................................................................... 3-15 Real-Time Issues .................................................................................................................................... 3-16 Real-Time Dispatch Parameter Table ................................................................................................. 3-17 The priocntl Command .................................................................................................................... 3-18 Kernel Thread Dispatching ................................................................................................................. 3-20 Processor Sets ........................................................................................................................................ 3-21 Using Processor Sets ............................................................................................................................. 3-22 The Run Queue ...................................................................................................................................... 3-23 CPU Activity .......................................................................................................................................... 3-24 CPU Performance Statistics ................................................................................................................. 3-25 Available CPU Performance Data ...................................................................................................... 3-26 CPU Control and Monitoring .............................................................................................................. 3-27 The mpstat Command ......................................................................................................................... 3-28 mpstat Data ........................................................................................................................................... 3-29

iv

Solaris Resource Manager .................................................................................................................... 3-30 Solaris Resource Manager Hierarchical Shares Example ............................................................... 3-31 Solaris Resource Manager Limit Node .............................................................................................. 3-32 Check Your Progress ............................................................................................................................ 3-33 Think Beyond ........................................................................................................................................ 3-34
System Caches .........................................................................................................................4-1

Module Overview ................................................................................................................................... 4-2 Objectives ................................................................................................................................................. 4-3 What Is a Cache? ..................................................................................................................................... 4-4 Hardware Caches .................................................................................................................................... 4-5 Static RAM (SRAM) and Dynamic Random Access Memory (DRAM) ......................................... 4-6 CPU Caches .............................................................................................................................................. 4-7 Cache Operation ...................................................................................................................................... 4-8 Cache Miss Processing ........................................................................................................................... 4-9 Cache Data Replacement ..................................................................................................................... 4-10 Cache Operation Flow .......................................................................................................................... 4-11 Cache Hit Rate ....................................................................................................................................... 4-12 Effects of CPU Cache Misses ............................................................................................................... 4-14 Cache Characteristics ............................................................................................................................ 4-15 Virtual Address Cache ......................................................................................................................... 4-16 Physical Address Cache ....................................................................................................................... 4-17 Direct-Mapped and Set-Associative Caches ..................................................................................... 4-18 Write-Through and Write-Back Caches ............................................................................................. 4-19 CPU Cache Snooping ........................................................................................................................... 4-20 Cache Thrashing .................................................................................................................................... 4-22 System Cache Hierarchies ................................................................................................................... 4-24 Relative Access Times .......................................................................................................................... 4-26 Cache Performance Issues ................................................................................................................... 4-27 Items You Can Tune ............................................................................................................................. 4-28


Cache Monitoring Command Summary ........................................................................................... 4-29 Check Your Progress ............................................................................................................................ 4-30 Think Beyond ........................................................................................................................................ 4-31
Memory Tuning .......................................................................................................................5-1

Module Overview ................................................................................................................................... 5-2 Objectives ................................................................................................................................................. 5-3 Virtual Memory ....................................................................................................................................... 5-4 Process Address Space ........................................................................................................................... 5-5 Types of Segments .................................................................................................................................. 5-6 Virtual Address Translation .................................................................................................................. 5-7 Page Descriptor Cache ........................................................................................................................... 5-8 Virtual Address Lookup ........................................................................................................................ 5-9 The Memory-Free Queue ..................................................................................................................... 5-10 The Paging Mechanism ........................................................................................................................ 5-11 Page Daemon Scanning ........................................................................................................................ 5-12 Page Scanner Processing ...................................................................................................................... 5-13 Default Paging Parameters .................................................................................................................. 5-14 The Clock Algorithm ............................................................................................................................ 5-15 The maxpgio Parameter ......................................................................................................................... 5-16 File System Caching .............................................................................................................................. 5-17 The Buffer Cache ................................................................................................................................... 5-18 The Page Cache ..................................................................................................................................... 5-19 Available Paging Statistics ................................................................................................................... 5-20 Additional Paging Statistics ................................................................................................................ 5-21 Swapping ................................................................................................................................................ 5-22 When Is the Swapper Desperate? ....................................................................................................... 5-23 What Is Not Swappable ....................................................................................................................... 5-24 Swapping Priorities .............................................................................................................................. 5-25 Swap In ................................................................................................................................................... 5-26

vi

Swap Space ............................................................................................................................................ 5-27 The tmpfs File System .......................................................................................................................... 5-28 Available Swapping Statistics ............................................................................................................. 5-29 Shared Libraries .................................................................................................................................... 5-30 Paging Review ....................................................................................................................................... 5-31 Memory Utilization .............................................................................................................................. 5-32 Total Physical Memory ........................................................................................................................ 5-33 File Buffering Memory ......................................................................................................................... 5-34 Kernel Memory ..................................................................................................................................... 5-35 Identifying an Applications Memory Requirements ...................................................................... 5-36 Free Memory .......................................................................................................................................... 5-37 Identifying a Memory Shortage .......................................................................................................... 5-38 Items You Can Tune ............................................................................................................................. 5-39 Check Your Progress ............................................................................................................................ 5-40 Think Beyond ........................................................................................................................................ 5-41
System Buses ...........................................................................................................................6-1

Module Overview ................................................................................................................................... 6-2 Objectives ................................................................................................................................................. 6-3 What Is a Bus? ......................................................................................................................................... 6-4 General Bus Characteristics ................................................................................................................... 6-5 System Buses ............................................................................................................................................ 6-6 System Buses: MBus and XDbus .......................................................................................................... 6-7 System Buses: UPA and Gigaplane Bus .............................................................................................. 6-8 Gigaplane XB Bus .................................................................................................................................... 6-9 The Gigaplane XB Bus .......................................................................................................................... 6-10 Peripheral Buses .................................................................................................................................... 6-11 SBus Compared to PCI Bus ................................................................................................................. 6-12 PIO and DVMA ..................................................................................................................................... 6-13 The prtdiag Utility .............................................................................................................................. 6-14

vii

prtdiag CPU Section ........................................................................................................................... 6-15 prtdiag Memory Section .................................................................................................................... 6-16 prtdiag I/O Section ............................................................................................................................ 6-17 Bus Limits ............................................................................................................................................... 6-18 Bus Bandwidth ...................................................................................................................................... 6-19 Diagnosing Bus Problems .................................................................................................................... 6-20 Avoiding Bus Problems ....................................................................................................................... 6-21 SBus Overload Examples ..................................................................................................................... 6-23 System Bus Overload Examples ......................................................................................................... 6-24 CPU Load on the System Bus .............................................................................................................. 6-25 Dynamic Reconfiguration Considerations ........................................................................................ 6-26 Tuning Reports ...................................................................................................................................... 6-27 Bus Monitoring Commands ................................................................................................................ 6-28 Check Your Progress ............................................................................................................................ 6-29 Think Beyond ........................................................................................................................................ 6-30
I/O Tuning ................................................................................................................................7-1

Module Overview ................................................................................................................................... 7-2 Objectives ................................................................................................................................................. 7-3 SCSI Bus Overview ................................................................................................................................. 7-4 SCSI Bus Characteristics ........................................................................................................................ 7-5 SCSI Speeds .............................................................................................................................................. 7-6 SCSI Widths ............................................................................................................................................. 7-7 SCSI Lengths ............................................................................................................................................ 7-8 SCSI Properties Summary ...................................................................................................................... 7-9 SCSI Interface Addressing ................................................................................................................... 7-10 SCSI Bus Addressing ............................................................................................................................ 7-11 SCSI Target Priorities ........................................................................................................................... 7-12 Fibre Channel ......................................................................................................................................... 7-13 Disk I/O Time Components ................................................................................................................ 7-15

viii

Disk I/O Time Calculation .................................................................................................................. 7-16 Disk Drive Features .............................................................................................................................. 7-17 Multiple Zone Recording ..................................................................................................................... 7-18 Drive Caching and Fast Writes ........................................................................................................... 7-19 Tagged Queueing .................................................................................................................................. 7-20 Mode Pages ............................................................................................................................................ 7-21 Device Properties From prtconf .................................................................................................................................. 7-22 Device Properties From scsiinfo .............................................................................................................................. 7-23 I/O Performance Planning .................................................................................................................. 7-24 Tape Drive Operations ......................................................................................................................... 7-25 Storage Arrays and JBODs ................................................................................................................... 7-26 Storage Array Architecture .................................................................................................................. 7-27 RAID ....................................................................................................................................................... 7-28 RAID-0 .................................................................................................................................................... 7-29 RAID-0 Stripe Sizing ............................................................................................................................ 7-30 RAID-1 .................................................................................................................................................... 7-31 RAID-0+1 ................................................................................................................................................ 7-32 RAID-3 .................................................................................................................................................... 7-33 RAID-5 .................................................................................................................................................... 7-34 Tuning the I/O Subsystem .................................................................................................................. 7-35 Available Tuning Data ......................................................................................................................... 7-36 Available I/O Statistics ........................................................................................................................ 7-37 Available I/O Data ............................................................................................................................... 7-38 Check Your Progress ............................................................................................................................ 7-39 Think Beyond ........................................................................................................................................ 7-40

ix

UFS Tuning ..............................................................................................................................8-1
Module Overview ................................................................................................................................... 8-2 Objectives ................................................................................................................................................. 8-3 The vnode Interface ................................................................................................................................ 8-5 The Local File System ............................................................................................................................. 8-6 Fast File System Layout ......................................................................................................................... 8-7 The Directory ........................................................................................................................................... 8-8 Directory Name Lookup Cache (DNLC) ............................................................................................. 8-9 The inode Cache ................................................................................................................................... 8-10 DNLC and inode Cache Management .............................................................................................. 8-12 The inode Structure ............................................................................................................................. 8-13 The inode Block Pointers .................................................................................................................... 8-14 UFS Buffer Cache .................................................................................................................................. 8-15 Hard Links ............................................................................................................................................. 8-16 Symbolic Links ...................................................................................................................................... 8-17 Allocating Directories and inodes ..................................................................................................... 8-18 The Veritas File System (VxFS) ........................................................................................................... 8-19 Comparison of UFS and VxFS ............................................................................................................. 8-20 Allocation Performance ........................................................................................................................ 8-21 Allocating Data Blocks ......................................................................................................................... 8-22 Quadratic Rehash .................................................................................................................................. 8-23 Fragments ............................................................................................................................................... 8-24 Logging File Systems ............................................................................................................................ 8-25 Application I/O ..................................................................................................................................... 8-26 Working With the segmap Cache ....................................................................................................... 8-28 File System Performance Statistics ..................................................................................................... 8-29 The fsflush Daemon .......................................................................................................................... 8-30 Direct I/O ............................................................................................................................................... 8-31 Accessing Data Directly With mmap .............................................................................................................................. 8-32 Using the madvise Call ........................................................................................................................ 8-33


File System Performance ...................................................................................................................... 8-34 Cylinder Groups .................................................................................................................................... 8-35 Sequential Access Workloads .............................................................................................................. 8-36 UFS File System Read Ahead .............................................................................................................. 8-37 Cluster Sizes for RAID Volumes ......................................................................................................... 8-38 Commands to Set the Cluster Size ...................................................................................................... 8-39 File System Write Behind ..................................................................................................................... 8-40 UFS Write Throttle ................................................................................................................................ 8-41 Random Access Workloads ................................................................................................................. 8-42 Tuning for Random I/O ...................................................................................................................... 8-43 Post-Creation Tuning Parameters ...................................................................................................... 8-44 Application I/O Performance Statistics ............................................................................................. 8-45 Check Your Progress ............................................................................................................................ 8-46 Think Beyond ........................................................................................................................................ 8-48
Network Tuning ......................................................................................................................9-1

Module Overview ................................................................................................................................... 9-2 Objectives ................................................................................................................................................. 9-3 Networks .................................................................................................................................................. 9-4 Network Bandwidth ............................................................................................................................... 9-5 Full Duplex 100 BASET Ethernet .......................................................................................................... 9-6 IP Trunking .............................................................................................................................................. 9-7 TCP Connections ..................................................................................................................................... 9-8 TCP Stack Issues ...................................................................................................................................... 9-9 TCP Tuning Parameters ....................................................................................................................... 9-10 Using the ndd Command ..................................................................................................................... 9-12 Network Hardware Performance ....................................................................................................... 9-13 NFS .......................................................................................................................................................... 9-14 NFS Client Retransmissions ................................................................................................................ 9-15 NFS Server Daemon Threads .............................................................................................................. 9-16

xi

The Cache File System .......................................................................................................................... 9-17 CacheFS Characteristics ....................................................................................................................... 9-18 CacheFS Benefits and Limitations ...................................................................................................... 9-19 Monitoring Network Performance ..................................................................................................... 9-20 Isolating Problems ................................................................................................................................ 9-21 Tuning Reports ...................................................................................................................................... 9-22 Check Your Progress ............................................................................................................................ 9-23 Think Beyond ........................................................................................................................................ 9-24
Performance Tuning Summary ...........................................................................................10-1

Module Overview ................................................................................................................................. 10-2 Objectives ............................................................................................................................................... 10-3 Some General Guidelines ..................................................................................................................... 10-4 What You Can Do ................................................................................................................................. 10-5 Application Source Code Tuning ....................................................................................................... 10-6 Compiler Optimization ........................................................................................................................ 10-7 Application Executable Tuning Tips .................................................................................................. 10-8 Performance Bottlenecks ...................................................................................................................... 10-9 Memory Bottlenecks ........................................................................................................................... 10-10 Memory Bottleneck Solutions ........................................................................................................... 10-11 I/O Bottlenecks ................................................................................................................................... 10-12 I/O Bottleneck Solutions ................................................................................................................... 10-13 CPU Bottlenecks .................................................................................................................................. 10-14 CPU Bottleneck Solutions .................................................................................................................. 10-15 Ten Tuning Tips .................................................................................................................................. 10-16 Check Your Progress .......................................................................................................................... 10-18 Think Beyond ...................................................................................................................................... 10-19

xii

Sa 400 RC

Caricato da

Informazioni sul documento

Descrizione originale:

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Sa 400 RC

Caricato da

Copyright:

Formati disponibili

Sun Educational Services

Solaris System Performance Management

Solaris System Performance Management

Sun Educational Services

About This Course

Solaris System Performance Management

Sun Educational Services

Solaris System Performance Management

About This Course, slide 2 of 15

Sun Educational Services

Solaris System Performance Management

About This Course, slide 3 of 15

Sun Educational Services

Solaris System Performance Management

About This Course, slide 4 of 15

Sun Educational Services

Solaris System Performance Management

About This Course, slide 5 of 15

Sun Educational Services

Solaris System Performance Management

About This Course, slide 6 of 15

Sun Educational Services

Solaris System Performance Management

About This Course, slide 7 of 15

Sun Educational Services

Solaris System Performance Management

About This Course, slide 8 of 15

Sun Educational Services

Guidelines for Module Pacing

Solaris System Performance Management

About This Course, slide 9 of 15

Sun Educational Services

Topics Not Covered

Solaris System Performance Management

About This Course, slide 10 of 15

Sun Educational Services

How Prepared Are You?

Solaris System Performance Management

About This Course, slide 11 of 15

Sun Educational Services

Solaris System Performance Management

About This Course, slide 12 of 15

Sun Educational Services

How to Use the Course Materials

About This Course, slide 13 of 15

Sun Educational Services

How to Use the Icons

Solaris System Performance Management

About This Course, slide 14 of 15

Sun Educational Services

Typographical Conventions and Symbols

Solaris System Performance Management

About This Course, slide 15 of 15

Sun Educational Services

Solaris System Performance Management

Sun Educational Services

Solaris System Performance Management

Sun Educational Services

Solaris System Performance Management

Sun Educational Services

Solaris System Performance Management

Sun Educational Services