SQL Server Performance Tuning

6/12/2007

1991-2007 Gert E.R. Drapers. All rights reserved.

Agenda
SQL-OS
Scheduler
Wait States

6/12/2007

1991-2007 Gert E.R. Drapers. All rights reserved.

Why care about the Scheduler?

When under load, EVERY system has a bottleneck

that limits performance
In real systems as utilization 100%, other components

provide back pressure

Which component may depend on the workload, time of
day, etc.
The usage pattern: OLTP, DDS, batch may expose
different patterns and bottlenecks on the same
configuration

Fix one bottleneck, and another surfaces

CPU, I/O, locking (concurrency), network, memory

In a perfect world, CPU is the last resource maxed

out

6/12/2007

1991-2007 Gert E.R. Drapers. All rights reserved.

Performance Layers
Physical I/O

Locking
Logical I/O

Networking

Query
Requests

6/12/2007

NET

Users /
Sessions

LOCKS

Object
Access

LIO

Cache
(Buffer
Pool)

1991-2007 Gert E.R. Drapers. All rights reserved.

PIO

Physical
Storage

What is SQLOS
Thin user-mode layer between server and
OS:
Scheduler and IO completion
Synchronization primitives
Memory Management
Resource Management
Deadlock Detection
Exception Handling Framework
Hosting (Services for external components)

6/12/2007

1991-2007 Gert E.R. Drapers. All rights reserved.

What SQLOS is not

OS Abstraction layer
Does not wrap all OS APIs (like .NET)
Platform neutral bridge to other OSs.

Component Library
No general purpose classes or functions
Does not define rules for programming

6/12/2007

1991-2007 Gert E.R. Drapers. All rights reserved.

Introduction Why SQLOS?

Windows is a general purpose OS

Optimizes for the common application

Provides only simple synchronization primitives
Slow and inflexible memory allocators
Dumb scheduling
Not optimal on high end boxes

All System services in one place

Better accounting and resource management

Factor common code

Caching framework
Custom memory allocators

6/12/2007

1991-2007 Gert E.R. Drapers. All rights reserved.

What is SMP
SMP Symmetric Multi-Processing
Front-bus point of contention
Difficult to scale beyond 32 CPU

Memory

Front-bus

CPU 0

CPU 1

...

CPU n

What is NUMA

NUMA Non-Uniformed Memory Access

Minimize/eliminate front-bus contention to
surpass scalability limits of SMP architecture
Performance penalty for accessing foreign node
memory
Application needs to be NUMA-aware to take
advantage the node-locality design
Memory

Foreign Memory Access

4x local
Local Memory Access

CPU 0

CPU 1

CPU 2

CPU 3

CPU

Memory

CPU

CPU

CPU n

What is Interleaved-NUMA

Enable NUMA hardware to behave as SMP

Memory are used by all CPUs
Each CPUs cache line access slice of memory from

all nodes

SQL Server 2000 should use interleaved-NUMA

Memory

CPU 0

CPU 1

CPU 2

Local Memory Access

Memory

CPU 3

CPU

CPU

Foreign Memory

CPU

CPU n

What is Soft-NUMA
Activates custom NUMA configuration on top
of any of hardware
Registry settings control final SoftNUMA
configuration
Provides greater performance, scalability,
and manageability on SMP as well as on real
NUMA hardware

6/12/2007

1991-2007 Gert E.R. Drapers. All rights reserved.

11

SOS & NUMA

Mimics HW
configuration
Supports NUMA
hardware
Locality at each level
server, node, CPU,
task running on CPU

NUMA Node
Memory

CPU

CPU 1

CPU 2

CPU 3

SOS Node
SOS
Memory
Node

Scheduler

6/12/2007

Scheduler

Scheduler

Scheduler

1991-2007 Gert E.R. Drapers. All rights reserved.

12

Server Architecture
SQLOS API

Protocols
Query Comp. and Execution Engines

SQLOS API

Resource
Monitor
Lazy
Writer
Scheduler
Monitor

SQLOS

Hosting API

Deadlock
Monitor

Lock
Manager

External Components

Storage Engine

Synchronization
Buffer
Pool

Scheduling

I/O

Memory Manager

= thread
6/12/2007

1991-2007 Gert E.R. Drapers. All rights reserved.

13

Agenda
SQL-OS
Scheduler
Wait States

6/12/2007

1991-2007 Gert E.R. Drapers. All rights reserved.

14

SOS Scheduler

Thin layer between SQL Server and the OS

Non-preemptive scheduling
Provides scheduling, I/O processing and
synchronization
Two modes of execution: Thread or Fiber
Thread is default
Fiber can provide performance boost for specific

usage scenarios

6/12/2007

Rarely used in production environment

Not all features works with fiber
No visibility down to the fiber level debugging
CLR not supported under Fiber mode
1991-2007 Gert E.R. Drapers. All rights reserved.

15

Benefits of Scheduler

Distribute scheduling synchronization

(Scheduler per CPU)
Reduced contention on Windows NT dispatcher
spinlock
Control preemption rates and expensive context
switching
Improved instruction/data caching behavior
Control code execution
Sync Free Global Objects (Scheduler based
objects, e.g. free-lock lists)

6/12/2007

1991-2007 Gert E.R. Drapers. All rights reserved.

16

SOS - Node
Maps to a single NUMA node or SMP box
Collection of CPUs (Schedulers)
Memory Node
Scheduler Management and Monitoring

4-Way SMP

2-Node NUMA

6/12/2007

1991-2007 Gert E.R. Drapers. All rights reserved.

17

SOS Node Architecture

CPU 0 CPU 1 CPU 2

CPU n

Sched 0

Sched n

SOS Node

Tasks

6/12/2007

Scheduler
Work
Dispatcher

Sched 1

Tasks

Scheduler
Work
Dispatcher

Sched 2

Tasks

Tasks
SOS
Schedules
Tasks

Scheduler
Work
Dispatcher

Scheduler
Work
Dispatcher

Tasks Load balanced across schedulers

1991-2007 Gert E.R. Drapers. All rights reserved.

18

SOS Executive Subsystem

Global state and configuration

OS global read only configuration object

Scheduling subsystem
Task
Execution unit maps to thread or fiber
Scheduler
Provides non-preemptive scheduling and task pool
In SS2K a connection was assigned to a UMS and stayed there and in
SS2005, a thread within a connection is what is assigned to a UMS
NUMA node
Provides node locality and pool of real threads
Synchronization objects
Library of synchronization objects built on top of non-preemptive
scheduling

Exception handling

6/12/2007

1991-2007 Gert E.R. Drapers. All rights reserved.

19

SOS Scheduler

Single CPU abstraction

Schedules Workers that run on top of SystemThreads
Binds Tasks and Workers
Fiber/Thread Mode and Preemptive/Nonpremptive
Manages Affinity
Thread Mode

Fiber Mode

Task (n)

Task (n)

Worker (m)

Worker (m)

SystemThread (m)

SystemThread (k<m)

OS Thread

OS Thread

System Threads
Data structure describing Physical Thread.
Used to execute thread or fiber.
All threads created in SOS Process are
associated with SOS on creation.

Allows SOS to do better accounting.

System Threads created outside of SOS can still

call SOS APIs.

6/12/2007

1991-2007 Gert E.R. Drapers. All rights reserved.

21

Workers
Data structure representing logical thread
It is associated with a scheduler
Workers created on demand up to a
configured limit
Free workers remain in a worker pool
Free workers can shrink after a timeout or on
memory pressure
Free workers can be migrated between
schedulers, Non-free workers do not migrate
between schedulers.

6/12/2007

1991-2007 Gert E.R. Drapers. All rights reserved.

22

Tasks

Tasks are assigned to the least loaded scheduler,

based on active number of tasks on CPU
Worker picks Task and runs it
Yield points sprinkled throughout code. Some SOS
APIs yield internally (e.g. Mem Allocation)
Tasks can be aborted.
Tasks do not migrate between schedulers
Timer Tasks
Internal Tasks that run every so often e.g. ShrinkDB
Do not hold onto worker while waiting for timeout
Checkpoint process is not part of the timer task.

6/12/2007

1991-2007 Gert E.R. Drapers. All rights reserved.

23

SOS Scheduling Mode

Non-preemptive mode (or co-operative mode)

Default

Preemptive mode
SOS Switch to preemptive mode
Code that could block outside of server (NT APIs)
Code we have no control over (XPs, DQ)

Thread runs independent of scheduler

Switch to non-preemptive mode to continue running in

context of scheduler
SOS APIs can be called from preemptive mode

Regular SQL tasks executed in co-operative

mode; XP/CLR executed in preemptive mode

6/12/2007

1991-2007 Gert E.R. Drapers. All rights reserved.

24

Scheduler Architecture
Idle worker

Current worker

Runnable Queue

Worker Dispatcher

Worker Pool

Scheduler
Timer Queue

Pending Queue

Wake event
Aborted Task List
IO Queues

6/12/2007

1991-2007 Gert E.R. Drapers. All rights reserved.

25

Resource Utilization
Execution model: Queues & Lists (simplified)

Running - 1/SQLOS

SPID
60
IO_Completion
SPID 51
60 Running

Runnable Queue 1/UMS

(Signal Waits)

SPID 51
SPID 64
SPID 87
SPID 52
SPID 93

Runnable
Runnable
Runnable
Runnable
Runnable

Waiter List (Resource Waits)

SPID 73 LCK_M_S
SPID 59 NETWORKIO
SPID 56 CXPACKET
Runnable
SPID 55 RESOURCE_SEMAPHORE

SQL Server 2000 Scheduler

Connection

2
3

1, 5
2

UMS

3
4

UMS

UMS

UMS

4 way CPU

5 connections each connection is assigned to a UMS and stays there.

If these are long running connections then UMS above with #1 and #5
connection would be twice as slow as the other three connections (#2,
#3 and #4)

6/12/2007

1991-2007 Gert E.R. Drapers. All rights reserved.

27

SQL Server 2005 Scheduler

Task -> Threads

1
Schd

2
3

Threads
Not
Connections

Schd
Schd

Schd

4 way CPU

5 connections each thread within a connection gets assigned to a

different UMS.
So, less likely to starve a single UMS

6/12/2007

1991-2007 Gert E.R. Drapers. All rights reserved.

28

I/O completion

I/O Completion port

Each node has a dedicated preemptive worker sitting on I/O

completion port
Node based API.

Polling based I/O:

Scheduler based API: Non Preemptive mode only

Supports disk and network IO only (network unused)
I/O is polled at scheduler context switch time
Both queues are entirely scanned, Completion routine is invoked
for I/Os that have completed
Polling IO Completion routines should NOT rely on Thread Local
Storage (TLS), take too long to execute, throw exceptions or
yield.
Possible to scan manually
When scheduler is idle, I/O completion will wake it up

6/12/2007

1991-2007 Gert E.R. Drapers. All rights reserved.

29

Dedicated Admin Connection (DAC)

DAC has its own hidden scheduler
Provides access to overloaded server
DAC resources are allocated during SQL Server
startup
Can be used to:

Run diagnostic queries

To control runaway queries that consume all resources

Considered to be a last resort to connect to

unhealthy server
Note:

Dont run complex queries or for application use!

6/12/2007

1991-2007 Gert E.R. Drapers. All rights reserved.

30

Agenda
SQL-OS
Scheduler
Wait States

6/12/2007

1991-2007 Gert E.R. Drapers. All rights reserved.

31

Scheduler State Transitions

Need a resource

RUNNING!
Switched

No new
request

Yield

Runnable List

New
request

WorkerPool

Timer
Request

Expired/
signaled

Timer
List

Resource
available

Created

Resource
Waiter List

15 min idle

Dead

6/12/2007

1991-2007 Gert E.R. Drapers. All rights reserved.

32

SQL Server Wait States

Every time SQL Server wants to access a

resource which is NOT immediately available, a
wait state is entered
When the resource becomes available the waiter is

signaled
Once the waiter is running again the resource is
actually being used

In SQL Server 2000 there are 77 distinct wait

states
Some wait states are overloaded and therefore reflect

one or more causes

6/12/2007

1991-2007 Gert E.R. Drapers. All rights reserved.

33

Retrieving Wait State Information

SQL Server 2000

dbo.sysprocesses
dbcc sqlperf(waitstats)

77 wait states

SQL Server 2005

sys.dm_exec_requests
sys.dm_os_waiting_tasks
sys.dm_os_wait_stats
201 wait states

6/12/2007

1991-2007 Gert E.R. Drapers. All rights reserved.

34

Interpreting Wait Times

Requests
Number of times the wait state was entered
Not number of time resource got used!

Resource Wait Time (T1 T0)

Signal Wait Time (T2 T1)
Time in runnable queue prior to continuation of execution

Wait Time (T2 T0)

Total time waiting on resource plus the time the worker thread

gets scheduled so it can use the resource

T0 = Request resource, enter wait state
T1 = Signal Resource Availability
T2 = Use resource, reset wait state
Signal Wait Time
Wait Time
6/12/2007

1991-2007 Gert E.R. Drapers. All rights reserved.

Time

35

How to use wait types and times?

Need to correlated Waits with Queues

Queues
Outstanding (unfulfilled) resource requests
Accessible through Performance Monitor counters
Measure resource utilization, like Avg Disk Queue Length

Waits
Anytime a SQL thread is not executing, a wait type is set

reflecting the reason for waiting

Waits from an Application or User connection perspective
77 distinct wait types for SQL Server 2000

Queues and Waits provide complementary

information
Both are essential for problem determination
Together, explain application performance

6/12/2007

1991-2007 Gert E.R. Drapers. All rights reserved.

36

Summary

Understanding the SQL Server execution model and the

relationship to wait states provide you the insights
needed to analyze and solve performance problems!

Application performance is all about waits and queues

Workloads may vary, shifting resource usage and utilization

System performance is all about identifying and

resolving bottlenecks, maximizing system capabilities

A well performing system, is a balanced system!

6/12/2007

1991-2007 Gert E.R. Drapers. All rights reserved.

37

6/12/2007

1991-2007 Gert E.R. Drapers. All rights reserved.

38