For GSS, October 2010

By Dor Laor

Agenda

KVM arch background

Live migration animation and more
Rhel6 and advantages over competition
Support of kvm
Q&A

The KVM approach

A hypervisor needs

A scheduler and memory management

Platform support code
An I/O stack
Device drivers
A management stack

Focus on virtualization, leave

other things to respective developers

Ordinar
Ordinar
y
Ordinar
y
Linux
y
Linux
Process
Linux
Process
Process

User
VM

User
VM

User
VM

KVM
Modules
Driver

Linux
Driver

Driver

Hardware

KVM's bible:

Do not duplicate existing functionality

Designed for hardware virtualization extensions
Work with the Linux development process
4

Control flow
Userspace

Kernel

Guest

ioctl()
Switch to
Guest
Mode
Native Guest
Execution
Kernel
exit
handler
Userspace
exit handler

KVM Memory Model

User
Address
Space

VMM userspace
code and data

phys_ram = mmap()
OR
phys_ram = malloc()

Guest physical
address space
Kernel
Address
Space

Virtio network architecture

Guest VM

r
x

t
x
QEMU
tap

kernel/HV

bridge

NIC

PCI device assignment network architecture (vt-d/sriov, rhel5)

Low performance overhead

Best throughput
Best Latency

Guest VM

t
x

r
x

SRIOV allows assignment of

virtual functions to the guest

QEMU
VF NIC #2
kernel/HV

Need to pin all of the guest

memory to RAM
No simple live migration
The guests needs latest drivers

VF NIC #1

Physical NIC

In-Kernel vhost-net architecture (rhel6)

Less context switching
Low latency
MSI
One less copy

Guest VM

QEMU

t
x

r
x

kernel/HV
vhost
tap

bridge
NIC

Vhost over sriov & macvtap architecture (rhel6)

Guest knows only about
virtio
Live migration works
Excellent performance
Full zero copy in 6.1

Guest VM

QEMU

t
x

r
x

kernel/HV
vhost
macvtap

VF NIC #1

VF NIC #2

Physical NIC

10

Memory management

Sharing
Swapping
Page Migration: move page to closer location on NUMA node
Live Migration: move entire guest to another host
Ballooning
Backed by large pages
Transparent huge pages by the host/guest kernels

11

Page sharing

G1

G2

Kernel

12

KSM before merging

13

pte
va1

pte
va2

0xaabbccdd...
writable

0xaabbccdd...
writable

2009 Red Hat, Inc

KSM after merging

pte
va1

pte
va2

0xaabbccdd...
wrprotect

14

2009 Red Hat, Inc

KVM using KSM for memory overcommit

RHEL 5.4 KVM KSM w/ Win2003
20

250000

19
18
17

200000

16
15
14

150000

13
12
11

100000

Op Per Sec

10
9
8

50000

7
6
5

0
1VM-4GB

5VM-20GB

10VM-40GB

VMs and Memory Used

Red Hat, Inc.

15VM-60GB

20VM-80GB

15

Status
Replaces physical host
... :)
RHEL6.0
Performance
oriented
RHEL5.4 1st release
KVM embraced
by Red Hat
KVM 'invented'
2006

16

Kvm in rhel6

Performance

CPU

Vhost kenrel virtio tech

preview only
MSI

Block

SMP Guest spin lock detector

RCUfying 64 VCPU
Userspace notifiers
Ticket spinlock patching

Qemu aio
Ext' ring entries

Qcow2

RAS

Qemu machine monitor

Guest stable abi

Vmchannel

CPU modeling

General

X2apic

Supporting rhel5.4 abi + migration

Cgroups
Memory

Networking

Transparent huge pages

KSM swapping
Shadow young PTE

Cache/batch

17

KVM Performance Linux Intel EPT

Performance results
Sybase Database July,

2009

Nehalem Single Guest Bare Metal /Xen/KVM - 8/16 cpu

300000.00

250000.00

200000.00

Xen

150000.00

KVM
Bare Metal

Trans/minute

100000.00

50000.00

0.00
8cpu

16cpu

Life has no rehearsals, only performances.

18

Live migration

Host A

Host B
Migrate to B

1. Setup
2. Transfer Memory

19

Algorithm

Host A

Host B

1. Setup
2. Transfer Memory

Whole Memory

1 -- 133000

20

Algorithm

Host A

Host B

1. Setup
2. Transfer Memory

VM State

3. Stop the VM
4. Transfer State

1 -- 50

21

Algorithm

Host A

Host B
Continue

1. Setup
2. Transfer Memory
3. Stop the VM
4. Transfer State
5. Continue VM

22

Comparison with Competition

Scaling

64 vcpu per guest (Vmware scales to 8)

hundreds of VMs per host

1k cpus for the host

vmware scales to 64 physical cpus only

IO multipath wins over VMW

Linux integration and features

KVMClock

X2apic interrupt controller

Virtio-serial (vmchannel)

Async page fault (not yet in rhel)

My grandfather once told me that there were two
kinds of people: those who do the work and those
who take the credit. He told me to try to be in the
first group; there was much less competition.

23

Comparison with Competition

HW enablement (RHEL)
open source
KSM and full swapping - XEN does not have
Hybrid mode - running standard Linux apps along with Vms over VMware
Real time host kernel and micro latency
SELinux
Spice
Better latency and lower ctx-switch - Architecture advantage
over XEN.

My grandfather once told me that there were two

kinds of people: those who do the work and those
who take the credit. He told me to try to be in the
first group; there was much less competition.

24

KVM Support

Only supported through libvirt

On issue, please ask:

Libvirt's xml
Qemu cmdline
Cpu consumption on host/guest
Guest type, guest up-to-date?
Qemu-kvm, host/guest kernel, seabios NVR
kvm_stat
kvmtrace

25

KVM Support cont'

IO issues

Make sure qemu runs with O_DIRECT (cache=off)

It's libvirt's shared image mode
Drastically improves performance over pass through
Might be worth to use tuned (but no barriers are risky)

Timing issues

http://cleo.tlv.redhat.com/qumrawiki/KVM/TimeKeeping
No known issues on windows
KVMclock works (up-to-date 5.5.z guest)

26

Q&A w/ beer

27