HOW to CONFIGURE

a LOGICAL DOMAIN

> OpenSolaris™ How To Guides

Brian Leonard, OpenSolaris Technical Marketing Engineer

Version 2.0 | Last updated: 06/18/09

About This OpenSolaris How To Guide
This Logical Domain How To guide is intended to help OpenSolaris system adminstrators configure their SPARC CMT
machines with logical domains (LDoms). Users are guided step-by-step through the process of installing the logical
domain manager and configuring a guest domain.

For more information about LDoms, see http://opensolaris.org/os/community/ldoms/

Contents
Overview Page 1

Configuring a Solaris 10 Guest Domain on OpenSolaris Page 1

Step 1: Check the Firmware Page 1
Step 2: Install the Logical Domains Manager Page 1
Step 3: Free Up Resources for the Guest Domain Page 2
Step 4: Create Services Needed to Support the Guest Domains Page 7
Step 5: Configure the Guest Domain Page 8
Step 6: Create a Virtual Boot Disk for the Guest Domain Page 8
Step 7: Create a Virtual Disk for the Guest Operating System Installation DVD Page 9
Step 8: Define a Virtual Network Interface Page 9
Step 9: Allocate Resources to the Guest Domain Page 9
Step 10: Log Into the Guest Domain Page 11
Step 11: Install Solaris 10 in the Guest Domain Page 11
Step 12: Operate the Guest Domain Page 12

For More Information Page 13

 1
OpenSolaris Operating System opensolaris.com

Logical Domain How To Guide

Overview
Logical Domains (LDoms) allow you to virtualize OpenSolaris by creating isolated environments in which you can run other
operating systems that are LDom aware, namely Solaris and OpenSolaris. The virtualization features provided by LDoms
are designed into the SPARC CMT microprocessors and are therefore only available on machines containing either the
UltraSPARC T1 (32 threads), UltraSPARC T2 (64 threads), and UltraSparc T2+ (64 threads) chips.

To begin taking advantage of LDoms, the Logical Domains Manager software needs to first be installed which then runs
OpenSolaris in a “paravirtualized” state, meaning OpenSolaris is now aware of the virtualization layer beneath it. At this
point, there's a single domain, the “primary” domain, which is ready to provide virtualization services, such as networking
and disk i/o, to guest domains.

Before guest domains can be configured, resources such as CPUs and memory need to be removed from the primary
domain so that they can later be assigned to the guest domain. As part of the guest domain's configuration, a virtual disk
(or disks) needs to be set up. These virtual disks can map either to an actual disk or to a single file, which is the approach
that is used in this How To guide.

Once created, the guest domain still needs an operating system. As stated above, this can be either Solaris or OpenSolaris.
Solaris 10 is used as the guest operating system for this guide.

Configuring a Solaris 10 Guest Domain on OpenSolaris

Step 1: Check the Firmware
As mentioned above, the SPARC CMT microprocessors have virtualization features built into their firmware. Before
installing the Logical Domains Manager, make sure you have the latest firmware installed. You can check the latest firware
revs at http://docs.sun.com/source/820-4914-10/chapter1.html#d0e419. See the "Required Software to Enable LDoms 1.1
Features" table and the "Required System Firmware Patches" table.

The exact steps for checking and updating your firmware vary depending on your actual hardware. Please reference
http://docs.sun.com/app/docs/prod/coolthreads.srvr#hic for the specific steps that match your hardware.

Step 2: Install the Logical Domains Manager

Before beginning, run the uname -m to verify your processor supports LDoms. The proper result will be sun4v.

jack@dcsw-t5240:~$ uname -m
sun4v

The Logical Domans Manger is available in the IPS repository:

jack@dcsw-t5240:~$ pkg info -r ldomsmanager

Name: system/ldoms/ldomsmanager
Summary: Logical Domains Manager
Category: System/Virtualization
State: Not installed
Publisher: opensolaris.org
Version: 1.1
Build Release: 5.11
Branch: 0.111
Packaging Date: Sat Apr 18 20:05:08 2009
Install it as follows:
Size: 8.16 MB
FMRI: pkg:/system/ldoms/ldomsmanager@1.1,5.11-0.111:20090418T200508Z
2
OpenSolaris Operating System opensolaris.com

Install it as follows:

jack@dcsw-t5240:~$ pfexec pkg install ldomsmanager

DOWNLOAD PKGS FILES XFER (MB)
Completed 1/1 40/40 3.78/3.78

PHASE ACTIONS
Install Phase 74/74
PHASE ITEMS
Indexing Packages 1/1

Step 3: Free Up Resources for the Guest Domain

Once the Domain Manager is installed, OpenSolaris is running in a domain - the control domain - which is named primary.
You can see it by running ldm list:

jack@dcsw-t5240:/usr/bin$ ldm list

NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
primary active -n-c-- SP 128 130848M 0.2% 3h 41m

The control domain is responsible for managing all of the other domains on the system and initially has all of the system
resources allocated to it. Before we can create a guest domain, we need to steal resources from the primary domain. In
the output above you can see that we have 128 virtual CPUs and 130GB of memory. The T5240 has two T2+ processors,
each with 8 cores and each core having 8 threads which gives us the total of 128 virtual CPUs. Theorectically, we could
create 127 guest domains, however allocating an entire core to a single domain is generally considered best for
peformance. That's because the T2+ processor in the T5240 I'm using has 4 cores, each represented as a virtual CPU.

You can view all of the free devices on the system by running:

ldm list-devices

At this point, the command does not return a list because all the system resources are assigned to the control domain. Use
the -a option to see all of the devices including the domain to which they're assigned:

jack@dcsw-t5240:~$ ldm list-devices

VCPU
PID %FREE PM
0 0 no
1 0 no
2 0 no
3 0 no
4 0 no
5 0 no
6 0 no
7 0 no
8 0 no
9 0 no
10 0 no
11 0 no
 3
OpenSolaris Operating System opensolaris.com

12 0 no
13 0 no
14 0 no
15 0 no
16 0 no
17 0 no
18 0 no
19 0 no
20 0 no
21 0 no
22 0 no
23 0 no
24 0 no
25 0 no
26 0 no
27 0 no
28 0 no
29 0 no
30 0 no
31 0 no
32 0 no
33 0 no
34 0 no
35 0 no
36 0 no
37 0 no
38 0 no
39 0 no
40 0 no
41 0 no
42 0 no
43 0 no
44 0 no
45 0 no
46 0 no
47 0 no
48 0 no
49 0 no
50 0 no
51 0 no
52 0 no
53 0 no
54 0 no
55 0 no
56 0 no
57 0 no
58 0 no
59 0 no
4
OpenSolaris Operating System opensolaris.com

60 0 no
61 0 no
62 0 no
63 0 no
64 0 no
65 0 no
66 0 no
67 0 no
68 0 no
69 0 no
70 0 no
71 0 no
72 0 no
73 0 no
74 0 no
75 0 no
76 0 no
77 0 no
78 0 no
79 0 no
80 0 no
81 0 no
82 0 no
83 0 no
84 0 no
85 0 no
86 0 no
87 0 no
88 0 no
89 0 no
90 0 no
91 0 no
92 0 no
93 0 no
94 0 no
95 0 no
96 0 no
97 0 no
98 0 no
99 0 no
100 0 no
101 0 no
102 0 no
103 0 no
104 0 no
105 0 no
106 0 no
107 0 no
 5
OpenSolaris Operating System opensolaris.com

108 0 no
109 0 no
110 0 no
111 0 no
112 0 no
113 0 no
114 0 no
115 0 no
116 0 no
117 0 no
118 0 no
119 0 no
120 0 no
121 0 no
122 0 no
123 0 no
124 0 no
125 0 no
126 0 no
127 0 no

MAU
ID CPUSET BOUND
0 (0, 1, 2, 3, 4, 5, 6, 7) primary
1 (8, 9, 10, 11, 12, 13, 14, 15) primary
2 (16, 17, 18, 19, 20, 21, 22, 23) primary
3 (24, 25, 26, 27, 28, 29, 30, 31) primary
4 (32, 33, 34, 35, 36, 37, 38, 39) primary
5 (40, 41, 42, 43, 44, 45, 46, 47) primary
6 (48, 49, 50, 51, 52, 53, 54, 55) primary
7 (56, 57, 58, 59, 60, 61, 62, 63) primary
8 (64, 65, 66, 67, 68, 69, 70, 71) primary
9 (72, 73, 74, 75, 76, 77, 78, 79) primary
10 (80, 81, 82, 83, 84, 85, 86, 87) primary
11 (88, 89, 90, 91, 92, 93, 94, 95) primary
12 (96, 97, 98, 99, 100, 101, 102, 103) primary
13 (104, 105, 106, 107, 108, 109, 110, 111) primary
14 (112, 113, 114, 115, 116, 117, 118, 119) primary
15 (120, 121, 122, 123, 124, 125, 126, 127) primary

MEMORY
PA SIZE BOUND
0x0 512K _sys_
0x80000 1536K _sys_
0x200000 94M _sys_
0x6000000 32M _sys_
0x8000000 96M _sys_
0xe000000 130848M primary
6
OpenSolaris Operating System opensolaris.com

IO
DEVICE PSEUDONYM BOUND OPTIONS
pci@400 pci_0 yes
pci@500 pci_1 yes

Notice in the output above that the system also has 16 MAUs, which are cryptographic Mathmatical Arithmetic Units that
are used to accelarate cryptographic processing. There's a MAU associated with each core and we'll need to deal with
those as well as we free up resources.

Free up 1 MAU for the guest domain by setting the control domain to 15 MAUs:

ldm set-mau 15 primary

Free up 8 CPUs (1 core) for the guest domain by setting the control domain to 120 CPUs:

ldm set-vcpu 120 primary

Free up 16,356 MBs of memory for the guest domain by setting the control domain to 114,492 MB:

jack@dcsw-t5240:~$ ldm set-memory 114492M primary

Initiating delayed reconfigure operation on LDom primary. All configuration
changes for other LDoms are disabled until the LDom reboots, at which time
the new configuration for LDom primary will also take effect.

Now list-devices shows the resources that have been made available for the guest domain:

jack@dcsw-t5240:~$ ldm list-devices

VCPU
PID %FREE
120 100
121 100
122 100
123 100
124 100
125 100
126 100
127 100

MAU
ID CPUSET
15 (120, 121, 122, 123, 124, 125, 126, 127)

MEMORY
PA SIZE
0x1c01c00000 16356M
 7
OpenSolaris Operating System opensolaris.com

Step 4 : Create Services Needed to Support the Guest Domains

Virtual disk and network services need to be established in the control domain to support the guest domains. Create the
virtual disk service named primary-vds:

ldm add-vds primary-vds0 primary

Create the virtual network service. In this example it is named primary-vsw0 and maps to the physical NIC nxge0:

ldm add-vsw net-dev=nxge0 primary-vsw0 primary

Create the Virtual Console Concentrator service. We will use this service to log into the domain. In this example the service
is named primary-vcc0:

ldm add-vcc port-range=5000-5100 primary-vcc0 primary

View all the configured services using the list-services command:

jack@opensolaris:~$ ldm list-services

VCC
NAME LDOM PORT-RANGE
primary-vcc0 primary 5000-5100

VSW
NAME LDOM MAC NET-DEV DEVICE DEFAULT-VLAN-ID PVID
VID MODE
primary-vsw0 primary 00:14:4f:f9:21:7c nxge0 switch@0 1 1

VDS
NAME LDOM VOLUME OPTIONS MPGROUP DEVICE
primary-vds0 primary

Reboot the primary domain so the changes can take effect. Note, if you know you're going to be creating multiple guest
domains, allocate the minimum number of resources you need for the guest domain to minimize the need for future
reboots:

pfexec reboot
...

After reboot, you can view the new configuration for the primary domain, now using only 120 CPUs and 114492M of
memory:

jack@dcsw-t5240:~$ ldm list

NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
primary active -n-c-- SP 120 114492M 0.1% 3m

Finally, enable the virtual network terminal server daemon, which provides console services to the guest domain:

svcadm enable vntsd

8
OpenSolaris Operating System oepnsolaris.com

Step 5: Configure the Guest Domain

Add a new guest domain called myldom:

ldm add-domain myldom

Allocate the MAU:

ldm set-mau 1 myldom

Allocate the 8 free CPUs to the guest domain:

ldm set-vcpu 8 myldom

Allocate the 16356 megabytes of free memory to the guest domain:

ldm set-memory 16356M myldom

Step 6: Create a Virtual Boot Disk for the Guest Domain

The guest domain needs a virtual boot disk. The device can either be an actual disk or a file in a ZFS file system, which was
the option chosen for this How To guide:

Create a new file system:

pfexec zfs create rpool/myldom

Create a file to use as the boot disk for the domain:

pfexec mkfile 10G /rpool/myldom/zdisk

Define a virtual disk service device named vol1 for the zdisk file and associate it with the primary domain:

ldm add-vdsdev /rpool/myldom/zdisk vol1@primary-vds0

Define a virtual disk for the guest domain named vdisk1 that uses the primary domain’s virtual disk service:

ldm add-vdisk vdisk1 vol1@primary-vds0 myldom

Set the auto-boot property to false so when we first start the domain it doesn't try to boot from this empty disk:

ldm set-var auto-boot\?=false myldom

 9
OpenSolaris Operating System opensolaris.com

Step 7: Create a Virtual Disk for the Guest Operating System Installation DVD
Either Solaris or OpenSolaris can be installed in the guest domain. If you prefer to install OpenSolaris, you'll need to use
the Automated Installer (AI). For more information on the AI, see http://www.opensolaris.com/use/Auto_Installer.pdf.

For this How To guide Solaris 10 will be installed as the guest operating system.

Download the Solaris 10 on SPARC ISO from http://www.sun.com/software/solaris/get.jsp.

Then verify it's MD5 checksum against the one provided:

jack@dcsw-t5240:~$ md5sum sol-10-u7-ga-sparc-dvd.iso

c72ad13efb20904251ed334f99f95bd6 sol-10-u7-ga-sparc-dvd.iso

Define a virtual disk service device for the ISO file and associate it with the primary domain. The disk service device
in this example is named solarisdvd:

ldm add-vdsdev ~/sol-10-u7-ga-sparc-dvd.iso solarisdvd@primary-vds0

Define a virtual disk for the guest domain that uses the primary domain’s virtual disk service. The virtual disk in this
example is named dvd:

ldm add-vdisk dvd solarisdvd@primary-vds0 myldom

Step 8: Define a Virtual Network Interface

Define a virtual network interface for the guest domain that uses the primary domain's virtual network service. The virtual
network interface in this example is named vnet0:

ldm add-vnet vnet0 primary-vsw0 myldom

Step 9: Allocate Resources to the Guest Domain

The guest domain is currently inactive:

jack@dcsw-t5240:/rpool/myldom$ ldm list

NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
primary active -n-cv- SP 120 114492M 0.5% 41m
myldom inactive ------ 8 16356M

Run the bind command to activate it:

ldm bind myldom

Not enough free memory present to meet this request
Could not bind requested memory for LDom myldom

Reduce the memory on the guest domain by 256M and try the bind again:

ldm set-memory 16100M myldom

ldm bind
10
OpenSolaris Operating System opensolaris.com

Now its state shows as bound:

jack@dcsw-t5240:/rpool/myldom$ ldm list

NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
primary active -n-cv- SP 120 114492M 0.2% 1h 8m
myldom bound ------ 5000 8 16100M

You can run the list command to see the full configuration of the new domain:

jack@dcsw-t5240:~$ ldm list -l myldom

NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
myldom bound ----- 5000 8 16100M MAC 00:14:4f:fb:b7:bf

HOSTID
0x84fbb7bf

VCPU
VID PID UTIL STRAND
0 120 100%
1 121 100%
2 122 100%
3 123 100%
4 124 100%
5 125 100%
6 126 100%
7 127 100%

MAU
ID CPUSET
15 (120, 121, 122, 123, 124, 125, 126, 127)

MEMORY
RA PA SIZE
0x2000000 0x1c02000000 16100M

NETWORK
NAME SERVICE DEVICE MAC MODE PVID VID
vnet primary-vsw0@primary network@0 00:14:4f:fb:2e:06 1

DISK
NAME VOLUME TOUT DEVICE SERVER MPGROUP
vdisk1 vol1@primary-vds0 disk@0 primary
dvd solarisdvd@primary-vds0 disk@1 primary

VCONS
NAME SERVICE PORT
myldom primary-vcc0@primary 5000
 11
OpenSolars Operating System opensolaris.com

Step 10: Log Into the Guest Domain

Notice from the output above that the virtual console is running at port 5000. From the control domain:
jack@dcsw-t5240:$ telnet localhost 5000
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Connecting to console "myldom" in group "myldom" ....
Press ~? for control options ..

In another terminal window, start the guest domain:

jack@dcsw-t5240:~$ ldm start myldom

LDom myldom started

Back in the guest domains console, you should find yourself at the OK prompt:

T5240, No Keyboard
Copyright 2008 Sun Microsystems, Inc. All rights reserved.
OpenBoot 4.29.0.a, 512 MB memory available, Serial #83528876.
Ethernet address 0:14:4f:fa:8c:ac, Host ID: 84fa8cac.
{0} ok

Step 11: Install Solaris 10 in the Guest Domain

Our next step is to install Solaris. First we need to find the device alias for the dvd virtual disk we configured earlier:

{0} ok devalias
dvd /virtual-devices@100/channel-devices@200/disk@1
vidsk1 /virtual-devices@100/channel-devices@200/disk@0
vnet0 /virtual-devices@100/channel-devices@200/network@0
net /virtual-devices@100/channel-devices@200/network@0
disk /virtual-devices@100/channel-devices@200/disk@0
virtual-console /virtual-devices/console@1
name aliases

Use the dvd device alias to boot and install the domain:

{0} ok boot /virtual-devices@100/channel-devices@200/disk@1

Boot device: /virtual-devices@100/channel-devices@200/disk@1 File and args:
SunOS Release 5.10 Version Generic_139555-08 64-bit
Copyright 1983-2009 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
Configuring devices.
Using RPC Bootparams for network configuration information.
Attempting to configure interface vnet0...
Skipped interface vnet0
Reading ZFS config: done.
Setting up Java. Please wait...
...
12
OpenSolaris Operating System opensolaris.com

At this point you run through a standard installation of Solaris 10. Some items to note, the installer will recognize the
virtual network interface, vnet0:

─ Host Name for vnet0 ─────────────────────────────────

Enter the host name which identifies this system on the network. The name
must be unique within your domain; creating a duplicate host name will cause
problems on the network after you install Solaris.
A host name must have at least one character; it can contain letters,
digits, and minus signs (-).
Host name for vnet0 myldom
───────────────────────────────────────────────
Esc-2_Continue Esc-6_Help

As well as the 10GB virtual disk that was created:

─ Select Disks ─────────────────────────────────────

On this screen you must select the disks for installing Solaris software.
Start by looking at the Suggested Minimum field; this value is the
approximate space needed to install the software you've selected. For ZFS,
multiple disks will be configured as mirrors, so the disk you choose, or the
slice within the disk must exceed the Suggested Minimum value.
NOTE: ** denotes current boot disk
Disk Device Available Space
=============================================================================
[X] c0d0 10152 MB (F4 to edit)
Maximum Root Size: 10152 MB
Suggested Minimum: 6424 MB
────────────────────────────────────────────────
Esc-2_Continue F3_Go Back F4_Edit F5_Exit F6_Help

Step 12: Operate the Guest Domain

When installation completes, the system will boot and can be operated as any other standalone Solaris 10 system.

Note, the system can also be halted from the primary domain. First, verify that the guest domain is active:

jack@dcsw-t5240:~$ ldm list

NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
primary active -n-cv- SP 120 114492M 0.9% 20m
myldom active -n---- 5000 8 16100M 2.1% 9m

Then stop the domain:

jack@dcsw-t5240:~$ ldm stop myldom

LDom myldom stopped

Stopping the domain using ldm stop causes an orderly shutdown. To immediately “power off” the machine, use ldm stop -f.
 13
OpenSolaris Operating System opensolaris.com

For More Information

For more information about Logical Domains and OpenSolaris, check out the following URLs:

Description
Logical Domains OpenSolaris Community
OpenSolaris LDoms Discussion Forum
OpenSolaris Automated Installion Project
OpenSolaris Home Page
http://opensolaris.org/os/community/ldoms/
http://opensolaris.org/os/community/ldoms/discussions/
http://opensolaris.org/os/project/caiman/auto_install/
http://www.opensolaris.com/
opensolaris.com
Sun Microsystems, Inc. 4150 Network Circle, Santa Clara, CA 95054 USA Phone 1-650-960-1300 or 1-800-555-9SUN Web sun.com
©2009 Sun Microsystems, Inc. All rights reserved. Sun, Sun Microsystems, the Sun logo, Solaris and OpenSolaris are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States and other countries.
SunWIN # 487538 Lit # SWWP11968-0 06/09