D74942GC40 Ag PDF

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY.
COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Activity Guide
D74942GC40
Administration
Edition 4.0 | February 2016 | D94670

Learn more from Oracle University at oracle.com/education/
Oracle Solaris Cluster 4.x
Oracle University and XTENTICS CONSULTANCY SERVICES PRIVATE LIMITED use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Copyright © 2016, Oracle and/or its affiliates. All rights reserved.
Disclaimer
This document contains proprietary information and is protected by copyright and other intellectual property laws. You may copy and
print this document solely for your own use in an Oracle training course. The document may not be modified or altered in any way.
Except where your use constitutes "fair use" under copyright law, you may not use, share, download, upload, copy, print, display,
perform, reproduce, publish, license, post, transmit, or distribute this document in whole or in part without the express authorization
of Oracle.
The information contained in this document is subject to change without notice. If you find any problems in the document, please
report them in writing to: Oracle University, 500 Oracle Parkway, Redwood Shores, California 94065 USA. This document is not
warranted to be error-free.
Restricted Rights Notice
If this documentation is delivered to the United States Government or anyone using the documentation on behalf of the United
States Government, the following notice is applicable:
U.S. GOVERNMENT RIGHTS

The U.S. Government’s rights to use, modify, reproduce, release, perform, display, or disclose these training materials are restricted
by the terms of the applicable Oracle license agreement and/or the applicable U.S. Government contract.
Trademark Notice
Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective
owners.
Author
Raghavendra J.S.
Technical Contributors and Reviewers

Hemachandran Namachivayam, Steve McKinty, Lisa Shepherd, Vidhya Cidambi, Uma Sannasi,
Venu Poddar, Nancy Chow, Sambit Nayak, Harish Mallya, Thorsten Fruauf, Sidney Jannatpour,
Stephen Baeten, Bikramjit Saikia
Editor
Chandrika Kennedy
Graphic Designers
Seema Bopaiah, Kavya Bellur
Publishers
Sumesh Koshy, Raghunath M, Asief Baig, Jayanthy Keshavamurthy, Joseph Fernandez
This book was published using: Oracle Tutor

Table of Contents
Practices for Lesson 1: Introduction ........................................................................................................ 1-1
Practices for Lesson 1: Overview............................................................................................................. 1-2
Practices for Lesson 2: Introducing Oracle Solaris Cluster Hardware and Software.............................. 2-1
Practices for Lesson 2............................................................................................................................. 2-2
Practices for Lesson 3: Establishing Cluster Node Console Connectivity.............................................. 3-1
Practice for Lesson 3: Overview .............................................................................................................. 3-2
Practice 3-1: Connecting to the Cluster Node Console ............................................................................. 3-3
Practices for Lesson 4: Preparing for the Oracle Solaris Cluster Installation ......................................... 4-1
Practice 4-1: Preparing for Installation ..................................................................................................... 4-3
Practices for Lesson 5: Configuring the Oracle Solaris Cluster Software............................................... 5-1
Practice 5-1: Installing and Configuring the Oracle Solaris Cluster Software.............................................. 5-3
Practices for Lesson 6: Administering Oracle Solaris Cluster ................................................................ 6-1
Practice 6-1: Performing Basic Cluster Administration .............................................................................. 6-3
Practices for Lesson 7: Configuring Volume Management with Oracle Solaris ZFS .............................. 7-1
Practice 7-1: Configuring Volume Management by Using Oracle Solaris ZFS............................................ 7-3
Practices for Lesson 8: Configuring Volume Management with Solaris Volume Manager ..................... 8-1
Practice 8-1: Configuring Volume Management by Using Solaris Volume Manager ................................... 8-3
Practices for Lesson 9: Managing the Public Network with IPMP ........................................................... 9-1
Practice 9-1: Configuring and Testing IPMP ............................................................................................. 9-3
Practices for Lesson 10: Managing Data Services, Resource Groups, and HA-NFS .............................. 10-1
Practices for Lesson 10: Overview ........................................................................................................... 10-2
Practice 10-1: Installing and Configuring HA for NFS................................................................................ 10-3
Practices for Lesson 11: Configuring Scalable Services and Advanced Resource Group
Relationships ..................................................................................................................................... 11-1
Practice 11-1: Installing and Configuring Apache as a Scalable Service on Oracle Solaris Cluster ............. 11-3
Practices for Lesson 12: Configuring Oracle Solaris Zones in Oracle Solaris Cluster ........................... 12-1
Practice 12-1: Building a Zone Cluster ..................................................................................................... 12-3
Practice 12-2: Importing an Installed Zone into an Existing Zone Cluster................................................... 12-7
Practice 12-3: Configuring a Scalable Application in a Zone Cluster .......................................................... 12-13
Practices for Lesson 13: Exploring Oracle Solaris Cluster Use Cases (Optional) .................................. 13-1
Practices for Lesson 13: Overview (Optional) ........................................................................................... 13-2
Practice 13-1: Integrating Oracle Database 12c into Oracle Solaris Cluster Software as a Failover
Application ......................................................................................................................................... 13-3
Practice 13-2: Install and Configure Oracle Solaris and Oracle Solaris Cluster Software Using Unified
Archives (Demonstration) ................................................................................................................... 13-18
Oracle Solaris Cluster 4.x Administration Table of Contents

i
Oracle Solaris Cluster 4.x Administration Table of Contents

ii
Practices for Lesson 1: Introduction

Chapter 1 - Page 1
Chapter 1
Introduction

Practices for Lesson 1:
Practices for Lesson 1: Overview
Practices Overview
This practice introduces you to your course infrastructure, which you will use for performing the
practices. The practices map to the respective lessons.

Chapter 1 - Page 2
Practices Infrastructure
Your practice environment is based on the Oracle VirtualBox virtualization software. VirtualBox
is a cross-platform virtualization application. Figure 1 shows the configured virtual machines
(VM).
Figure 1: Oracle VirtualBox Virtual Machines
The VirtualBox environment consists of the following VMs:
Name of the VM Description

This VM is installed with Oracle Solaris 11.3
LiveMedia for x86.This VM runs in GUI mode and
enables students to use the pconsole utility to
install, configure, and administer Oracle Solaris
Administrative Workstation Cluster 4.3 software.
This VM also contains the IPS repository for
Oracle Solaris 11.3 OS and Oracle Solaris Cluster
4.3 software, required for running the course
practices.
This VM is installed with Oracle Solaris 11.3 text
Cluster_Node_1 installer for x86 and acts as a regular node in a
two-node cluster.
This VM is installed with Oracle Solaris 11.3 text
Cluster_Node_2 installer for x86 and acts as a regular node in a
two-node cluster.

Chapter 1 - Page 3
Name of the VM Description

This is the Sun ZFS Storage 7000 appliance
Sun_ZFS_Storage_700 simulator and provides shared storage to the
cluster nodes.
This VM is the Sun ZFS Storage 7000 appliance
simulator and provides shared storage to the
Sun_ZFS_Storage_7000_DB
cluster nodes to perform the Oracle Database 12c
practice.
This VM is preinstalled with Oracle Solaris 11.3
Cluster_Node_1_UA OS and Oracle Solaris Cluster 4.3 and used to
perform the Oracle Database 12c practice.
This VM is preinstalled with Oracle Solaris 11.3
Cluster_Node_2_UA OS and Oracle Solaris Cluster 4.23 and used to
perform the Oracle Database 12c practice.
The VMs are further configured to communicate with the host machine through the shared
directory. The shared directories are as listed in the following table:
Resource Name Location Description
Host share directory /opt/ora Contains various course files
Demo files /opt/ora/demo Contains the demonstration files that
are used in the course
OVA files /opt/ora/images Contains a backup of all virtual
machines used in the course.
If any virtual machine gets corrupted
while performing a practice, you can
delete the virtual machine and import
the respective VM from this folder.
However, after reimporting, you will
have to reperform previous practices
to bring the VM to the required start
state of the practice where the VM
got corrupted.
Lab files /opt/ora/lab Contains practice files that are
required to perform certain practices
The details of the shared directories can be verified in the respective VM settings. Each VM has
an entry in the /etc/vfstab file, which stores information about the mount points and related
directories on the system.
Logging In to the Practice Environment

When you first log in to the practice environment, you are prompted to provide a login and
password for the host system:
• Userid: root
• Password: 0racle
After you have gained access to the host system, the user account and password for each
virtual machine is as follows:

Chapter 1 - Page 4
For the Administrative Workstation VM:

User account: oracle1
Password: oracle1
Administrator privileges: root/oracle1
For other VMs:
• User account: oracle1
• Password: oracle1
• Administrator privileges: As the oracle1 user, use su - to switch to the primary
administrator (root) role. The password is oracle1. The oracle1 user switches to
root because root is configured as a role by default. The first username created on the
system (during the OS installation) is the initial privileged user who can assume the
administrator role. This can be verified in the /etc/user_attr database file.
The following table lists the login credentials for the cluster nodes and the administration
workstation:
Virtual Machines Regular User/Password Superuser/Password
Cluster_Node_1 oracle1/oracle1 root/oracle1
Cluster_Node_2 oracle1/oracle1 root/oracle1
Administration Workstation oracle1/oracle1 root/oracle1
Cluster_Node_1_DB oracle1/oracle1 root/oracle1
Cluster_Node_2_DB oracle1/oracle1 root/oracle1
Task: Becoming Familiar with Your Practice Environment (Example)

1. On your host system, start the Oracle VM VirtualBox Manager by double-clicking its icon on
your desktop.

Chapter 1 - Page 5
How to Start the VM?

2. In the Oracle VM VirtualBox Manager window, double-click the Administrative
Workstation VM to start it. Alternatively, you can simply select the Administrative
Workstation VM and click the Start button.
How to Log In to the VM?

3. After the Administrative Workstation VM is powered ON, the Username login screen
appears. Enter oracle1 as the username and click the Log In button.
Note: It might take a few minutes for the Username login screen to appear.

Chapter 1 - Page 6
4. When the password login screen appears, enter the password oracle1 and click the Log
In button.
5. Open a terminal window by right-clicking the desktop and selecting Open Terminal. In the
terminal window, run the su - command to assume the administrator privileges. The
password is oracle1.
oracle1@adminws:~$ su –
Password: oracle1
Oracle Corporation SunOS 5.11 11.3 September 2015
root@adminws:~#

Chapter 1 - Page 7
How to Power Off the VM?

6. At times, you may need to power off a VM and close its window. You may also need to shut
down a VM to comply with the maximum recommended number of VMs running
simultaneously, which is currently limited to three VMs.
Now, practice shutting down a VM by using the Administrative Workstation VM. To shut
down the VM, click the “close” button (x) in the top-right corner of the VM window.
7. When the Close Virtual Machine dialog box appears, select “Power off the machine” and
click OK.

Chapter 1 - Page 8
Note: You can verify that the VM is shut down by checking the status that appears under
the VM’s name in the Oracle VM VirtualBox Manager window. The status for the
Administrative Workstation VM must be “Powered Off.”

Chapter 1 - Page 9

Chapter 1 - Page 10
Introducing Oracle Solaris
Cluster Hardware and
Software
Chapter 2
Practices for Lesson 2: Introducing Oracle Solaris Cluster Hardware and Software
Chapter 2 - Page 1
Practices for Lesson 2
This lesson has no practices.
Practices for Lesson 2: Introducing Oracle Solaris Cluster Hardware and Software
Chapter 2 - Page 2
Establishing Cluster Node
Console Connectivity
Chapter 3
Practices for Lesson 3: Establishing Cluster Node Console Connectivity

Chapter 3 - Page 1
Practice for Lesson 3: Overview
Practice Overview
The Oracle Solaris pconsole utility is used to connect to the cluster nodes. The pconsole
utility creates a host terminal window for each remote host that you specify on the command
line. The utility also opens a central, or master, console window that you can use to send input
to all the nodes at the same time.
Note
• Command output or values may vary across systems.
• To accommodate the complete command output, the font size of the output is reduced in
a few places.
• Your system performance depends on the network speed and network load. If you find
your VM is too slow to proceed with, it is suggested that you restart the VM.

Chapter 3 - Page 2
Practice 3-1: Connecting to the Cluster Node Console
Overview
In this practice, you install and configure the pconsole utility on an administrative workstation
to manage a cluster environment.
In this practice, you perform the following tasks:
• Update host name resolution.
• Install the pconsole utility.
• Configure the pconsole utility.
• Use the pconsole utility.
Preparation
Before proceeding with the tasks in this practice, start the Sun_ZFS_Storage_7000,
Cluster_Node_1, Cluster_Node_2, and Administrative Workstation VMs in that
order. Wait for the VMs to boot properly until you see the login prompt.
This practice assumes that the Oracle Solaris 11.3 OS is already installed on the two cluster
nodes and the administrative workstation.
For the purpose of this practice, use the Administrative Workstation (adminws) system
to install, configure, and use the pconsole utility.
Task 1: Update Host Name Resolution

1. Log in to your Administrative Workstation VM as the oracle1 user.
2. Verify that the Oracle Solaris 11.3 OS is installed on the administrative workstation.
$ cat /etc/release
Oracle Solaris 11.3 X86
Copyright (c) 1983, 2015, Oracle and/or its affiliates. All rights reserved.
Assembled 22 June 2015
3. Ensure that the /etc/inet/hosts file has entries for the IP addresses and host names of
the clnode1 and clnode2 cluster nodes.
$ cat /etc/inet/hosts
…
192.168.1.101 cluster-1 clnode1
…
4. Verify that the config/host property of the system/name-service/switch service is
set to “files nis” to ensure that the name resolution first happens by using the
/etc/inet/hosts file before trying any other name service.
$ svcprop system/name-service/switch | grep config/host
config/host astring files\ nis
…
If the config/host property is not set, use the svccfg command as the root role to set
the config/host property of the system/name-service/switch service to “files
nis”.

Chapter 3 - Page 3
# svccfg -s system/name-service/switch
svc:/system/name-service/switch> setprop config/host = astring: "files nis"
svc:/system/name-service/switch> quit
# svcadm refresh name-service/switch
# svcprop system/name-service/switch | grep config/host

config/host astring files\ nis
sysconfig/config_properties astring sc_ns_switch_default:config/default
sc_ns_switch_host:config/host sc_ns_switch_netgroup:config/netgroup
root@adminws:~#
Note: Ensure that you exit the root role before proceeding to the next task.
Task 2: Install the pconsole Utility

1. Ensure that you are logged in as the oracle1 user.
2. Verify that the solaris and ha-cluster publishers correspond to the Oracle Solaris 11.3
OS and Oracle Solaris Cluster 4.3 repositories, respectively.
$ pkg publisher
PUBLISHER TYPE STATUS P LOCATION
solaris origin online F http://adminws:10001/
ha-cluster origin online F http://adminws:10002/
Note: By default, the pkg publisher command must list the solaris publisher having
https://pkg.oracle.com as the URI. If not set, type the following commands to set the origin
for the solaris publisher to get the Oracle Solaris 11.3 OS software packages and the
ha-cluster publisher to get the Oracle Solaris Cluster 4.3 software packages from a local
repository:
$ pfexec pkg set-publisher –G ‘*’ –g http://adminws:10001/ solaris
$ pfexec pkg set-publisher –g http://adminws:10002/ ha-cluster
$ pkg publisher
3. Install the terminal/pconsole package.
$ pkg info terminal/pconsole
pkg: info: no packages matching the following patterns you specified are
installed on the system. Try specifying -r to query remotely:
terminal/pconsole
$ pfexec pkg install terminal/pconsole
Packages to install: 1
Services to change: 1
Create boot environment: No
Create backup boot environment: No
DOWNLOAD PKGS FILES XFER (MB) SPEED

Completed 1/1 10/10 0.0/0.0 49.2k/s
PHASE ITEMS
Installing new actions 25/25
Updating package state database Done
Updating package cache 0/0
Updating image state Done
Creating fast lookup database Done

Chapter 3 - Page 4
$ pkg info terminal/pconsole
Name: terminal/pconsole
Summary: Parallel console
Description: pconsole is an administrative tool for working with clusters
of machines: it distributes the commands you type to each node
in the cluster.
Category: Applications/System Utilities
State: Installed
Publisher: solaris
Version: 1.0
Build Release: 5.11
Branch: 0.175.3.0.0.25.3
Packaging Date: June 22, 2015 12:47:25 AM
Size: 73.89 kB
FMRI: pkg://solaris/terminal/pconsole@1.0,5.11-
0.175.3.0.0.25.3:20150622T004725Z
4. Install the Oracle Solaris Cluster man page packages:
$ pkg info ha-cluster/system/manual
pkg: info: no packages matching the following patterns you specified are
installed on the system. Try specifying -r to query remotely:
ha-cluster/system/manual
$ pfexec pkg install ha-cluster/system/manual


Completed 2/2 227/227 0.5/0.5 576k/s
PHASE ITEMS
$ pkg info ha-cluster/system/manual

Name: ha-cluster/system/manual
Summary: Oracle Solaris Cluster Manual Pages
Description: Oracle Solaris Cluster On-Line Manual Pages
Category: System/HA Cluster
State: Installed
Publisher: ha-cluster
Version: 4.3 (Oracle Solaris Cluster 4.3.0.22.0)

Chapter 3 - Page 5
Build Release: 5.11

Branch: 0.22.0
Packaging Date: August 26, 2015 08:13:57 PM
Size: 1.94 MB
FMRI: pkg://ha-cluster/ha-cluster/system/manual@4.3,5.11-
0.22.0:20150826T233558Z
$ pkg info ha-cluster/system/manual/data-services

$ pfexec pkg install ha-cluster/system/manual/data-services

Completed 1/1 74/74 0.1/0.1 255k/s
PHASE ITEMS
$ pkg info ha-cluster/system/manual/data-services

Name: ha-cluster/system/manual/data-services
Summary: Oracle Solaris Cluster Data Services Manual Pages
Description: Oracle Solaris Cluster Data Services On-Line Manual Pages
Category: System/HA Cluster
State: Installed
Build Release: 5.11
Branch: 0.22.0
Packaging Date: August 26, 2015 07:35:54 PM
Size: 338.77 kB
FMRI: pkg://ha-cluster/ha-cluster/system/manual/data-
services@4.3,5.11-0.22.0: 20150826T233558Z
Installing the Oracle Solaris Cluster man page packages on the administrative workstation
enables you to view the various man pages before you actually install the software on the
cluster nodes.

Chapter 3 - Page 6
Task 3: Configure the pconsole Utility

1. Ensure that you are logged in as the oracle1 user.
2. Assign the /usr/cluster/man directory path to the PATH environment variable in the
.profile file of the oracle1 user.
$ vi ~/.profile
...
export PATH=/usr/cluster/man:/usr/bin:/usr/sbin
...
3. Set the environment variables by running the .profile login script.
$ echo $PATH
/usr/bin:/usr/sbin
$ . ~/.profile
$ echo $PATH
/usr/bin:/usr/sbin:/usr/cluster/man
4. Assume the root role.

5. Grant the Parallel Console Access rights profile to the oracle1 user.
# usermod –P +“Parallel Console Access” oracle1
Note: Ignore the following message:

UX: usermod: oracle1 is currently logged in, some changes may not take effect
until next login.
Note: Throughout the practices of this course, you use the pconsole utility as the
oracle1 user with the Parallel Console Access rights profile.
6. Exit the root role.
7. Verify that the Parallel Console Access rights profile is granted to the oracle1 user.
$ profiles | grep –i parallel
Parallel Console Access
Task 4: Use the pconsole Utility

1. Ensure that the Cluster_Node_1 and Cluster_Node_2 VMs acting as the two cluster
nodes are powered on or booted.
2. Ensure that you are logged in to the Administrative Workstation as the oracle1
user.
3. Start the pconsole utility to manage the cluster nodes.
$ pconsole clnode1 clnode2 &
This command opens a host terminal window for each cluster node. It also opens a central
or master console window that propagates what you input there to each of the cluster node
terminals.
The pconsole utility uses the Secure Shell (SSH) protocol to establish a connection with
the cluster nodes. Therefore, on the host terminal windows of the cluster nodes, you should
see a message asking you to confirm the connection to the cluster nodes.
4. Specify oracle1 as the password in the master console window to accept an SSH
connection to the cluster nodes.

Chapter 3 - Page 7
5. Place the cursor in the master console window, and press the Enter key a couple of times.
You should see a response in the host terminal window of each cluster node. The
pconsole utility is now ready for use.
6. Adjust the two host terminal windows for each cluster node and the master console window
on the desktop of the administrative workstation, as shown in the following screenshot:
Summary: Now, you have successfully established console access to the cluster nodes
using the pconsole utility. Adjusting the two host terminal windows and the master
console window is just for your convenience and is not a mandatory step.

Chapter 3 - Page 8
Preparing for the Oracle
Solaris Cluster Installation
Chapter 4
Practices for Lesson 4: Preparing for the Oracle Solaris Cluster Installation
Chapter 4 - Page 1
Practice Overview
In this practice, you verify the Oracle Solaris 11.3 OS installed on your practice systems, identify
a cluster topology, select quorum devices, verify the cluster interconnect configuration, and
select public network interfaces.
Note
a few places.
your VM too slow to proceed with, it is suggested that you restart the VM.
Chapter 4 - Page 2
Practice 4-1: Preparing for Installation
Overview
In this practice, you prepare your assigned cluster nodes for installing the Oracle Solaris Cluster
4.3 software.
• Verify the operating system environment.
• Identify the cluster topology.
• Select quorum devices.
• Verify the cluster-private interconnect configuration.
• Select public network interfaces.
Preparation
To begin this practice, you must be connected to the cluster nodes through the pconsole utility
from the administrative workstation and you must be logged in as the oracle1 user.
Task 1: Verify the Operating System Environment

1. Verify that the Oracle Solaris 11.3 operating system is installed on the cluster nodes.
$ cat /etc/release
Assembled 06 October 2015
2. Record the size of (physical) memory on the designated cluster nodes by using the
/usr/sbin/prtconf command.
$ /usr/sbin/prtconf | grep –i memory
Example
$ /usr/sbin/prtconf | grep -i mem
Memory size: 4096 Megabytes
Node 1 memory: _______________
Node 2 memory: _______________
3. Determine whether ZFS is the root file system on the cluster nodes by using the following
commands:
$ df –kh
Filesystem Size Used Available Capacity Mounted on
rpool/ROOT/solaris 49G 2.9G 42G 7% /
....
$ zpool status
pool: rpool
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM

rpool ONLINE 0 0 0
Chapter 4 - Page 3
c1t0d0s0 ONLINE 0 0 0
errors: No known data errors
$ zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
rpool 49.8G 7.25G 42.5G 14% 1.00x ONLINE -
4. Verify that at least 750 MB of swap space is given to each cluster node by using the swap
–l command. The swap space must be in zvol.
$ swap -l
Task 2: Identify the Cluster Topology

1. Record the desired topology configuration of your cluster in the following table:
Number of nodes
Number of storage arrays
Types of storage arrays
2. Verify that the storage arrays in your cluster are properly connected to your target topology.
Note: In this course, you will be creating a two-node cluster by using a preconfigured
Sun_ZFS_Storage_7000 appliance simulator as the storage device.
Task 3: Select Quorum Devices

1. Record the number of quorum devices that you must configure after the cluster host
software installation.
Number of quorum devices: __________
Note: Consult with your instructor if you are not sure about your quorum device
configuration.
2. Decide whether the scinstall utility will automatically be able to choose a quorum device
(it can for a single quorum device for a two-node cluster).
Automatic quorum configuration: (yes/no)?
3. If there can be no automatic quorum configuration (for a three-node cluster, for example),
enter the format command and record the logical path to the disks that you want to use as
quorum disk drives in your storage arrays.
Quorum disks: ________________________________
Chapter 4 - Page 4
4. Press Ctrl + D to exit the format utility.

Note: You will be configuring only a two-node cluster as you perform the practices inside
the VirtualBox environment, so ignore steps 3 and 4. Specify yes for automatic quorum
device selection and configuration by the scinstall utility in step 2.
Task 4: Verify the Cluster-Private Interconnect Configuration

This task describes how to verify the cluster-private interconnect configuration.
Perform the following steps to configure an Ethernet-based, point-to-point interconnect:
1. Determine the names of your cluster-private interconnect adapters by using the dladm
show-phys command, followed by the ipadm show-addr command.
$ dladm show-phys
$ ipadm show-addr
Refrain from assigning IP addresses to the existing public network interfaces.
2. Determine the cluster-private interconnect interfaces for as many interfaces as are
applicable in your practice environment.
The following figure is a schematic representation of a cluster that uses an Ethernet-based,
point-to-point cluster-private interconnect configuration:
Note: In your VirtualBox practice environment, cluster nodes are configured with Ethernet-
based, point-to-point interconnects. Skip the steps listed under the switch-based Ethernet
interconnect section.
Perform the following steps to configure a switch-based Ethernet interconnect:
1. Record the logical names of the cluster interconnect interfaces.
Note: You can use the strategy of determining the cluster transport interfaces by using the
dladm show-phys command, followed by the ipadm show-addr command. Do not use
IP addresses in the existing public network space.
$ dladm show-phys
$ ipadm show-addr
Chapter 4 - Page 5
2. Determine the interconnect interfaces for as many interfaces as are applicable in your
practice environment.
The following figure is a schematic representation of a cluster that uses Ethernet-based,
cluster-private interconnects with switches:
3. Verify that each Ethernet interconnect interface is connected to the correct switch by
pinging each other across the private switches.
Note: If you have any doubt about the private interconnect cabling, consult with your
instructor now. Do not continue this practice until you are confident that your cluster-private
interconnects are cabled correctly, and that you know the names of the cluster-private
transport adapters.
Note: In your given cluster setup environment inside VirtualBox, net1 and net3 are the
designated switch-based cluster-private interconnects.
Task 5: Select Public Network Interfaces

Ask your instructor to help you identify the public network interfaces on each cluster node that
can be used in IPMP groups.
Record the logical names of the potential IPMP Ethernet interfaces on each cluster node, in the
following table:
System Primary IPMP interface Backup IPMP interface
Node 1
Node 2
Note: It is important that you are sure about the logical name of each public network interface.
In your given cluster setup environment inside VirtualBox, net0 is the primary IPMP interface,
while net2 is the backup IPMP interface.
Chapter 4 - Page 6
Configuring the Oracle
Solaris Cluster Software
Chapter 5
Practices for Lesson 5: Configuring the Oracle Solaris Cluster Software

Chapter 5 - Page 1
Practice Overview
In this practice, you install and configure the Oracle Solaris Cluster software.
Note
• To accommodate the complete command output, the font size of the output is reduced
in a few places.

Chapter 5 - Page 2
Practice 5-1: Installing and Configuring the Oracle Solaris Cluster

Software
Overview
In this practice, you install and configure the Oracle Solaris Cluster software. You verify the
practice environment, update local name resolution, install the Oracle Solaris Cluster packages,
configure a new cluster, verify an automatically selected quorum device, configure a quorum
device, and verify the cluster configuration and status.
• Establish connection with the cluster nodes.
• Prepare the cluster node environment.
• Install the Oracle Solaris Cluster packages.
• Authorize the acceptance of cluster configuration commands from the sponsor node.
• Configure a new cluster by using the all-nodes-at-once method.
• Configure a new cluster by using the one-node-at-a-time method.
• Verify the cluster configuration.
Preparation
Before proceeding with this practice, start the Sun_ZFS_Storage_7000, Cluster_Node_1,
Cluster_Node_2, and Administrative Workstation VMs, in that order. Wait for the
VMs to boot properly until you see the login prompt.
For the purpose of this practice, use the Administrative Workstation (adminws) system
to install and configure the Oracle Solaris Cluster software by using the pconsole utility.
Task 1: Establish Connection with the Cluster Nodes

1. Ensure that the Cluster_Node_1 and Cluster_Node_2 VMs acting as the two cluster
nodes are powered on.
2. Log in to the Administrative Workstation VM as the oracle1 user and open a
terminal window.
3. Start the pconsole utility to open the host terminal window for each cluster node and a
master console window.
$ pconsole clnode1 clnode2 &
4. Use the master console window to log in to the two cluster nodes with oracle1 as the
username and password.
Note: You perform the remaining tasks of this practice in the master console window,
unless instructed otherwise.

Chapter 5 - Page 3
Task 2: Prepare the Cluster Node Environment

Perform the following steps in the master console window on the administrative workstation:
1. Verify that the Oracle Solaris 11.3 OS is installed on the cluster nodes.
$ cat /etc/release
Assembled 22 June 2015
2. Ensure that the CONSOLE=/dev/console line in the /etc/default/login file is
commented out on the cluster nodes.
$ cat /etc/default/login | grep CONSOLE
# If CONSOLE is set, root can only login on that device.
#CONSOLE=/dev/console
3. Choose the primary group package of the Oracle Solaris Cluster 4.3 software to install.
Primary group package: _________________________________________________
For the purpose of this practice, you select the ha-cluster-full primary group package
of the Oracle Solaris Cluster 4.3 software to install on the cluster nodes.
4. If necessary, edit the /etc/inet/hosts file of the cluster nodes and add the IP
addresses of your administrative workstation and of all your cluster nodes.
…
192.168.1.100 adminws
…
5. Verify that the Network Auto-Magic (NWAM) feature is disabled on the cluster nodes.
$ netadm list
TYPE PROFILE STATE
ncp Automatic disabled
ncp DefaultFixed online
loc Automatic offline
loc NoNet offline
loc DefaultFixed online
Note: NWAM activates a single network interface and disables all others. For this reason,
NWAM cannot coexist with the Oracle Solaris Cluster software and you must disable it
before you configure or run your cluster. To disable NWAM, you enable the
defaultfixed profile.
# netadm enable -p ncp defaultfixed
# netadm list -p ncp defaultfixed
6. Grant the Service Management and Service Operator rights profiles to the oracle1
user on the cluster nodes.
a. While logged in as the oracle1 user on the cluster nodes, assume the root role.
b. Grant the Service Management and Service Operator rights profiles to the
oracle1 user.

Chapter 5 - Page 4
# usermod –P +”Service Management”,”Service Operator” oracle1
Ignore the following error.
UX: usermod: oracle1 is currently logged in, some changes may not take effect
until next login.
c. Exit the root role to return to the prompt of the oracle1 user on the cluster nodes.
7. Verify that the local_only property of the network/rpc/bind service is set to false
to enable external access to remote procedure call (RPC) communication.
$ svcprop network/rpc/bind:default | grep local_only
config/local_only boolean false
If the local_only property is set to true, use the svccfg command as follows to set the
local_only property of network/rpc/bind service to false.
Note: During the installation of the Oracle Solaris OS, a restricted network profile is used
that disables external access for certain network services. The restricted services include
the RPC communication service, which is required for cluster communication.
Use the following commands to restore external access to RPC communication:
$ svccfg
svc:> select network/rpc/bind
svc:/network/rpc/bind> setprop config/local_only=false
svc:/network/rpc/bind> quit
$ svcadm refresh network/rpc/bind:default
$ svcprop network/rpc/bind:default | grep local_only
config/local_only boolean false
Task 3: Install the Oracle Solaris Cluster Packages

Perform the following steps in the master console window on the administrative workstation as
the oracle1 user:
1. Verify that the solaris and ha-cluster publishers are set.
$ pkg publisher
2. If the publishers are not, set the solaris and ha-cluster publishers for the Oracle
Solaris 11.3 OS and Oracle Solaris Cluster 4.3 software packages.
$ pfexec pkg set-publisher –G ‘*’ –g http://adminws:10001/ solaris
$ pfexec pkg set-publisher –g http://adminws:10002/ ha-cluster
$ pkg publisher
3. Install the primary group package of the Oracle Solaris Cluster 4.3 software that you
selected in Step 3 of Task 2.
$ pfexec pkg install –-accept ha-cluster-full
4. Verify the Oracle Solaris Cluster software package installation.
$ pkg info ha-cluster-full
Name: ha-cluster/group-package/ha-cluster-full

Chapter 5 - Page 5
Summary: Oracle Solaris Cluster full installation group package

Description: Oracle Solaris Cluster full installation group package
Category: Meta Packages/Group Packages
State: Installed
Build Release: 5.11
Branch: 0.24.0
Packaging Date: August 26, 2015 03:29:30 AM
Size: 434.00 B
FMRI: pkg://ha-cluster/ha-cluster/group-package/ha-cluster-
full@4.3,5.11-0.20.0:20150826T233336Z
5. Assign the /usr/cluster/bin and /usr/cluster/man directory paths to the PATH
environment variable in the .profile login script of the oracle1 and root user.
$/# vi .profile
...
export PATH=/usr/cluster/bin:/usr/cluster/man:/usr/bin:/usr/sbin
...
$/# . .profile
$/# echo $PATH
/usr/cluster/bin:/usr/cluster/man:/usr/bin:/usr/sbin
Task 4: Authorize the Acceptance of Cluster Configuration Commands from the

Sponsor Node
1. Determine the sponsor (primary) node that will run the cluster creation command.
The sponsor node is clnode1.
2. On all the nodes that you will configure the cluster, other than the sponsor node, authorize
acceptance of commands from the sponsor node as root role.
clnode2# /usr/cluster/bin/clauth enable -n clnode1
clnode2# /usr/cluster/bin/clauth show
===== Cluster Installation Service =====
svc:/network/rpc/scrinstd:default online
Control node clnode1
Authentication protocol sys

Chapter 5 - Page 6
Task 5: Configure a New Cluster by Using the All-Nodes-at-Once Method
If you prefer to use the one-node-at-a-time method, skip this task and complete Task 6 instead.
Perform the following steps to configure the entire cluster by using the all-nodes-at-once
method.
Complete the following steps in the host terminal window of the clnode1 cluster node as
identified in Task 4:
1. While logged in as the oracle1 user on the clnode1 cluster node, assume the root role.
2. Verify that the /usr/cluster/bin and /usr/cluster/man directory paths are set the
.profile login script of the root role
# echo $PATH
If the path variables are not set, assign the /usr/cluster/bin and /usr/cluster/man
directory paths to the PATH environment variable in the .profile login script of the root
role.
# vi .profile
...
...
# . .profile
# echo $PATH
3. Start the scinstall utility.
# /usr/cluster/bin/scinstall
4. Make the following choices, as the installation proceeds:
a. From the Main menu, select option 1, Create a new cluster or add a
cluster node.
b. From the New Cluster and Cluster Node menu, select option 1, Create a new
cluster.
c. Specify Yes when asked whether you want to continue.
The scinstall utility now checks whether the value of the local_only property of
the network/rpc/bind service is correctly set to false.
d. Press the Enter key to continue.
e. From the Typical or Custom Mode menu, select option 1, Typical.
f. Specify a cluster name, such as cluster1.
g. Specify the name of the other cluster node, such as clnode2.
h. Press Ctrl + D to finish adding the cluster nodes, to display the complete list of cluster
nodes added.
i. Specify Yes to confirm the listing of the cluster nodes.
j. Select the cluster transport adapters that will be used as the cluster-private
interconnect.
For this practice, select net1 and net3, one after the other.
k. Specify No when asked whether you want to disable the automatic quorum device
selection.

Chapter 5 - Page 7
l. Specify Yes when asked whether it is okay to create the new cluster.
m. Specify No when asked to interrupt cluster creation for cluster check errors.
The cluster configuration begins. Somewhere towards the end of the cluster configuration,
the clnode2 cluster node reboots and you lose the pconsole connection to the node.
After the clnode2 cluster node successfully reboots and joins the cluster, the scinstall
utility then proceeds to configure the clnode1 cluster node. After the configuration, the
clnode1 cluster node also reboots to join the cluster. At this point, you now lose the
pconsole connection to the clnode1 cluster node too and only the master console
window remains open. Press Ctrl + D to close the master console window.
After the clnode1 cluster node also successfully reboots and joins the cluster, reestablish
the pconsole connection to the cluster nodes to continue with cluster configuration and
administration.
Note: Refer to the various messages displayed in the console window of the cluster nodes
when they reboot.
Task 6: Configure a New Cluster by Using the One-Node-at-a-Time Method

Perform the following steps to configure the first node in your cluster (the one that will be
assigned node ID 1). You must wait for this node to complete and reboot into the cluster before
configuring the other node.
Note: If you chose to install your cluster by using the all-nodes-at-once method, do not
perform this task.
Complete the following steps in the host terminal window of the clnode1 cluster node:
1. Assume the root role while logged in as the oracle1 user on the clnode1 cluster node.
environment variable in the .profile login script of the root role.
# vi .profile
...
...
# . .profile
# echo $PATH

5. As the installation proceeds, make the following choices:
cluster node.
b. From the Install menu, select option 2, Create just the first node of a new
cluster on this machine.
c. Answer Yes to confirm the creation of just the first node of a new cluster.
d. Press Enter to continue after the local_only property value is checked to be false.
e. From the Type of Installation menu, select option 1, Typical.
f. Furnish your assigned cluster name.
g. Do not allow the cluster check to run.

Chapter 5 - Page 8
h. Furnish the names of the other nodes to be added later.

i. Verify the list of node names.
j. Select the transport adapters. If you are asked (with a tagged VLAN-capable adapter)
whether it is a dedicated cluster transport adapter, answer Yes.
k. For a two-node cluster, do not disable the automatic quorum selection (answer No).
l. Reply Yes to the automatic reboot question.
m. Examine the scinstall command options for correctness. Accept them if they seem
appropriate.
You must wait for this node to complete rebooting to proceed to the second node.
Note: After cluster configuration completes, the node reboots, causing the host terminal
window to close. You have to reestablish the connection to the clnode1 cluster node by
using the pconsole utility. Refer to the messages displayed in the console window of the
cluster nodes.
Complete the following steps in the host terminal window of the clnode2 cluster node:
1. Assume the root role, while logged in as the oracle1 user on the clnode1 cluster node.
environment variable in the .profile login script of the root role.
# vi .profile
...
...
# . .profile
# echo $PATH

5. As the installation proceeds, make the following choices:
cluster node.
b. From the Install menu, select option 3, Add this machine as a node in an
existing cluster.
c. Answer Yes to confirm the addition of a node to an existing cluster.
d. Press Enter to continue after the local_only property value is checked to be false.
e. From the Type of Installation menu, select option 1, Typical.
f. Provide the name of a sponsoring node, clnode1.
g. Provide the name of the cluster that you want to join.
Note: Use cluster show -t global on the first cluster node (the sponsoring
node) if you have forgotten the name of the cluster.
h. Press Enter to continue.
i. Do not allow the cluster check to run.

Chapter 5 - Page 9
j. Use auto-discovery for the transport adapters.

k. Reply Yes to the automatic reboot question.
l. Examine and approve the scinstall command-line options.
You must wait for this node to complete rebooting to proceed to the next step.
6. Exit the master console window on the administrative workstation.
Note: After the cluster configuration completes, the node reboots, causing the host terminal
window to close. You have to reestablish the connection to the clnode2 cluster node by
using the pconsole utility. Refer to the messages displayed in the console window of the
cluster nodes.
Task 7: Verify the Cluster Configuration

Perform the following steps on any one of the cluster nodes as the oracle1 user to verify the
cluster configuration:
1. Run the clquorum status command.
$ clquorum status
=== Cluster Quorum ===
--- Quorum Votes Summary from (latest node reconfiguration) ---
Needed Present Possible

------ ------- --------
2 3 3
--- Quorum Votes by Node (current status) ---
Node Name Present Possible Status

--------- ------- -------- ------
clnode2 1 1 Online
clnode1 1 1 Online
--- Quorum Votes by Device (current status) ---
Device Name Present Possible Status

----------- ------- -------- ------
d1 1 1 Online
You must see the correct node and disk quorum votes.
2. Run the clinterconnect status command.
$ clinterconnect status
=== Cluster Transport Paths ===
Endpoint1 Endpoint2 Status

--------- --------- ------
clnode2:net3 clnode1:net3 Path online
You must see active redundant interconnect paths between all pairs of nodes.

Chapter 5 - Page 10
3. Run the cldevice list -v command.

$ cldevice list –v
DID Device Full Device Path
---------- ----------------
d1 clnode2:/dev/rdsk/c0t600144F0C96CB51800005382CD780002d0
… …
Each shared (dual-ported) DID device must show a logical path from each cluster node.
4. Run the cluster show and cluster status commands.
$ cluster show
$ cluster status
The cluster status, node names, transport configuration, and quorum device information
must be complete.

Chapter 5 - Page 11
Chapter 5 - Page 12
Administering Oracle Solaris
Cluster
Chapter 6
Practices for Lesson 6: Administering Oracle Solaris Cluster

Chapter 6 - Page 1
Practice Overview
In this practice, you perform basic cluster administration tasks such as verifying basic cluster
configuration and status, reassigning a quorum device, adding a quorum server quorum device,
preventing cluster amnesia, and changing the cluster private IP address range.
• Verify basic cluster configuration and status.
• Reassign a quorum device.
• Configure the Oracle Solaris Cluster quorum server software.
• Add a quorum server quorum device.
• Prevent cluster amnesia.
Note
a few places.

Chapter 6 - Page 2
Practice 6-1: Performing Basic Cluster Administration
Overview
In this practice, you perform basic cluster administration tasks such as verifying basic cluster
configuration and status, reassigning a quorum device, adding a quorum server quorum device,
preventing cluster amnesia, and changing the cluster private IP address range.
Preparation
Cluster_Node_1, Cluster_Node_2, and Administrative Workstation VMs, in that
Before you begin this practice, ensure that you have completed the tasks covered in the
previous practices.
For the purpose of this practice, run the pconsole utility on the Administrative
Workstation (adminws) system to open a master console window and the host terminal
window for each cluster node.
Task 1: Verify Basic Cluster Configuration and Status

Perform the following steps as the oracle1 user in the host terminal window of the clnode1
cluster node:
1. Verify the cluster global properties and node properties.
$ cluster show -t global
=== Cluster ===
Cluster Name: cluster1

clusterid: 0x53955907
installmode: disabled
heartbeat_timeout: 10000
heartbeat_quantum: 1000
private_netaddr: 172.16.0.0
private_netmask: 255.255.240.0
max_nodes: 62
max_privatenets: 10
num_zoneclusters: 12
num_xip_zoneclusters: 3
udp_session_timeout: 480
concentrate_load: False
resource_security: SECURE
global_fencing: prefer3
Node List: clnode2, clnode1
$ clnode show
=== Cluster Nodes ===
Node Name: clnode2

Node ID: 1
Enabled: yes
privatehostname: clusternode1-priv

Chapter 6 - Page 3
reboot_on_path_failure: disabled
globalzoneshares: 1
defaultpsetmin: 1
quorum_vote: 1
quorum_defaultvote: 1
quorum_resv_key: 0x5395590700000001
Transport Adapter List: net1, net3
Node Name: clnode1

Node ID: 2
Enabled: yes
privatehostname: clusternode2-priv
reboot_on_path_failure: disabled
globalzoneshares: 1
defaultpsetmin: 1
quorum_vote: 1
quorum_defaultvote: 1
quorum_resv_key: 0x5395590700000002
Transport Adapter List: net1, net3
2. Use the clquorum (clq) command to verify the current cluster membership and the
quorum status.
$ clq status
=== Cluster Quorum ===
--- Quorum Votes Summary from (latest node reconfiguration) ---
Needed Present Possible

------ ------- --------
2 3 3
--- Quorum Votes by Node (current status) ---
Node Name Present Possible Status

--------- ------- -------- ------
clnode2 1 1 Online
clnode1 1 1 Online
--- Quorum Votes by Device (current status) ---
Device Name Present Possible Status

----------- ------- -------- ------
d1 1 1 Online

Chapter 6 - Page 4
3. Record the quorum configuration from the previous step.

Quorum votes needed: _____
Quorum votes present: _____
Quorum votes possible: _____
4. Verify all cluster disk paths.
$ cldev status
=== Cluster DID Devices ===
Device Instance Node Status

--------------- ---- ------
/dev/did/rdsk/d1 clnode1 Ok
clnode2 Ok
/dev/did/rdsk/d10 clnode1 Ok
clnode2 Ok
<… output omitted…>
5. Verify the interconnect status.
$ clintr status
=== Cluster Transport Paths ===
Endpoint1 Endpoint2 Status

--------- --------- ------
6. Verify the revision of the currently installed Oracle Solaris Cluster software on each node.
$ clnode show-rev –v
Oracle Solaris Cluster 4.3.0.20.0 for Solaris 11 i386
ha-cluster/data-service/apache: 4.3-0.20.0
ha-cluster/data-service/dhcp: 4.3-0.20.0
ha-cluster/data-service/dns: 4.3-0.20.0
<… output omitted…>

Chapter 6 - Page 5
Task 2: Reassign a Quorum Device

Assume that your quorum device (or one of your quorum devices) has failed. Perform the
following tasks from any one node in the cluster to reassign the quorum device:
1. Verify the status of your current quorum device(s).
$ clq status
$ cldev status d#
2. Show all disk paths.
$ cldev list -v
3. Choose a different shared disk to be the new quorum device, and add your new quorum
device.
$ clq add new-d-#
$ clq status
$ clq status new-d-#
4. Remove your old broken quorum device.
$ clq remove old-d-#
5. Verify the quorum status again.
$ clq status
Task 3: Configure Oracle Solaris Cluster Quorum Server Software

Perform the following steps to configure your administrative workstation as the quorum server:
1. Ensure that the quorum server machine is connected to a public network that is accessible
to the cluster nodes.
2. Ensure that you are logged in to the selected quorum server machine (admin workstation)
as the oracle1 user.
3. Ensure that the solaris and ha-cluster publishers are valid.
adminws$ pkg publisher
4. Install the Quorum Server group package.
adminws$ pfexec pkg install ha-cluster-quorum-server-full
adminws$ pkg info ha-cluster-quorum-server-full
Name: ha-cluster/group-package/ha-cluster-quorum-server-full
Summary: Oracle Solaris Cluster Quorum Server full group package
Description: Oracle Solaris Cluster Quorum Server full group package
Category: Meta Packages/Group Packages
State: Installed
Build Release: 5.11
Branch: 0.20.0
Packaging Date: June 24, 2015 03:29:57 AM
Size: 434.00 B
FMRI: pkg://ha-cluster/ha-cluster/group-package/ha-cluster-quorum-
server-full@4.3,5.11-0.20.0:20150624T032957Z
oracle1@adminws:~$

Chapter 6 - Page 6

6. Add the Oracle Solaris Cluster Quorum Server binary location to your PATH environment
variable by adding the following entry to the .profile file:
# vi .profile
...
adminws# . .profile
7. Specify quorum configuration information in the /etc/scqsd/scqsd.conf file as follows:
# vi /etc/scqsd/scqsd.conf
…
/usr/cluster/lib/sc/scqsd -d /var/scqsd -i quorumserver -p 9000
8. Save and close the /etc/scqsd/scqsd.conf file.
9. Start the newly configured quorum server.
adminws# /usr/cluster/bin/clquorumserver start quorumserver
Task 4: Add a Quorum Server Quorum Device

To get the IP address of the quorum server that you configured in Task 3, either use the ipadm
command or refer to the /etc/inet/hosts file. In your cluster practice environment, the IP
address of the administrative workstation, now acting as a quorum server, is 192.168.1.100.
You can add the quorum device just by using the IP address, or you can use the host name, if it
is resolvable on your cluster nodes.
Perform the following steps on any one of the cluster nodes as the oracle1 user:
1. Add the quorum server quorum device.
$ clq add -t quorum_server -p qshost=qserver_name_or_IP -p port=9000 myqserver
$ clq status
The name that you give to the device on the cluster side as the last argument to the
command is not important. You can use any name.
2. Delete your disk quorum device(s).
$ clq remove old-d#
$ clq status
3. Add back the disk quorum devices and remove the quorum server quorum device.
$ clq add d#
$ clq status
$ clq remove myqserver
$ clq status
If you have a cluster with nonstorage nodes, having a quorum server quorum device is not
necessarily exactly what you want, because it enables the nonstorage node to stay alive
even if both the storage nodes have failed.
Task 5: Prevent Cluster Amnesia

Perform the following steps to demonstrate how the cluster prevents cluster amnesia by using
persistent reservations on the quorum device:
1. Verify that all nodes are active cluster members.

Chapter 6 - Page 7
2. As the root role, shut down the clnode1 and clnode2 cluster nodes with init 0, one
after the other.
# init 0
Note: Wait until you see the message on the console screen.
…
syncing file systems... done
WARNING: CMM: Node being shut down.
System is halted. It is now safe to power it off.

3. Boot the clnode1 cluster node.
The clnode1 cluster node must hang waiting for operational quorum. This is because the
reservation key for the clnode1 cluster node has been removed from the quorum disk(s).
Note: You may see output like the following on the clnode1 console screen:
…
Aug 21 08:33:00 clnode1 genunix: NOTICE: CMM: Node clnode1: attempting to join cluster.
Aug 21 08:34:02 clnode1 genunix: NOTICE: CMM: Cluster doesn't have operational quorum
yet; waiting for quorum.
…
4. Now boot the clnode2 cluster node.
Note: Both cluster nodes must complete the cluster software startup sequence and join the
clustered operations.

Chapter 6 - Page 8
Configuring Volume
Management with Oracle
Solaris ZFS
Chapter 7
Practices for Lesson 7: Configuring Volume Management with Oracle Solaris ZFS
Chapter 7 - Page 1
Practice Overview
In this practice, you configure volume management by using ZFS.
Chapter 7 - Page 2
Practice 7-1: Configuring Volume Management by Using Oracle

Solaris ZFS
Overview
In this practice, you create a ZFS pool and file system for cluster data, create a snapshot and
then modify your data, and manually migrate your ZFS pool to another node.
• Create a ZFS pool and file system for cluster data.
• Create a snapshot and modify your data.
• Migrate your ZFS pool manually to another node.
Preparation
Task 1: Create a ZFS Pool and File System for Cluster Data
Perform the following steps as the oracle1 user in the host terminal window of the clnode1
cluster node:
1. Select disk drives from the shared storage to use in a new ZFS pool for data.
$ zpool status
$ cldev list –v
$ clq status
Note: Ensure that you do not use any disks already in use in the existing device group.
Note the logical device name of the identified disk, referred to as c#t#d# in the next step.
Consult with your instructor if you are unsure.
3. Create a ZFS pool that provides mirroring for the selected disks.
# zpool create nfspool mirror c#t#d# c#t#d#
# zpool status
Note: Use d3 and d4 to create the mirrored pool.
4. Create a ZFS file system within the pool.
# zfs create nfspool/nfs
# zfs list -r nfspool
NAME USED AVAIL REFER MOUNTPOINT
nfspool 124K 1.95G 32K /nfspool
nfspool/nfs 31K 1.95G 31K /nfspool/nfs
# df –kh
...
nfspool 2.0G 32K 2.0G 1% /nfspool
nfspool/nfs 2.0G 31K 2.0G 1% /nfspool/nfs
...
Chapter 7 - Page 3
5. Change the mount point.

# zfs set mountpoint=/testnfs nfspool/nfs
# df –kh
...
nfspool/nfs 2.0G 31K 2.0G 1% /testnfs
...
6. Populate your file system with some simple files.
# cd /testnfs
# touch file1 file2 file3
# vi my_letter
ZFS is much better than that legacy stuff
# ls
file1 file2 file3 my_letter
# cd
Task 2: Create a Snapshot and Modify Your Data

Perform the following steps as the root role in the host terminal window of the clnode1 cluster
node:
1. Snapshot your ZFS file system.
# zfs snapshot nfspool/nfs@SAVEMYWORK
nfspool/nfs 31.5K 1.95G 31.5K /testnfs
Now, verify the value of the listsnapshots pool property.
# zpool get listsnapshots nfspool

NAME PROPERTY VALUE SOURCE
nfspool listsnapshots off default
As displayed here, the listsnapshots property is off by default. Set it on and verify
its status.
# zpool set listsnapshots=on nfspool
# zpool get listsnapshots nfspool
NAME PROPERTY VALUE SOURCE
nfspool listsnapshots on local
Now, when you display the descendent file systems of the nfspool pool, the
snapshots are included.
nfspool/nfs 31.5K 1.95G 31.5K /testnfs
nfspool/nfs@SAVEMYWORK 0 - 31.5K -
Chapter 7 - Page 4
2. Modify the content of your file system.

# cd /testnfs
# touch file4
# ls
file1 file2 file3 file4 my_letter
# rm file2
# ls
# vi my_letter
ZFS is much better than that legacy stuff
Here is more information about my class
# cd
Task 3: Migrate Your ZFS Pool Manually to Another Node

Perform the following steps as the root role.
On the host terminal window of the clnode1 cluster node:
1. Export your nfspool pool.
# zpool list
nfspool 1.98G 177K 1.98G 0% 1.00x ONLINE -
# zfs list | grep nfs

nfspool/nfs 52K 1.95G 32.5K /testnfs
nfspool/nfs@SAVEMYWORK 19.5K - 31.5K -
# zpool export nfspool
# zpool list
# zpool status
On the host terminal window of the clnode2 cluster node:
3. Import the nfspool pool.
# zpool list
# zpool import nfspool

# zpool list
nfspool 1.98G 214K 1.98G 0% 1.00x ONLINE -
Chapter 7 - Page 5
# zpool status
pool: nfspool
state: ONLINE
config:

nfspool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c0t600144F0C96CB51800005382CDB50004d0 ONLINE 0 0 0
c0t600144F0C96CB51800005382CDCC0005d0 ONLINE 0 0 0
pool: rpool
state: ONLINE
config:

rpool ONLINE 0 0 0

nfspool/nfs 52K 1.95G 32.5K /testnfs
nfspool/nfs@SAVEMYWORK 19.5K - 31.5K -
# df –kh
...
nfspool/nfs 2.0G 32K 2.0G 1% /testnfs
...
Chapter 7 - Page 6
4. Recover the snapshot that migrated along with the pool.

# cd /testnfs
# ls
# zfs rollback nfspool/nfs@SAVEMYWORK
# ls
Chapter 7 - Page 7
Chapter 7 - Page 8
Configuring Volume
Management with Solaris
Volume Manager
Chapter 8
Practices for Lesson 8: Configuring Volume Management with Solaris Volume Manager
Chapter 8 - Page 1
Practice Overview
In this practice, you configure Solaris Volume Manager for use with Oracle Solaris Cluster.
Chapter 8 - Page 2
Practice 8-1: Configuring Volume Management by Using Solaris

Volume Manager
Overview
In this practice, you configure Solaris Volume Manager by performing the following tasks:
• Initialize the Solaris Volume Manager local metadb replicas.
• Select the Solaris Volume Manager demo volume disk drives.
• Configure Solaris Volume Manager disk sets and volumes for a web server.
• Create a cluster file system.
• Test cluster file systems.
• Manage disk sets.
Preparation
Cluster_Node_1, Cluster_ Node_2, and Administrative Workstation VMs, in that
previous practices.
At least one local disk on each cluster node must have a small unused slice that you can use for
the local metadb replicas. The examples in this practice use slice 6 of the boot disk, which must
be reserved ahead of time. During this practice, you create two data service disk sets, each of
which contains a single mirrored volume.
Task 1: Initialize the Solaris Volume Manager Local metadb Replicas

Before you can use Solaris Volume Manager to create disk sets and volumes, you must
initialize the state database and create one or more replicas.
Perform the following steps in the master console window on the Administrative Workstation
VM:
1. Assume the root role, while logged in as the oracle1 user on the cluster nodes.
2. Verify that the boot disk has a small slice, s6, reserved for creating state database replica.
# zpool status
pool: rpool
state: ONLINE
config:

rpool ONLINE 0 0 0
Chapter 8 - Page 3
# prtvtoc /dev/rdsk/c1t0d0s0
* /dev/rdsk/c1t0d0s0 partition map
*
* Dimensions:
* 512 bytes/sector
* 63 sectors/track
* 255 tracks/cylinder
* 16065 sectors/cylinder
* 6526 cylinders
* 6524 accessible cylinders
*
* Flags:
* 1: unmountable
* 10: read-only
*
* First Sector Last
* Partition Tag Flags Sector Count Sector Mount Directory
0 0 00 16065 104679540 104695604
2 5 01 0 104808060 104808059
6 0 00 104695605 112455 104808059
8 1 01 0 16065 16064
root@clnode1:~#
Note: Output varies from system to system.

3. Create three database replicas on the unused boot disk slice by using the metadb
command.
# metadb -a -f -c 3 c#t#d#s6
4. Verify that the replicas are configured and operational.
# metadb
flags first blk block count
a u 16 8192 /dev/dsk/c1t0d0s6
a u 8208 8192 /dev/dsk/c1t0d0s6
a u 16400 8192 /dev/dsk/c1t0d0s6
Task 2: Select the Solaris Volume Manager Demo Volume Disk Drives
Perform the following steps in the host terminal window of the clnode1 cluster node to select
the Solaris Volume Manager demo volume disk drives:
1. As the oracle1 user or root role, use the cldev list -v command to list all of the
available DID drives.
$ cldev list –v
$ clq status
Make note of the DID numbers of shared disks that are already in use. Make sure that you
do not use those shared disks in this practice.
Chapter 8 - Page 4
2. Record the logical path and DID path numbers of the two disks that you will use to create
the demonstration disk set, webds, and volumes in the following table:
Disk Set Volumes Primary Disk Mirror Disk
Webds d100
Note
− You need to record only the last portion of the DID path. The first part is the same
for all DID devices: /dev/did/rdsk.
− Make sure that the disks you select are not local devices. They must be dual-hosted
and available to more than one cluster node.
− For the purpose of this practice, select the d5 and d6 shared disks for the webds
disk set.
Task 3: Configure Solaris Volume Manager Disk Sets and Volumes for a Web
Server
Perform the following steps to create demonstration disk sets and volumes to use in later
practices:
1. Log in as the oracle1 user on the clnode1 cluster node, and assume the root role.
2. Create the webds disk set, and configure the nodes that are physically connected to it.
# metaset -s webds -a -h clnode1 clnode2
3. Add the same nodes as disk set mediators to each disk set.
# metaset -s webds -a -m clnode1 clnode2
4. Add the two disks chosen in Task 2 to the webds disk set.
# metaset -s webds -a /dev/did/rdsk/d5 /dev/did/rdsk/d6
Note: Here, add d5 and d6 to the webds disk set.
5. Verify the status of the new disk set.
# metaset -s webds
Set name = webds, Set number = 1
Host Owner
clnode1 Yes
clnode2
Mediator Host(s) Aliases

clnode1
clnode2
Driv Dbase
d5 Yes
d6 Yes
# medstat -s webds
Mediator Status Golden
clnode1 Ok No
Chapter 8 - Page 5
clnode2 Ok No
# cldg status
=== Cluster Device Groups ===
--- Device Group Status ---
Device Group Name Primary Secondary Status

----------------- ------- --------- ------
webds clnode1 clnode2 Online
6. Create a submirror on each of your disks in the webds disk set.
# metainit -s webds d0 1 1 /dev/did/rdsk/d5s0
webds/d0: Concat/Stripe is setup
# metainit -s webds d1 1 1 /dev/did/rdsk/d6s0
webds/d1: Concat/Stripe is setup
7. Create a mirror volume, d99, by using the d0 submirror.
# metainit -s webds d99 -m d0
webds/d99: Mirror is setup
8. Attach the second submirror, d1, to the mirror volume d99.
# metattach -s webds d99 d1
webds/d99: submirror webds/d1 is attached
9. Create a 500 MB soft partition, d100, on top of your mirror. This is the volume that you
actually use for your file system data.
# metainit -s webds d100 -p d99 500m
d100: Soft Partition is setup
10. Verify the status of the new volume.
# metastat -s webds
webds/d100: Soft Partition
Device: webds/d99
State: Okay
Size: 1024000 blocks (500 MB)
Extent Start Block Block count
0 1024 1024000
webds/d99: Mirror
Submirror 0: webds/d0
State: Okay
Submirror 1: webds/d1
State: Resyncing
Resync in progress: 12 % done
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 4173824 blocks (2.0 GB)
webds/d0: Submirror of webds/d99

State: Okay
Chapter 8 - Page 6

Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
d5s0 0 No Okay Yes
webds/d1: Submirror of webds/d99

State: Resyncing
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
d6s0 0 No Okay Yes
Device Relocation Information:

Device Reloc Device ID
d6 Yes id1,did@n600144f0c96cb51800005382cdf50007
d5 Yes id1,did@n600144f0c96cb51800005382cde70006
Task 4: Create a Cluster File System

Perform the following steps to create a cluster file system in the webds disk set:
1. Ensure that you are logged in as the oracle1 user on the clnode1 cluster node and have
assumed the root role.
2. Create a file system on d100 in the webds disk set by using the following command, and
type y to construct a new file system:
# newfs /dev/md/webds/rdsk/d100
3. On all nodes, create a global mount point for the new file system.
# mkdir /global/web
4. On all nodes, add a mount entry in the /etc/vfstab file for the new file system with the
global mount option.
# vi /etc/vfstab
...
/dev/md/webds/dsk/d100 /dev/md/webds/rdsk/d100 \
/global/web ufs 2 yes global
Note: Do not use the line continuation character (\) in the vfstab file.
5. On the clnode1 cluster node, mount the /global/web file system.
# mount /global/web
6. Verify that the file system is mounted and available on all nodes.
# mount | grep global/web
# ls /global/web
lost+found
# df –kh | grep global/web
Chapter 8 - Page 7
Task 5: Test Cluster File Systems
Perform the following steps to test the accessibility of the webds cluster file system from
different cluster nodes:
1. Ensure that you are logged in as the oracle1 user on both the cluster nodes and have
assumed the root role.
2. On the clnode2 cluster node, move into the /global/web file system.
# cd /global/web
3. On the clnode1 cluster node, try to unmount the /global/web file system.
# umount /global/web
You should get an error that the file system is busy.
4. On the clnode2 cluster node, move out of the /global/web file system (cd /) and try to
unmount it again on the clnode1 cluster node.
5. On the clnode2 cluster node, mount the /global/web file system.
6. Try unmounting and mounting /global/web from all nodes.
Ensure that the /global/web cluster file system remains mounted on the clnode1
cluster node before you proceed to the next task.
Task 6: Manage Disk Sets

Perform the following steps to migrate a disk set between cluster nodes:
1. Ensure that you are logged in as the oracle1 user on the clnode1 cluster.
2. Make sure that the webds disk set is online.
$ cldg status

----------------- ------- --------- ------
Note: You can bring a device group online to a selected node as follows:
# cldg switch -n node_to_switch_to devgrpname
Chapter 8 - Page 8
3. Verify the current demonstration disk set configuration.

$ cldg show
=== Device Groups ===
Device Group Name: webds

Type: SVM
failback: false
Node List: clnode1, clnode2
preferenced: true
numsecondaries: 1
diskset name: webds
4. Reboot the clnode1 cluster node.
5. On the clnode2 cluster node, verify that the webds disk set has automatically migrated to
the clnode2 cluster node by using the cldg status command.
$ cldg status

----------------- ------- --------- ------
6. After clnode1 reboots, the webds disk set should remain mastered by the clnode2
cluster node.
7. Use the cldg switch command from either node to migrate the webds disk set to the
clnode1 cluster node.
# cldg switch -n clnode1 webds
# cldg status
Chapter 8 - Page 9
Chapter 8 - Page 10
Managing the Public Network
with IPMP
Chapter 9
Practices for Lesson 9: Managing the Public Network with IPMP

Chapter 9 - Page 1
Practice Overview
In this practice, you configure and test IPMP.

Chapter 9 - Page 2
Practice 9-1: Configuring and Testing IPMP
Overview
In this practice, you verify the local-mac-address? variable, verify the adapters for the IPMP
group, and enter test addresses in the /etc/inet/hosts file. Next, you verify that IPMP is
configured and test IPMP failover and failback.
• Verify the local-mac-address? variable.
• Verify the adapters for the IPMP group.
• Verify or enter test addresses in the /etc/inet/hosts file.
• Configure IPMP with public network adapters and test addresses.
• Verify that IPMP is configured.
• Verify IPMP failover and failback.
Preparation
previous practices.
Task 1: Verify the local-mac-address? Variable

Perform the following steps in the master console window on the administrative workstation:
1. Log in as the oracle1 user and assume the root role.
2. Configure the cluster nodes to use unique MAC addresses.
Note: All public-network adapters must use network interface cards (NICs) that support
local MAC address assignment. The local-mac-address? variable must use the default
value true for Ethernet adapters. The Oracle Solaris Cluster software does not support a
local-mac-address? value of false for Ethernet adapters.
# eeprom local-mac-address?=true
# eeprom "local-mac-address?"
local-mac-address?=true
3. Reboot the cluster nodes to enable unique MAC address assignment.
# init 6
Task 2: Verify the Adapters for the IPMP Group

Perform the following steps from the host terminal window of each cluster node, one at a time:
1. Ensure that you are logged in as the oracle1 user on the cluster nodes.
2. Identify the redundant network adapters on the public network.
You might have already made a note of this during an earlier practice.

Chapter 9 - Page 3
3. Your primary public network adapter must be the only one currently configured on the
public network.
$ ipadm show-if
IFNAME CLASS STATE ACTIVE OVER
lo0 loopback ok yes --
sc_ipmp0 ipmp ok yes net0
net0 ip ok yes --
net1 ip ok yes --
net3 ip ok yes --
clprivnet0 ip ok yes --
$ ipadm show-addr
4. You can verify your secondary public network adapter by:
• Making sure that it is not configured as a private transport
• Making sure that it can snoop public network broadcast traffic
Assume the root role and use the following commands:
# ipadm create-ip net2
# snoop -d net2
In another terminal window of the same cluster node or another cluster node, use the
following command:
$ ping -s pubnet_broadcast_addr
For example, ping the broadcast address, 192.168.1.255:
$ ping -s 192.168.1.255
Task 3: Verify or Enter Test Addresses in the /etc/inet/hosts File

It is a good idea, although not required, to have test IP addresses listed in the hosts file.
Although one cluster node does not need to know anything about another cluster node’s test
addresses, it is advisable to have all test addresses in the /etc/inet/hosts file for all nodes
to indicate that these addresses are reserved and what they are reserved for.
Perform the following steps from the host terminal window of each cluster node:
1. Log in as the oracle1 user on the cluster nodes and assume the root role.
2. Verify the IP addresses to be used for each network adapter on each cluster node.
3. Type the IP addresses in /etc/inet/hosts on each node, if they are not already there.
The following is a sample entry of test addresses in the /etc/inet/hosts file of a cluster
node. It really does not matter whether you use the same names as another group working
on another cluster as long as the IP addresses are different:
# IPMP Test Addresses
192.168.1.150 clnode1-local-net2
192.168.1.151 clnode1-net0-ipmp-test
192.168.1.153 clnode2-local-net2

Chapter 9 - Page 4
Task 4: Configure IPMP with Public Network Adapters and Test Addresses
Perform the following steps in the host terminal window of each cluster node, one at a time:
2. View the existing IPMP group information.
# ipmpstat –g
GROUP GROUPNAME STATE FDT INTERFACES
sc_ipmp0 sc_ipmp0 ok -- net0
Note: During cluster configuration, the scinstall utility creates an IPMP group called
sc_ipmp0.
3. Add the second public network adapter to the IPMP group:
# ipadm add-ipmp -i net2 sc_ipmp0
# ipmpstat –g
GROUP GROUPNAME STATE FDT INTERFACES
sc_ipmp0 sc_ipmp0 ok -- net2 net0
4. Add a data address to the second public network adapter, net2.
# ipadm create-addr -T static -a 192.168.1.150/24 sc_ipmp0/v4add2
5. Add test addresses to the first and second public network adapters:
# ipadm create-addr -T static -a 192.168.1.151/24 net0/test
Similarly, perform steps 3 through 5 on the other cluster node. Ensure that you use the
correct second public network adapter, data address, and test addresses.
For example, run the following commands on the other cluster node:
# ipadm add-ipmp -i net2 sc_ipmp0
# ipadm create-addr -T static -a 192.168.1.153/24 sc_ipmp0/v4add2
Task 5: Verify That IPMP Is Configured

Perform the following steps in the host terminal window of each cluster, one at a time:
1. Ensure that you are logged in as the oracle1 user on the cluster node.
2. Verify the IPMP configuration.
$ ipmpstat -a
$ ipmpstat -t
$ ipmpstat –i
3. Verify IPMP cluster-wide status with the clnode status –m command.
$ clnode status –m
--- Node IPMP Group Status ---
Node Name Group Name Status Adapter Status

--------- ---------- ------ ------- ------
clnode2 sc_ipmp0 Online net2 Online

Chapter 9 - Page 5
Task 6: Verify IPMP Failover and Failback
Perform the following steps from the host terminal window of a cluster node:
1. As the oracle1 user in the Administrative Workstation system, launch ping -s
nodename, and keep it running.
adminws$ ping –s nodename
2. Simulate the unplugging of the Ethernet cable from the network adapter that currently has
the physical interface of the cluster node on it.
Note: If you do not know to simulate the unplugging of ethernet cable, check with your
instructor.
3. Observe the cluster node messages in the console or in the /var/adm/messages file.
4. Observe the output of the clnode status -m command.

--------- ---------- ------ ------- ------
clnode1 sc_ipmp0 Online net0 Offline
5. Observe the behavior of your command from step 1 (keep it running).
6. Simulate the reattachment of the broken Ethernet cable.
7. Observe the messages and the behavior of your command from step 1.
8. Observe the output of the clnode status -m command.

--------- ---------- ------ ------- ------

Chapter 9 - Page 6
Managing Data Services,
Resource Groups, and HA-
NFS
Chapter 10
Practices for Lesson 10: Managing Data Services, Resource Groups, and HA-NFS
Chapter 10 - Page 1
Practice Overview
In this practice, you install and configure HA for NFS.
Chapter 10 - Page 2
Practice 10-1: Installing and Configuring HA for NFS
Overview
In this practice, you install and configure HA for NFS.
Preparation
previous practices.
Workstation (adminws) system to open the host terminal window for each cluster node and a
master console window.
Task 1: Install and Configure the HA for NFS Agent and Server
In earlier practices, you created the cluster file system for NFS. Confirm that this file system is
available and ready to configure for Oracle Solaris Cluster HA for NFS.
Perform the following steps on the specified cluster nodes as the oracle1 user, unless
otherwise specified:
In the master console window:
1. Verify that the HA for NFS software package is already installed on both cluster nodes.
$ pkg info ha-cluster/data-service/nfs
2. Verify the grace_period property of the NFS protocol.
$ sharectl get -p grace_period nfs
grace_period=90
3. Lower the value of the grace_period property of the NFS protocol.
$ sharectl set -p grace_period=10 nfs
$ sharectl get -p grace_period nfs
grace_period=10
Note: Lowering the grace period speeds up switchovers and failovers of the HA for NFS
resource. Oracle Solaris 11.3 OS uses NFS Version 4 by default. This is a stateful protocol,
which intentionally delays resumption of NFS activity so that clients have a chance to
reclaim their state any time that the server is recovering, which includes any cluster
switchover or failover. The grace_period property controls the length of the delay.
4. Ensure that the /etc/inet/hosts file on the two cluster nodes and the administrative
workstation already contains an entry for the logical host name resource cluster1-nfs.
$ cat /etc/inet/hosts | grep cluster1-nfs
192.168.1.120 cluster1-nfs
5. Identify the node that is currently importing the nfspool ZFS storage pool, which you
created in the lesson titled “Configuring Volume Management with ZFS.”
$ zpool status
Chapter 10 - Page 3
6. On the node identified in step 5, assume the root role and mount the nfspool/nfs ZFS
file system under /global/nfs.
# zfs set mountpoint=/global/nfs nfspool/nfs
# df –kh | grep nfspool/nfs
nfspool/nfs 2.0G 31K 2.0G 1% /global/nfs
In the host terminal window of the cluster node identified in step 5

7. Create a directory within the /global/nfs file system for the HA for NFS resource to
maintain administrative information.
# cd /global/nfs
# mkdir SUNW.nfs
8. Create the dfstab.nfs-res file in the /global/nfs/SUNW.nfs directory and add the
entry to share the /global/nfs/data directory.
# cd SUNW.nfs
# vi dfstab.nfs-res
share -F nfs -o rw /global/nfs/data
9. Create the directory specified in the dfstab.nfs-res file.
# cd /global/nfs
# mkdir data
# touch data/sample.file
# chmod 777 data
# cd
Note: You change the mode of the data directory only for the purpose of demonstration. In
this practice, you would be more specific about the share options in the dfstab.nfs-res
file.
Task 2: Register and Configure the Oracle Solaris Cluster HA for NFS Data
Service
In the host terminal window of any one cluster node:
1. Register the SUNW.nfs and SUNW.HAStoragePlus resource types.
$ clrt register SUNW.nfs
$ clrt register SUNW.HAStoragePlus
$ clrt list –v
Resource Type Node List
------------- ---------
SUNW.LogicalHostname:5 <All>
SUNW.SharedAddress:3 <All>
SUNW.nfs:3.3 <All>
SUNW.HAStoragePlus:11 <All>
Chapter 10 - Page 4
2. Create the NFS failover resource group.

$ clrg create –n clnode1,clnode2 -p Pathprefix=/global/nfs nfs-rg
3. Create the logical host name resource for the NFS failover resource group.
$ clrslh create -g nfs-rg cluster1-nfs
Note: It is safe to ignore the following warning message:

clrslh: Failed to retrieve netmask for the given hostname(s)/IP(s). Will try
again when the resource being brought online.
4. Create the SUNW.HAStoragePlus resource and set the value of Zpools to nfspool.
$ clrs create -t SUNW.HAStoragePlus -g nfs-rg -p Zpools=nfspool nfs-stor
5. Bring the resource group to a managed state and then bring it online.
$ clrg online -M nfs-rg
6. Create the SUNW.nfs resource and set the value of Resource_dependencies to nfs-
stor resource.
$ clrs create -t SUNW.nfs -g nfs-rg -p Resource_dependencies=nfs-stor nfs-res
7. Verify that the data service is online.
$ clrs list –v
Resource Name Resource Type Resource Group
------------- ------------- --------------
nfs-res SUNW.nfs:3.3 nfs-rg
nfs-stor SUNW.HAStoragePlus:11 nfs-rg
cluster1-nfs SUNW.LogicalHostname:5 nfs-rg
$ clrs status
=== Cluster Resources ===
Resource Name Node Name State Status Message

------------- --------- ----- --------------
nfs-res clnode1 Online Online - Successfully started NFS service.
clnode2 Offline Offline
nfs-stor clnode1 Online Online

cluster1-nfs clnode1 Online Online - LogicalHostname online.

$ clrg status
=== Cluster Resource Groups ===
Group Name Node Name Suspended Status

---------- --------- --------- ------
nfs-rg clnode1 No Online
clnode2 No Offline
Chapter 10 - Page 5
Task 3: Verify Access by NFS Clients

In this task, you verify that NFS clients can access the HA for NFS cluster file system.
On the Administrative Workstation:
1. Verify that you can access the cluster file system.
oracle1@adminws:~$ ls /net/cluster1-nfs/global/nfs/data
sample.file
2. Copy the test.nfs file from the /opt/ora/lab directory into your home directory.
oracle1@adminws:~$ cp /opt/ora/lab/test.nfs $HOME
3. View the $HOME/test.nfs script content.
oracle1@adminws:~$ cat test.nfs
#!/bin/ksh
cd /net/cluster1-nfs/global/nfs/data
while :
do
date > date.out
cat date.out
rm date.out
sleep 1
done
oracle1@adminws:~$
Note: When you run the test.nfs script in the next task, it creates and writes a file
containing a timestamp to the NFS-mounted cluster file system. The script also displays the
file to standard output (stdout). This script helps you determine the duration for which the
NFS data service is interrupted during switchovers and takeovers.
Task 4: Observe Oracle Solaris Cluster HA for NFS Failover Behavior

1. Determine the name of the node that is currently primary for the nfs-rg resource group.
$ clrg status nfs-rg

---------- --------- --------- ------
clnode2 No Offline
Chapter 10 - Page 6

2. Run the test.nfs script.
oracle1@adminws:~$ chmod u+x test.nfs
oracle1@adminws:~$ ./test.nfs
3. Transfer control of the NFS resource to the other node.
$ clrg switch -n dest-node nfs-rg
Substitute the name of your offline node for dest-node.
4. Observe the messages displayed by the test.nfs script.
For example:
oracle1@adminws:~$ ./test.nfs
Thursday, August 19, 2015 07:00:56 AM UTC

...
5. How long was the data service interrupted during the switchover from one physical host to
another?
__________________________________________________
6. Stop the test.nfs script.
In the master console window:
7. Verify which file systems the nodes are now mounting and exporting by using the share
command.
__________________________________________________
__________________________________________________
__________________________________________________
8. Observe the additional IP address associated with the Logical Hostname resource
configured as a virtual interface on one of the adapters in your IPMP group by using the
ipadm show-addr command.
$ ipadm show-addr
Chapter 10 - Page 7

9. On one node of the cluster, transfer control of the NFS service back to its preferred host by
using the clrg switch command.
$ clrg switch -n dest-node nfs-rg
Task 5: Generate Cluster Failures and Observe the Behavior of the NFS Failover
Generate failures in your cluster to observe the recovery features.
Try to generate the following failures:
• Power down a cluster node.
• Simulate the unplugging of network cable of one public network interface.
• Simulate the multiple public network failure on a single node.
Try your tests while the resource group is in its normal, nonsuspended state. Repeat some tests
after suspending the resource group:
$ clrg suspend nfs-rg
Note: By suspending a resource group, you will temporarily be suspending the automatic
recovery actions of the resource group. You might need to suspend the automatic recovery
of a resource group to investigate and fix a problem in the cluster.
When you are satisfied with your results, remove the suspension.
$ clrg resume nfs-rg
Task 6: Make a Customized Application Fail Over with a Generic Data Service
Resource
In this task, you can see how easy it is to get any daemon to fail over in the cluster, by using the
Generic Data Service.
Perform the following steps on the specified cluster nodes as the root role:
1. On each cluster node, create a daemon that represents your customized application.
# vi /var/tmp/myappdaemon
#!/bin/ksh
while :
do
sleep 10
done
2. Make sure that the file is executable on all nodes.
3. From any one cluster node, create a new failover resource group for your application.
# clrg create –n clnode1,clnode2 myapp-rg
4. From one cluster node, register the Generic Data Service resource type.
# clrt register SUNW.gds
5. From one cluster node, create the new resource and enable the resource group and
resources.
# clrs create -g myapp-rg -t SUNW.gds -p Start_Command=/var/tmp/myappdaemon \
-p Probe_Command=/bin/true -p Network_aware=false myapp-res
# clrg online -M myapp-rg
# clrg status myapp-rg
# clrs status myapp-res
# ps –ef | grep myappdaemon
Chapter 10 - Page 8
6. Verify the behavior of your customized application.

a. Verify that you can manually switch the group from the other cluster node.
# clrg switch –n dest_node myapp-rg
# clrg status myapp-rg
b. On the destination cluster node, kill the myappdaemon process.

# pfexec kill -9 pid
$ clrs status
Wait a little while and note that the myappdaemon process restarts on the same node.
Also, wait until clrs status shows that the resource is fully online again.
c. Repeat step 6b a few times.
Eventually, the resource group switches over to the other node.
Chapter 10 - Page 9
Chapter 10 - Page 10
Configuring Scalable
Services and Advanced
Resource Group
Relationships
Chapter 11
Practices for Lesson 11: Configuring Scalable Services and Advanced Resource Group Relationships
Chapter 11 - Page 1
Practice Overview
In this practice, you install and configure Apache web server as a scalable data service on the
Oracle Solaris Cluster software.
Chapter 11 - Page 2
Practice 11-1: Installing and Configuring Apache as a Scalable

Service on Oracle Solaris Cluster
Overview
In this practice, you install and configure Apache as a scalable service on the Oracle Solaris
Cluster software.
• Prepare for Apache data service configuration.
• Configure the Apache environment.
• Test the server on each node before configuring the data service resources.
• Register and configure the Apache data service.
• Verify the Apache web server access.
• Observe cluster failures.
• Configure advanced resource group relationships.
Preparation
previous practices.
Task 1: Prepare for Apache Data Service Configuration

Perform the following steps on each node of the cluster:
2. Verify that the Solaris Cluster Apache data service software package is installed.
$ pkg info apache
3. Ensure that the /etc/inet/hosts file on the two cluster nodes and the administrative
workstation contains an entry for the logical host name resource cluster1-web.
$ cat /etc/inet/hosts | grep cluster1-web
192.168.1.121 cluster1-web
Task 2: Configure the Apache Environment

Perform the following steps on any one of the cluster node:
1. While logged in as the oracle1 user on the cluster node, assume the root role.
Chapter 11 - Page 3
2. Make a resource-specific copy of the /usr/apache2/bin/apachectl script and edit it.

# mkdir /global/web/bin
# cp /opt/ora/lab/apachectl /global/web/bin/
# chmod u+w /global/web/bin/apachectl
# vi /global/web/bin/apachectl
a. Add a line to make an Apache runtime directory. This directory is deleted every time
you reboot. Letting this script create it resolves the problem. Just add the line in bold as
the second line of the file.
#!/bin/sh
mkdir -p /var/run/apache2
b. Locate the line:
HTTPD='/usr/apache2/2.2/bin/httpd'
And change it to:
HTTPD='/usr/apache2/2.2/bin/httpd -f /global/web/conf/httpd.conf'
3. Copy the sample /etc/apache2/2.2/httpd.conf example file to the
/global/web/conf/httpd.conf file.
# mkdir /global/web/conf
# cp /etc/apache2/2.2/httpd.conf /global/web/conf/httpd.conf
4. Edit the /global/web/conf/httpd.conf file, and change the entries as shown in the
following table:
Old Entry New Entry
Listen 80 Listen cluster1-web:80
ServerName 127.0.0.1 ServerName cluster1-web
DocumentRoot "/var/apache2/2.2/htdocs" DocumentRoot "/global/web/htdocs"
<Directory "/var/apache2/2.2/htdocs"> <Directory "/global/web/htdocs">
ScriptAlias /cgi-bin/ ScriptAlias /cgi-bin/ "/global/web/cgi-
"/var/apache2/2.2/cgi-bin/" bin/"
<Directory "/var/apache2/2.2/cgi-bin"> <Directory "/global/web/cgi-bin">
Note: The changes mentioned in the table are shown in the order of their appearance in the
httpd.conf file, so you can search for the first place to change, change it, then search for
the next, and so on.
5. Create directories for the HTML and CGI files and populate with the sample files.
# cp -rp /var/apache2/2.2/htdocs /global/web
# cp -rp /var/apache2/2.2/cgi-bin /global/web
6. Copy the test-apache.cgi file from the classroom server to /global/web/cgi-bin.
# cp /opt/ora/lab/test-apache.cgi /global/web/cgi-bin/
You use the test-apache.cgi file to test the scalable service.
7. Make sure that test-apache.cgi can be executed by all users.
# chmod 755 /global/web/cgi-bin/test-apache.cgi
Chapter 11 - Page 4
Task 3: Test the Server on Each Node Before Configuring the Data Service
Resources
Perform the following steps on both the cluster nodes, one node at a time:
1. While logged in as the oracle1 user on a cluster node, assume the root role.
2. Temporarily configure the logical shared address (on one node).
# ipadm create-addr -T static -a 192.168.1.121/24 net0/web
3. Start the server (on that node).
# /global/web/bin/apachectl start
4. Verify that the server is running.
clnode1:/# ps –ef | grep apache2
webservd 4604 4601 0 10:20:05 ? 0:00 /usr/apache2/2.2/bin/httpd
-f /global/web/conf/httpd.conf -k start
root 4601 1 0 10:20:04 ? 0:01 /usr/apache2/2.2/bin/httpd
5. Connect to the Apache web server from the web browser on your Administrative
Workstation by using the following URL:
http://cluster1-web/
6. Stop the Apache web server.

# /global/web/bin/apachectl stop
7. Verify that the server has stopped.
# ps -ef | grep apache2
8. Delete the logical IP address.
# ipadm delete-addr net0/web
Task 4: Register and Configure the Apache Data Service

Perform the following steps on any one of the cluster nodes:
2. Register the resource type required for the Apache data service.
$ clrt register SUNW.apache
3. Create a failover resource group for the shared address resource. If you have more than
two nodes, you can include more than two nodes after the -n.
$ clrg create -n clnode1,clnode2 sa-rg
Chapter 11 - Page 5
4. Add the SharedAddress logical host name resource to the resource group.
$ clrssa create -g sa-rg cluster1-web
clrssa: Failed to retrieve netmask for the given hostname(s)/IP(s). Will try
5. Bring the failover resource group online.
$ clrg online -M sa-rg
6. Create a scalable resource group to run on all nodes of the cluster. (The example assumes
two nodes.)
$ clrg create -S -n clnode1,clnode2 web-rg
7. Add a storage resource to the scalable group.
$ clrs create -g web-rg -t SUNW.HAStoragePlus \
-p FilesystemMountPoints=/global/web -p AffinityOn=true web-stor
8. Add an application resource to the scalable resource group.
$ clrs create -g web-rg -t SUNW.apache -p Bin_dir=/global/web/bin \
-p Scalable=true -p Resource_dependencies=cluster1-web,web-stor \
apache-res
9. Bring the scalable resource group online.
$ clrg online -M web-rg
10. Verify that the data service is online.
$ clrg status

---------- --------- --------- ------
nfs-rg clnode1 No Offline
clnode2 No Online
myapp-rg clnode1 No Online

clnode2 No Offline
sa-rg clnode1 No Online

clnode2 No Offline
web-rg clnode1 No Online

clnode2 No Online
$ clrs status

------------- --------- ----- --------------
nfs-res clnode1 Offline Offline
clnode2 Online Online - Service is online.
Chapter 11 - Page 6
nfs-stor clnode1 Offline Offline

clnode2 Online Online
cluster1-nfs clnode1 Offline Offline
clnode2 Online Online - LogicalHostname online.
myapp-res clnode1 Online Online - Service is online.

cluster1-web clnode1 Online Online - SharedAddress online.

apache-res clnode1 Online Online – Service is online.

clnode2 Online Online - Service is online.
web-stor clnode1 Online Online

clnode2 Online Online
Task 5: Verify the Apache Web Server Access

Perform the following steps on the Administrative Workstation:
1. Open a web browser, such as Firefox.
2. Connect to the web server from a browser anywhere other than from a cluster node by
using http://cluster1-web/cgi-bin/test-apache.cgi.
3. Click the Refresh or Reload button repeatedly on the browser.
The test-apache.cgi script shows the name of the cluster node that is currently
servicing the request. The load balancing is not performed on a round-robin basis, so you
might see several consecutive requests serviced by the same node. Over time, however,
you should see the load balancing to be about 50-50.
Task 6: Observe Cluster Failures

Cause as many of the following failures (one at a time, and fix each one before going on to the
next one) as you can, given your physical access to the cluster.
Induce the following failure scenarios to observe the behavior of the scalable service:
• Fail a single public network interface on the node where the SharedAddress resource
is online.
• Fail all public network interfaces on that node.
• Reboot one of the nodes running the scalable service.
• Shut down one of the cluster nodes running the scalable service.
Task 7: Configure Advanced Resource Group Relationships

In this task, you observe the effects of configuring variations of the RG_affinities property.
Perform the following steps on any one of the cluster nodes:
1. Ensure that you are logged as the oracle1 user on the cluster node.
2. If you have not yet done so, complete Task 6 titled “Make a Customized Application Fail
Over with a Generic Data Service Resource” of Practice 10-1.
Chapter 11 - Page 7
3. Put your myapp-rg resource group offline.

$ clrg offline myapp-rg
4. Create a strong negative affinity with the Apache resource group as the source and
myapp-rg as the target. This means that Apache categorically refuses to run on any node
where myapp-rg is running.
$ clrg set -p RG_affinities=--myapp-rg web-rg
5. Switch myapp-rg to any cluster node where Apache is running. Observe what happens to
the Apache data service. Also, observe the console messages.
$ clrg status web-rg

---------- --------- --------- ------
clnode2 No Online
$ clrg status myapp-rg

---------- --------- --------- ------
myapp-rg clnode1 No Offline
clnode2 No Offline
$ clrg switch -n apacheclusternode myapp-rg

(C258827) NOTICE: Resource group(s) {web-rg on node clnode1} were either
switched to different nodes or brought offline as a result of their strong
resource group affinities for resource group myapp-rg.
$ clrg status
…

---------- --------- --------- ------

clnode2 No Offline
…
web-rg clnode1 No Offline

clnode2 No Online
Chapter 11 - Page 8
6. Switch myapp-rg to another node where Apache is running. Observe what happens to the
Apache data service. Does it come back online on the first node?
$ clrg switch -n apacheclusternode myapp-rg
(C258827) NOTICE: Resource group(s) {web-rg on node clnode2} were either
resource group affinities for resource group myapp-rg.
$ clrg status

---------- --------- --------- ------
…
clnode2 No Online
…
clnode2 No Offline
7. Switch myapp-rg offline. Can Apache come back online on both the cluster nodes? Now
remove the relationship.
$ clrg status
$ clrg set -p RG_affinities="" web-rg
8. Set a weak positive affinity so that myapp-rg (source) always prefers to run on the same
node as nfs-rg (target).
$ clrg set -p RG_affinities=+nfs-rg myapp-rg
9. Print out the value of Nodelist for myapp-rg.
$ clrg show -p Nodelist myapp-rg
=== Resource Groups and Resources ===
Resource Group: myapp-rg

Nodelist: clnode1 clnode2
10. Switch your nfs-rg so that it is not on the preferred node of myapp-rg.
$ clrg status
$ clrg switch -n nonpreferrednode nfs-rg
11. Bring myapp-rg online without specifying the node. Where does it end up? Why does it not
end up on the nonpreferred node of its own node list?
$ clrg online myapp-rg
$ clrg status
12. Switch myapp-rg so it is no longer on the same node as nfs-rg. Check whether this can
be done. Notice that a weak affinity is a preference and not a requirement.
$ clrg switch -n othernode myapp-rg
$ clrg status
Chapter 11 - Page 9
13. Switch myapp-rg offline and now change the affinity to a strong positive affinity with
delegation. What is the difference between ++ and +++ (use +++)? The answer is a few
steps further on.
$ clrg set -p RG_affinities=+++nfs-rg myapp-rg
14. Bring myapp-rg online without specifying the node. Identify the node on which it becomes
online.
$ clrg status

---------- --------- --------- ------
clnode2 No Online

clnode2 No Offline
…
$ clrg online myapp-rg

(C254375) NOTICE: Resource group nfs-rg was also included in the request, due
to a +++ affinity of resource group myapp-rg (possibly transitively).
$ clrg status

---------- --------- --------- ------
clnode2 No Online

clnode2 No Online
…
Note: The above output is given only for your reference.

15. Switch myapp-rg so that it is no longer on the same node as nfs-rg. Note what happens.
$ clrg switch -n othernode myapp-rg
clrg: (C406107) Cannot switch resource group myapp-rg online on node clnode1
because it has strong positive affinities for resource group(s) {nfs-rg},
which are not online on that node.
$ clrg status

---------- --------- --------- ------

clnode2 No Online

clnode2 No Online
…
16. Switch the target group nfs-rg. Note what happens.
$ clrg switch -n othernode nfs-rg
(C258827) NOTICE: Resource group(s) {myapp-rg on node clnode2} were either
resource group affinities for resource group nfs-rg.
$ clrg status

---------- --------- --------- ------
clnode2 No Offline

clnode2 No Offline
…
17. What happens if Resource Group Manager (RGM) wants to fail over the source myapp-rg
because its fault monitor indicates application failure?
Kill myappdaemon on whichever node it is running a few times (be patient with restarts) and
observe the console messages.
Configuring Oracle Solaris
Zones in Oracle Solaris
Cluster
Chapter 12
Practices for Lesson 12: Configuring Oracle Solaris Zones in Oracle Solaris Cluster
Chapter 12 - Page 1
Practice Overview
• Create a zone cluster and recreate your scalable web application within the zone
cluster.
• Import an installed zone into an existing zone cluster.
• Configure the Apache scalable application, running in the global cluster, as a scalable
application in the zone cluster by reusing the Apache binaries of the global cluster.
Chapter 12 - Page 2
Practice 12-1: Building a Zone Cluster
Overview
In this practice, you create a zone cluster and recreate your scalable web application within the
zone cluster.
Preparation
Before proceeding with the tasks in this practice, start Sun_ZFS_Storage_7000,
Cluster_Node_1, Cluster_Node_2, and Administrative Workstation VMs, one after
the other. Wait for the VMs to boot properly until you see the login prompt.
previous practices.
Workstation (adminws) system to open the host terminal window for each cluster node and
the master console window.
Task 1: Create Static Addresses for Zone Cluster Nodes

On all cluster nodes, edit the /etc/inet/hosts file as the root role and provide static IP
addresses on your public subnet for the two nodes (zones) of the zone cluster.
# vi /etc/inet/hosts
192.168.1.140 zcnode1
192.168.1.141 zcnode2
Task 2: Configure a Zone Cluster

Perform the following steps on the specified cluster node to configure a zone cluster:
1. Assume the root role on a cluster node, such as clnode1.
2. Create a zone cluster, myzc.
# clzc configure myzc
myzc: No such zone cluster is configured.
Use 'create' to begin configuring a new cluster zone
clzc:myzc> create
clzc:myzc> set zonepath=/zones/myzc
clzc:myzc> add node
clzc:myzc:node> set physical-host=clnode1
clzc:myzc:node> set hostname=zcnode1
clzc:myzc:node> add net
clzc:myzc:node:net> set address=zcnode1
clzc:myzc:node:net> set physical=sc_ipmp0
clzc:myzc:node:net> end
clzc:myzc:node> end
clzc:myzc> add node
Chapter 12 - Page 3
clzc:myzc:node> end
clzc:myzc> commit
clzc:myzc> exit
3. Verify the zone cluster configuration.
# clzc verify myzc
Task 3: Install the Zone Cluster (On any one of the cluster nodes)
Perform the following steps on the specified cluster node to install a zone cluster:
1. Log in to any one cluster node as the root role.
2. Install the zone cluster myzc by using the clzc install command.
The -v option shows the output of the underlying zoneadm command on the console of
each cluster node:
# clzc status myzc
=== Zone Clusters ===
--- Zone Cluster Status ---
Name Brand Node Name Zone Host Name Status Zone Status
---- ----- --------- -------------- ------ -----------
myzc solaris clnode1 zcnode1 Offline Configured
clnode2 zcnode2 Offline Configured
# clzc install -v myzc

Waiting for zone install commands to complete on all the nodes of the zone
cluster "myzc"...
// zoneadm output for installation appears on the physical node consoles only
# clzc status myzc

---- ----- --------- -------------- ------ -----------
myzc solaris clnode1 zcnode1 Offline Installed
clnode2 zcnode2 Offline Installed
Note: The zone cluster installation could take anywhere from 30 to 45 minutes.
Task 4: Boot the Zone Cluster

From any global cluster node, as the root role, boot all zones in the zone cluster.
Note: The zone name (from the point of view of the zone commands in the global zone) will
be the same as the name of the zone cluster (myzc) on all physical nodes.
# clzc boot myzc
Waiting for zone boot commands to complete on all the nodes of the zone
cluster "myzc"...
# clzc status myzc
Chapter 12 - Page 4
---- ----- --------- -------------- ------ -----------
myzc solaris clnode1 zcnode1 Offline Running
clnode2 zcnode2 Offline Running
Task 5: Access the Zone Consoles

In this task, you will access the zone console and customize your zone.
Perform the following steps on each global cluster node, one at a time:
1. Ensure that you have assumed the root role on each global cluster node.
2. Connect to the zone console on each global cluster node.
# zlogin -C myzc
You still see the zone doing its first boot. Navigate through the System Configuration Tool
screens and specify the following values, when asked for:
• Computer name: zcnode1/zcnode2
• DNS Name Service: Do not configure DNS
• Name Service: None
• Time Zone: timezone
• Root password: 2secure
• Username: oracle1
• User password: oracle1
When you are finished with the system configuration of each zone cluster node, it will
display the console login prompt.
zcnode console login:
3. Log in to each zone cluster node as the oracle1 user and assume the root role.
4. Set your zone cluster environment on both the cluster nodes:
a. Create a better login environment.
# vi ~/.profile
...
export PATH=/usr/cluster/bin:/usr/bin:/usr/sbin
...
# . .profile
b. Make any other desired customizations (for example, change your shell if you want, but
do it carefully).
5. While still logged in to each zone cluster node, explore the zone cluster environment from
the point of view of the zone:
# uname -a
# cat /etc/inet/hosts
# clnode status
# clq status // see an error: You cannot run this command from zone cluster
# clrg status
Chapter 12 - Page 5
# clrs status
# clrt list –v
6. Exit from the zone console by using the ~. or ~~. key combinations.
Task 6: Explore the Zone Cluster from the Global Zone Point of View
Perform the following steps on the global cluster node:
1. Ensure that you are logged as the oracle1 user on the global cluster node, not the zone
cluster node.
2. Explore various cluster resource and resource group status options:
$ clnode status
$ clnode status -Z myzc
$ clnode status -Z all
$ clrg status
$ clrg status -Z myzc
$ clrg status -Z all
$ clrs status
$ clrs status -Z myzc
$ clrs status -Z all
$ clrs list -v
$ clrs list -v -Z myzc
$ clrs list -v -Z all
$ clrt list -v
$ clrt list -v -Z myzc
$ clrt list -v -Z all
Task 7: Experiment with Zone Cluster Failures

Cause as many of the following failures as you can, given access to the cluster. Ensure that you
induce the failure only one at a time, and fix each one before going on to induce the next failure.
Induce the following failure scenarios to observe the behavior of the zone cluster:
• Halt a zone in a zone cluster.
• Reboot a zone in a zone cluster.
• Reboot a cluster node hosting a zone in a zone cluster.
Chapter 12 - Page 6
Practice 12-2: Importing an Installed Zone into an Existing Zone

Cluster
Overview
In this practice, you import the installed zone (zone1) into an existing zone cluster (myzc).
Task 1: Removing a Zone from a Zone Cluster

Perform the following steps to remove the clnode1 cluster node from a zone cluster in an
organized fashion:
Perform the following steps on the specified cluster node to remove the zone cluster:
1. Assume the root role on the clnode1 cluster node.
2. Verify the status of the myzc zone cluster.
root@clnode1:~# clzc status myzc
---- ----- --------- -------------- ------ -----------
myzc solaris clnode1 zcnode1 Online Running
clnode2 zcnode2 Online Running
3. Shut down the zone-cluster node you want to remove by specifying the node and its zone
cluster.
root@clnode1:~# clzc halt -n clnode1 myzc
Waiting for zone halt commands to complete on all the nodes of the zone
cluster "myzc"...
---- ----- --------- -------------- ------ -----------
myzc solaris clnode1 zcnode1 Offline Installed
Chapter 12 - Page 7
4. Uninstall the zone-cluster node.

root@clnode1:~# clzc uninstall -n clnode1 myzc
Are you sure you want to uninstall zone cluster myzc (y/[n])?y
Waiting for zone uninstall commands to complete on all the nodes of the zone
cluster "myzc"...
---- ----- --------- -------------- ------ -----------
myzc solaris clnode1 zcnode1 Offline Configured
5. Remove the zone-cluster node from configuration and verify the status of the myzc zone
cluster.
root@clnode1:~# clzc configure myzc
clzc:myzc> remove node physical-host=clnode1
clzc:myzc> info
zonename: myzc
zonepath: /zones/myzc
autoboot: true
hostid:
brand: solaris
bootargs:
pool:
limitpriv:
scheduling-class:
ip-type: shared
enable_priv_net: true
attr:
name: cluster
type: boolean
value: true
node:
physical-host: clnode2
hostname: zcnode2
net:
address: zcnode2
physical: sc_ipmp0
defrouter not specified
allowed-address not specified
configure-allowed-address not specified
clzc:myzc> exit
root@clnode1:~#
Chapter 12 - Page 8
6. Verify that the node was removed from the zone cluster.
---- ----- --------- -------------- ------ -----------
root@clnode1:~#
Task 2: Importing a New Zone into a Zone Cluster

Perform the following steps on the specified cluster node to import the zone into the zone
cluster:
1. Assume the root role on a cluster node, such as clnode1.
2. Verify that the zone that you want to import is in an installed state.
root@clnode1:~# zoneadm list -cv
ID NAME STATUS PATH BRAND IP
0 global running / solaris shared
1 zone1 running /zones/zone1 solaris Shared
root@clnode1:~#
3. Halt the zone1 zone.
The zone to be imported must be in an installed state.
root@clnode1:~# zoneadm -z zone1 halt
- zone1 installed /zones/zone1 solaris shared
root@clnode1:~#
4. Run the clzonecluster configure command on a configured zone cluster to set the
necessary properties and import the zone.
clzc:myzc> add node
clzc:myzc:node> import-zone zonename=zone1
This operation renames the zone: zone1 to the zone-cluster's name: myzc. Do
you want to proceed ([y]/n)? y
Validating the resources and properties...
Import Operation Failed: Error while validating zone properties.
Zone:'zone1' is not in the 'installed' state.
Zonepath of zone cluster does not match zonepath of the zone:'zone1'
clzc:myzc:node> exit
Are you able to import the zone? Why? Zone paths are different.
When you import an installed zone into a new zone cluster, zone properties such as
zonepath, ip-type, and brand must be defined in the zone cluster configuration before
you run the import-zone command.
Chapter 12 - Page 9
5. Verify that the zonepath of the zone and the zone cluster are the same.
a. Verify the zonepath of the zone1 zone.
clnode1:~# zoneadm list -cv
b. Verify the zonepath of the myzc zone cluster.
# clzc show -v myzc | grep zonepath
or
# zonecfg –z myzc info | grep zonepath
c. Move the zonepath of the zone1 zone if it is different from the myzc zone cluster.
From the above output, you can observe that the zonepath of zone1 and myzc are
different.
root@clnode1:~# zoneadm -z zone1 move /zones/myzc
- zone1 installed /zones/myzc solaris shared
root@clnode1:~#
6. Run the clzonecluster configure command on a configured zone cluster to import
the zone1 zone.
clzc:myzc> add node
clzc:myzc:node> import-zone zonename=zone1
This operation renames the zone: zone1 to the zone-cluster's name: myzc. Do
you want to proceed ([y]/n)? y
Validating the resources and properties...
Extracting Zone Configuration data ...
Importing zone configuration into the zone cluster....
clzc:myzc:node> end
clzc:myzc> commit
Renaming the zone:zone1 to myzc...
clzc:myzc> exit
root@clnode1:~#

---- ----- --------- -------------- ------ -----------
clnode1 zcnode1 Offline Installed
Boot the myzc zone cluster.

root@clnode1:~# clzc boot -n clnode1 myzc
---- ----- --------- -------------- ------ -----------
myzc solaris clnode2 zcnode2 online Running
clnode1 zcnode1 Offline Running
root@clnode1:~#
7. Install the cluster packages on zcnode1.
root@clnode1:~# clzc install-cluster -n clnode1 myzc
Installing the cluster packages in zone cluster "myzc" ...
---- ----- --------- -------------- ------ -----------
root@clnode1:~#
Now, you have successfully imported the zone1 zone into an existing zone cluster
(myzc).
8. Change the hostname from zone1 to zcnode1.
Note: It is not required to change the hostname of zone1. However, you must change the
hostname so that other practices are functional.
root@clnode1# zlogin myzc
root@zone1# svccfg -s system/identity:node setprop config/nodename="zcnode1"
root@zone1# svccfg -s system/identity:node setprop config/loopback="zcnode1"
root@zone1# svccfg -s system/identity:node refresh
root@zone1# svcadm restart system/identity:node

root@zone1# svccfg -s system/identity:node listprop config
config application
config/enable_mapping boolean true
config/ignore_dhcp_hostname boolean false
config/nodename astring zcnode1
config/loopback astring zcnode1
root@zone1# exit
Login and verify the changes.
root@clnode1# zlogin myzc

[Connected to zone 'myzc' pts/2]
Last login: Fri Jan 8 12:59:08 2016 on pts/2
root@zcnode1# hostname
zcnode1
Practice 12-3: Configuring a Scalable Application in a Zone Cluster
Overview
In this practice, you configure the Apache scalable application, running in the global cluster, as
a scalable application in the zone cluster by reusing the Apache binaries of the global cluster.
Preparation
Before proceeding with the tasks in this practice, start Sun_ZFS_Storage_7000,
Cluster_Node_1, Cluster_Node_2, and Administrative Workstation VMs, one after
the other. Wait for the VMs to boot properly until you see the login prompt.
previous practices.
Workstation (adminws) system to open the host terminal window for each cluster node and
the master console window.
Task 1: Delete the Scalable Apache Data Service from the Global Cluster
Perform the following steps on a global cluster node:
1. Ensure that you are logged in as the oracle1 user on the global cluster node.
2. From any node of the global cluster (logged into the physical node, not the zone), delete the
Apache resource groups and all their contained resources.
$ clrg delete -F web-rg
$ clrg delete -F sa-rg
Task 2: Adding the Cluster File System to a Zone Cluster

Perform the following steps to add a cluster file system for the Apache web server data service
to the zone cluster:
1. Assume the root role while logged in as the oracle1 user on any one cluster node.
2. Add the /zonecfs/web-lofs highly available local file system for use by the myzc zone
cluster. Ensure that Oracle Solaris Cluster does not manage the loopback file system
during the initial zone cluster configuration by setting the value of the cluster-control
property to false.
clzc:myzc> add net
clzc:myzc:net> set address=web-lofs-lh
clzc:myzc:net> end
clzc:myzc> add fs
clzc:myzc:fs> set dir=/zonecfs/web-lofs
clzc:myzc:fs> set special=/global/web
clzc:myzc:fs> set type=lofs
clzc:myzc:fs> set cluster-control=false
clzc:myzc:fs> end
clzc:myzc> verify
clzc:myzc> commit
clzc:myzc> exit
# clzc show –v myzc
Zone Cluster Name: myzc
zonename: myzc
autoboot: TRUE
brand: solaris
hostid: <NULL>
bootargs: <NULL>
pool: <NULL>
limitpriv: <NULL>
scheduling-class: <NULL>
ip-type: shared
enable_priv_net: TRUE
resource_security: SECURE
--- Solaris Resources for myzc ---
Resource Name: net

address: web-lofs-lh
physical: auto
Resource Name: fs
dir: /zonecfs/web-lofs
special: /global/web
raw:
type: lofs
options: []
cluster-control: false
--- Zone Cluster Nodes for myzc ---
Node Name: clnode1

hostname: zcnode1
--- Solaris Resources for clnode1 ---
Resource Name: net

address: zcnode1
physical: sc_ipmp0
defrouter: <NULL>
allowed-address: <NULL>
Node Name: clnode2

hostname: zcnode2
--- Solaris Resources for clnode2 ---
Resource Name: net

address: zcnode2
physical: sc_ipmp0
defrouter: <NULL>
allowed-address: <NULL>
Note: Output might differ.

3. Assume the root role and log in to the myzc zone.
# zlogin myzc
4. Create the /zonecfs/web-lofs directory on both the zone cluster nodes.
# mkdir –p /zonecfs/web-lofs
5. Ensure that the /etc/inet/hosts file on both the zone cluster nodes include entries for
the two zone clusters and for web-lofs-lh.
# cat /etc/inet/hosts
...
192.168.1.140 zcnode1
192.168.1.141 zcnode2
...
192.168.1.124 web-lofs-lh
...
6. Exit the root role and then reboot the myzc zone cluster from any one global cluster node.
# clzc reboot myzc
Task 3: Configure the Apache Web Server Environment in the Zone Cluster
Perform the following steps on any one zone cluster node:
1. Ensure that you are logged in as the root role on a zone cluster node.
# zlogin myzc
2. Edit the HTTPD entry in the /zonecfs/web-lofs/bin/apachectl script:
HTTPD='/usr/apache2/2.2/bin/httpd -f /global/web/conf/httpd.conf'
And change it to:
HTTPD='/usr/apache2/2.2/bin/httpd -f /zonecfs/web-lofs/conf/httpd.conf'
3. Edit the /zonecfs/web-lofs/conf/httpd.conf file, and change the entries as shown
in the following table:
Old Entry New Entry
Listen cluster1-web:80 Listen web-lofs-lh:80
ServerName cluster1-web ServerName web-lofs-lh
DocumentRoot "/global/web/htdocs" DocumentRoot "/zonecfs/web-lofs/htdocs"
<Directory "/global/web/htdocs"> <Directory "/zonecfs/web-lofs/htdocs">
ScriptAlias /cgi-bin/ /global/web/cgi- ScriptAlias /cgi-bin/ "/zonecfs/web-
bin/" [single line] lofs/cgi-bin/" [single line]
<Directory "/global/web/cgi-bin"> <Directory "/zonecfs/web-lofs/cgi-bin">
Note: The changes mentioned in the preceding table are shown in the order of their
appearance in the httpd.conf file, so you can search for the first place to change,
change it, then search for the next, and so on.
4. Exit the zone cluster node to return to the global cluster node.
5. Change the myzc zone cluster configuration to ensure that Oracle Solaris Cluster now
manages the loopback file system by setting the value of the cluster-control property
to true.
clzc:myzc> select fs dir=/zonecfs/web-lofs
clzc:myzc:fs> set cluster-control=true
clzc:myzc:fs> end
clzc:myzc> verify
clzc:myzc> commit
clzc:myzc> exit
# clzc show –v myzc
6. Reboot the myzc zone cluster.
# clzc reboot myzc
Task 4: Create a Scalable Apache Data Service in the Zone Cluster

Perform the following steps on any one zone cluster node:
1. Log in as the root role on a zone cluster node.
# zlogin myzc
2. Register resource types in the zone cluster.
# clrt list -v
# clrt register SUNW.apache
# clrt register SUNW.HAStoragePlus
Note: It is safe to ignore warning messages, if any.
# clrt list -v
3. Create a scalable resource group containing the shared address.
# clrg create -n zcnode1,zcnode2 sa-rg
# clrssa create -g sa-rg web-lofs-lh

clrssa: Failed to retrieve netmask for the given hostname(s)/IP(s). Will try
# clrg online -M sa-rg

# clrg status
# clrs status
4. Create a scalable group containing the Apache resource.

# clrg create -S -n zcnode1,zcnode2 web-rg
# clrs create -g web-rg -t SUNW.HAStoragePlus \
-p FilesystemMountPoints=/zonecfs/web-lofs -p AffinityOn=false web-stor
# clrg online -M web-rg
# clrg status
# clrs status
# clrs create -g web-rg -t SUNW.apache -p Bin_dir=/zonecfs/web-lofs/bin \
-p Scalable=true -p Resource_dependencies=web-lofs-lh,web-stor apache-res
# clrg status
# clrs status
Task 5: Verify That the Web Server Works

Perform the following steps on the Administrative Workstation:
1. Open a web browser, such as Firefox.
2. Navigate to http://web-lofs-lh/cgi-bin/test-apache.cgi.
3. Click the reload or refresh button several times to verify the behavior of the scalable
application.
Exploring Oracle Solaris
Cluster Use Cases (Optional)
Chapter 13
Practices for Lesson 13: Exploring Oracle Solaris Cluster Use Cases (Optional)
Chapter 13 - Page 1
Practices for Lesson 13: Overview (Optional)
Practice Overview
• Integrate Oracle Database 12c into Oracle Solaris Cluster software as a failover
application.
• Install and configure Oracle Solaris and Oracle Solaris Cluster software by using
Unified Archives (demonstration).
Chapter 13 - Page 2
Practice 13-1: Integrating Oracle Database 12c into Oracle Solaris

Cluster Software as a Failover Application
Overview
In this practice, you integrate Oracle Database 12c into Oracle Solaris Cluster software as a
failover application.
In this practice, you complete the following tasks:
• Task 1: Create a logical IP entry in the /etc/inet/hosts file.
• Task 2: Create the oracle user and the required group accounts.
• Task 3: Set system parameters (on both nodes as the root role).
• Task 4: Create a shared storage file system (Solaris Volume Manager or ZFS pool).
• Task 5: Prepare the oracle user environment.
• Task 6: Copy the Oracle database files.
• Task 7: Install the required packages.
• Task 8: Disable Access Control of X Server on the Admin Workstation.
• Task 9: Run the runInstaller, netca, and dbca installation scripts.
• Task 10: Prepare an Oracle Instance for Cluster integration.
• Task 11: Register the resource types.
• Task 12: Create resources and resource groups for Oracle.
• Task 13: Verify that Oracle runs properly in the cluster.
Preparation
Oracle database installation is very time consuming and could take anywhere between 120 and
150 minutes. Therefore, to save time, you are given access to a new set of virtual machines
with preinstalled Oracle database using Oracle Solaris ZFS only (refer to Task 4 of this
practice).
Before proceeding with the tasks in this practice, perform the following steps:
1. Power off the following virtual machines:
• Administrative Workstation
• Sun_ZFS_Storage_7000
• Cluster_Node_1
• Cluster_Node_2
2. Reduce the memory of the Administrative Workstation VM.
a. Select the Administrative Workstation VM in VirtualBox Manager.
b. Click Settings->System.
c. Change the Base Memory to 1584 from 2560 MB.
d. Click OK.
Chapter 13 - Page 3
3. Boot the following virtual machines in the given order, one after the other. Wait for the
virtual machines to boot properly until you see the login prompt.
• Administrative Workstation
• Sun_ZFS_Storage_7000_DB
• Cluster_Node_1_DB
Task 1 to Task 9 of this practice are already performed for you to save time and to speed
up the practice. However, these tasks are documented here for your reference and you
just need to verify them before proceeding to the next step.
Note: You must verify the following steps:
• Task 4, Step 2c
• Task 8
You start integrating the Oracle database instance with the cluster from Task 10
onwards.
Task 1: Create a Logical IP Entry in the /etc/inet/hosts File

Note: This task is already performed for you.
Assume the root role and create an entry for a logical IP address for Oracle in the
/etc/inet/hosts file on all the cluster nodes as follows:
# vi /etc/inet/hosts
...
192.168.1.126 ora-lh
Task 2: Create the oracle user and the required group accounts
To create the oracle user and the dba and oinstall group accounts, perform the following
steps from within the master console window on the administrative workstation:
4. Create the dba and oinstall accounts on both the cluster nodes.
# groupadd dba
# groupadd oinstall
5. Create the oracle user on both the cluster nodes.
# useradd -d /export/home/oracle -m -s /bin/bash -g dba \
-G oinstall oracle
80 blocks
6. Create the oracle user password on both the cluster nodes.
# passwd oracle
New Password: oracle1
Re-enter new Password: oracle1
passwd: password successfully changed for oracle
Chapter 13 - Page 4
7. Provide administrative access to the oracle user by assuming the root role on both the
cluster nodes.
# usermod -R root oracle
# usermod -P +"System Administrator" oracle
8. Verify the administrative access provided for the oracle user on both the cluster nodes.
# su - oracle
oracle@clnode1:~$ roles
root
oracle@clnode1:~$ exit
Task 3: Set the system parameters (on both nodes as root role)
Perform the following steps from within the master console window on the administrative
workstation as the root role:
1. Configure system kernel parameters for the oracle user.
# projmod -U oracle -K "project.max-shm-memory=(priv,6g,deny)" default
# projmod -sK "project.max-sem-nsems=(priv,512,deny)" default
# projmod -sK "project.max-sem-ids=(priv,128,deny)" default
# projmod -sK "project.max-shm-ids=(priv,128,deny)" default
2. Set the system swap space.
Oracle database installation requires swap space of up to 4 to 5 GB.
Increase the swap space to meet the Oracle database installation requirement:
# swap -lh
swapfile dev swaplo blocks free
/dev/zvol/dsk/rpool/swap 274,2 4K 1.0G 1.0G
# swap -d /dev/zvol/dsk/rpool/swap
# zfs volsize=5G rpool/swap
# swap -a /dev/zvol/dsk/rpool/swap
# swap -lh
swapfile dev swaplo blocks free
/dev/zvol/dsk/rpool/swap 274,2 4K 5.0G 5.0G
Task 4: Create a Shared Storage File System

You can use either Solaris Volume Manager or ZFS pool to create a shared storage file system
for the Oracle database software.
Note: This task is already performed for you using Oracle Solaris ZFS.
Perform any one of the following two steps of this task:
1. Creating a shared storage file system for Oracle Database software (Solaris Volume
Manager)
Perform the following steps on any node physically connected to the storage as root role.
This practice assumes that local metadbs have been already created.
Just to speed up the practice, you will create unmirrored volumes. You can always add
another disk and mirror the parent volume later at your leisure, or pretend that you were
going to.
Chapter 13 - Page 5
a. Verify that the metadb replicas are configured and operational on both the cluster
nodes.
root@clnode1:~# metadb
flags first blk block count
a u 16 8192 /dev/dsk/c1t0d0s6
a u 8208 8192 /dev/dsk/c1t0d0s6
a u 16400 8192 /dev/dsk/c1t0d0s6
root@clnode1:~#
b. Select a disk from shared storage. Make sure that you do not use any disks already in
use in existing device groups.
root@clnode1:~# cldev list -v
. . .
d7 clnode2:/dev/rdsk/c0t600144F0AE8319BA0000508E382B0010d0
. . .
root@clnode1:~#
Note: For the purpose of this practice, select the d7 shared disk for the orads disk
set.
c. Create the orads disk set by using the shared disk identified in the previous step.
root@clnode1:~# metaset -s orads -a -h clnode1 clnode2
root@clnode1:~# metaset -s orads -a /dev/did/rdsk/d7
d. Create a soft partition (d200) of a mirror volume for Oracle. (Solaris Volume Manager
still calls it a mirror, although we only have one submirror. This gives us flexibility to
add the other half of the mirror at our leisure later.)
root@clnode1:~# metainit -s orads d21 1 1 /dev/did/rdsk/d7s0
orads/d21: Concat/Stripe is setup
root@clnode1:~# metainit -s orads d20 -m d21
orads/d20: Mirror is setup
root@clnode1:~# metainit -s orads d200 -p d20 9g
d200: Soft Partition is setup
e. Create a cluster file system.
root@clnode1:~# newfs /dev/md/orads/rdsk/d200
newfs: construct a new file system /dev/md/orads/rdsk/d100: (y/n)? y
…
f. Create a mount point and an entry in /etc/vfstab on all the nodes.
root@clnode1:~# mkdir /oracledb
root@clnode1:~# vi /etc/vfstab
. . .
/dev/md/orads/dsk/d200 /dev/md/orads/rdsk/d200 /oracledb ufs \
2 yes global
root@clnode1:~#
g. On the node on which you created the storage, mount the file system and modify its
ownership.
root@clnode1:~# mount /oracledb
root@clnode1:~# chown oracle:dba /oracledb
Chapter 13 - Page 6
h. Verify that the file system is mounted and available on both the cluster nodes.
# mount | grep /oracledb
# ls /oracledb
lost+found
# df -kh | grep /oracledb
2. Create a shared storage file system for Oracle database files (ZFS).
Perform the following steps on any one node physically connected to the storage as root
role.
To simplify, we will create a ZFS storage pool with one disk. You can always add another
disk and mirror the existing disk later.
a. Select a disk from shared storage. Make sure that you do not use any disks already in
use in existing device groups.
# zpool status
# metaset
# cldev list -v
. . .
. . .
#
Note: For the purpose of this practice, select the d7 shared disk to create oraclepool.
b. Create a ZFS storage pool and a file system for Oracle. Verify that your file system is
mounted and change its permissions.
root@clnode1:~# zpool create oraclepool c0t600144F0AE8319BA0000508E382B0010d0
root@clnode1:~# zfs create -o mountpoint=/oracledb oraclepool/orafs
root@clnode1:~# df -h
root@clnode1:~# chown oracle:dba /oracledb
Note: You must perform this task.
c. Ensure that the status of oraclepool on clnode1 is online before you proceed to
Task 5.
root@clnode1:~# zpool list
oraclepool 9.94G 6.69G 3.24G 67% 1.00x ONLINE -
If you observe that the oraclepool pool is in OFFLINE or FAULTED or DEGRADED

state, perform the following steps as the root role to bring the state of the
oraclepool pool to ONLINE and to mount the oraclepool/orafs file system.
root@clnode1:~# zpool clear oraclepool

root@clnode1:~# zpool export oraclepool
root@clnode1:~# zpool import oraclepool
root@clnode1:~# zfs mount oraclepool/orafs
Chapter 13 - Page 7
Task 5: Prepare the oracle User Environment

To configure the environment variables required for the Oracle software, run the following
commands from both the cluster nodes:
1. Switch to the oracle user.
root@clnode1:~# su - oracle
oracle@clnode1:~$ pwd
/export/home/oracle
2. Edit the .profile file with the following entries:
oracle@clnode1:~$ vi .profile
...
...
ORACLE_BASE=/oracledb
ORACLE_HOME=$ORACLE_BASE/12c/db_1
ORACLE_UNQNAME=orcl
ORACLE_SID=orcl
PATH=$PATH:/usr/openwin/bin:$ORACLE_HOME/bin
export ORACLE_BASE ORACLE_HOME ORACLE_UNQNAME ORACLE_SID PATH
...
oracle@clnode1:~$ . .profile
oracle@clnode1:~$ exit
Task 6: Copy the Oracle Database Files

1. While logged in as the oracle1 user on clnode1, assume the root role.
2. Create the oracledb12c directory under the root directory and change the ownership as
follows assuming the root role:
root@clnode1:~# cd
root@clnode1:~# mkdir /oracledb12c
root@clnode1:~# chown –R oracle:dba /oracledb12c
3. Copy the Oracle database files from the /opt/ora/oracledb12c/ directory to
/oracledb12c as follows:
root@clnode1:~# cp
/opt/ora/lab/oracledb12c/solaris.x64_12cR1_database_1of2.zip /oracledb12c/
root@clnode1:~# cp
/opt/ora/lab/oracledb12c/solaris.x64_12cR1_database_2of2.zip /oracledb12c/
4. Unzip the copied files.
root@clnode1:~# cd /oracledb12c
root@clnode1:/oracledb12c# unzip solaris.x64_12cR1_database_1of2.zip
root@clnode1:/oracledb12c# unzip solaris.x64_12cR1_database_2of2.zip
root@clnode1:/oracledb12c# chown -R oracle:dba database
root@clnode1:/oracledb12c# ls
database
solaris.x64_12cR1_database_1of2.zip
solaris.x64_12cR1_database_2of2.zip
root@clnode1:/oracledb12c# exit
Chapter 13 - Page 8
Task 7: Install the Required Packages

Perform the following steps from within the master console window on the administrative
workstation as the root role to verify and install the required packages.
1. Install the required packages.
# pkg install compatibility/packages/SUNWxwplt
# pkg install SUNWmfrun
# pkg install xclock
Task 8: Disable Access Control of X Server on the Admin Workstation

1. To allow client GUIs to be displayed, run the following command on the admin workstation:
oracle1@adminws:~$ /usr/openwin/bin/xhost +
access control disabled, clients can connect from any host
2. From the admin workstation, log in as the oracle user to verify the following client GUI on
both the nodes:
$ ssh –X oracle@clnode1
Passwd:
$ xclock
Note: Ensure that the clock appears on the screen.
Task 9: Running the runInstaller Installation Script

1. From the admin workstation, log in to clnode1 as the oracle user and ensure that
xclock appears.
passwd:
$ xclock
2. Change the directory to the location of the Oracle software.
oracle@clnode1:~$ cd /oracledb12c/database
3. Run the runInstaller script.
oracle@clnode1:/oracledb12c/database$ ./runInstaller &
Starting Oracle Universal Installer...
Checking Temp space: must be greater than 180 MB. Actual 3753 MB Passed
Checking swap space: must be greater than 150 MB. Actual 4183 MB Passed
Checking monitor: must be configured to display at least 256 colors. Actual
16777216 Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2012-11-
21_08-23-32AM. Please wait ..
. . .
Note: Output on your system might differ.
Chapter 13 - Page 9
4. Respond to the dialog boxes.

Step Window/Page Description Choices or Values
a. Configure Security Update Unselect I wish to receive security
update via My Oracle Support and click
Next.
Click Yes in the My Oracle Support
Username/Email Address Not Specified
dialog box.
b. Download Software Select Skip software updates and click
Updates Next.
c. Select Installation Option Select Install database software only
and click Next.
d. Grid Installation Option Select Single instance database
installation and click Next.
e. Product Languages Select English and click Next.
f. Database Edition Select Standard Edition One (5.8GB) and
click Next.
g. Installation Location Ensure Oracle Base is set as /oracledb.
Ensure Software Location is set as
/oracledb/12c/db_1 and click Next.
h. Create Inventory Change the inventory directory from
/oraInventory to
/oracledb/oraInventory by browsing and
click Next.
Click Yes to ignore the message in the [INS-
32055] The central Inventory is
located in the Oracle base dialog box,
and then click Next.
i. Operating System Groups Ensure dba is present in the Database
Administrator (OSDBA) Group and click Next.
j. Perform Prerequisite The tests will either all succeed or you will be
Checks warned about swap space requirements.
Click Next, and if you get a pop-up warning
window, click Yes to proceed.
k. Summary Save the db.rsp file and click Install.
l. Install Product Installation starts and it will take around 30 to 45

minutes depending on the system configuration.
Installation time varies from system to system.
Open a terminal window as the root user on the
installer node (clnode1) and run the script:
/oracledb/oraInventory/orainstRoot.sh
Step Window/Page Description Choices or Values

Now run the following script:
/oracledb/12c/db_1/root.sh
Give /usr/bin as the default path name for the
local bin directory.
Click OK in the script prompt window.
m. Finish Click Close to complete the installation.
5. Run netca on any one node (clnode1).

a. From the admin workstation, log in to clnode1 as the oracle user and run netca.
passwd:
$ netca &
b. Respond to the netca dialog boxes.
1. Select Listener configuration and click Next.
2. Select Add and click Next.
3. Type LISTENER in the Listener name text box and click Next.
4. Ensure TCP is the selected protocol and click Next.
5. Select the Use the standard port number of 1521 option button and click
Next.
6. Select No when asked to configure another listener and click Next.
7. Click Next and Finish.
6. Run dbca on any one node (clnode1).
Caution: While working inside the VirtualBox environment, we observe that the cluster
nodes behave absurdly and many a time report short of memory. We advise you to
reboot the two cluster nodes along with the storage appliance node before starting the
dbca script. In such a case, after reboot, you will observe that the oraclepool pool
goes into FAULTED state. Run the following commands as the root role to bring the
state of the oraclepool pool to ONLINE and to mount the oraclepool/orafs file
system:
# zpool clear oraclepool
# zfs mount oraclepool/orafs
a. From the admin workstation, log in to clnode1 as the oracle user and run dbca.
passwd:
$ dbca &
b. Respond to the dbca dialog boxes.
1. Click Next on the welcome screen.
2. Select Create a Database and click Next.
3. Select Advanced Mode and click Next.
4. Select General Purpose or Transaction Processing and click Next.
5. Type orcl in Global Database Name and SID text boxes and click Next.
6. Deselect the Configure Enterprise manager option and click Next.

7. Select Use the Same Administrative Password for All Accounts
and give the password as India1947.
8. Click Next on the Network Configuration (Listener Selection) screen.
9. Ensure that the Use Database File Locations from Template option is
selected, and deselect the Specify Fast Recovery Area option and click
Next.
10. Accept default values (Memory, Sizing, Character set, connection mode) and click
Next.
11. Select Create Database and click Next.
12. Verify Database summary on the Confirmation screen and click Finish.
13. Database Configuration Assistant (on clnode1) will start installation. After the
installation, click Close.
Note: Installation may take 30 to 45 minutes depending on the system configuration.
Task 10: Prepare an Oracle Instance for Cluster Integration

The following steps have to be performed on the same node from which the Oracle software
was installed, in this case clnode1. Perform the following steps as the oracle user to prepare
the Oracle instance for Solaris Cluster software integration:
1. Start the Oracle database instance (as user oracle).
clnode1$ sqlplus /nolog
SQL> connect / as sysdba
Connected to an idle instance.
SQL>STARTUP
ORACLE instance started.
Total System Global Area 1252663296 bytes

Fixed Size 2360584 bytes
Variable Size 452987640 bytes
Database Buffers 788529152 bytes
Redo Buffers 8785920 bytes
Database mounted.
Database opened.
SQL> quit
Note: It may take 15 to 20 minutes to bring up the database.

2. Configure an Oracle user for the fault monitor (as user oracle).
$ sqlplus /nolog
Connected.
SQL> create user sc_fm identified by sc_fm;
SQL> grant create session, create table to sc_fm;
SQL> grant select on v_$sysstat to sc_fm;
SQL> alter user sc_fm default tablespace users quota 1m on users;
SQL> quit
3. Test the new Oracle user account used by the fault monitor (as user oracle).
$ sqlplus sc_fm/sc_fm
SQL> select * from sys.v_$sysstat;
SQL> quit
4. Create a sample table. You can be creative about your data values.
$ sqlplus /nolog
SQL> create table mytable (Name VARCHAR(20), age NUMBER(4));
SQL> insert into mytable values ('vincent', 14);
SQL> insert into mytable values ('theo', 24);
SQL> commit;
SQL> select * from mytable;
SQL> quit
Note: This sample table is created only for a quick verification that the database is running
properly.
5. Shut down the Oracle instance.
$ sqlplus /nolog
SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> quit
Note: Shutting down the database may take up to 10 minutes.

6. Stop the Oracle listener.
$ lsnrctl stop
Note: It is safe to ignore the error/warning messages, if any.

7. Configure the Oracle listener (as user oracle).
a. Edit the $ORACLE_HOME/network/admin/listener.ora file.
$ vi $ORACLE_HOME/network/admin/listener.ora
b. Add the following lines at the top:
SID_LIST_LISTENER =
(SID_LIST =
(SID_DESC =
(SID_NAME = orcl)
(ORACLE_HOME = /oracledb/12c/db_1)
(GLOBAL_DBNAME = orcl)
)
)
c. Modify the HOST variable to match the logical host name ora-lh.
LISTENER =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = ora-lh)(PORT = 1521))
(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))
)
)
8. Configure the tnsnames.ora file (as user oracle).
Modify the HOST variable to match the logical host name ora-lh of
$ORACLE_HOME/network/admin/tnsnames.ora.
oracle@clnode1:~$ vi /oracledb/12c/db_1/network/admin/tnsnames.ora
# tnsnames.ora Network Configuration File:
/oracledb/12c/db_1/network/admin/tnsnames.ora
# Generated by Oracle configuration tools.
LISTENER_ORCL =
ORCL =
(DESCRIPTION =
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = orcl)
)
)
oracle@clnode1:~$
Task 11: Register the Resource Types as root

1. Register the SUNW.HAStoragePlus type.
If necessary (if not done in a previous practice), register the SUNW.HAStoragePlus type
from any one node in the cluster.
# clrt register SUNW.HAStoragePlus
2. Register the SUNW.oracle_server and SUNW.oracle_listener resource types from
any one of the cluster nodes.
# clrt register SUNW.oracle_server
# clrt register SUNW.oracle_listener
Task 12: Create Resources and Resource Groups for Oracle

Perform the following steps from the node that owns the Oracle storage as root role, to create
the resource group and resources necessary for the Oracle data service:
1. Create an empty resource group.
# clrg create -n clnode1,clnode2 ora-rg
2. Add a LogicalHostname resource.

# clrslh create -g ora-rg ora-lh
Note: It is safe to ignore the warning message.
3. Add an HAStoragePlus resource for the Oracle database software.
Note: In this practice, oraclepool is created by using ZFS in Task 4, Step 2. So perform
the step listed under the ZFS section.
For ZFS
# clrs create -g ora-rg -t HAStoragePlus -p Zpools=oraclepool ora-stor
For Solaris Volume Manager
# clrs create -g ora-rg -t HAStoragePlus \
-p FilesystemMountPoints=/oracledb \
-p AffinityOn=true ora-stor
4. Put the resource group online (required by the dependency specification in the next step).
# clrg online -M ora-rg
# clrg status ora-rg
5. Create an oracle_server resource.
# /usr/cluster/bin/clrs create -g ora-rg \
-t oracle_server \
-p Resource_dependencies=ora-stor \
-p ORACLE_SID=orcl -p ORACLE_HOME=/oracledb/12c/db_1 \
-p Alert_log_file=/oracledb/diag/rdbms/orcl/orcl/trace/alert_orcl.log \
-p Connect_string=sc_fm/sc_fm \
-p START_TIMEOUT=3600 \
ora-server-res
# clrs status ora-server-res
Note: This step might take up to 15 to 20 minutes. Ignore any validation errors/warning
messages that are displayed after performing this step.
6. Create an oracle_listener resource.
# clrs create -g ora-rg -t oracle_listener \
-p ORACLE_HOME=/oracledb/12c/db_1 \
-p Listener_name=LISTENER \
-p Resource_dependencies=ora-stor \
ora-listener-res
Note: Ignore any validation errors/warning messages displayed after performing this step.
7. Verify the application.
# clrg status

---------- --------- --------- ------
ora-rg clnode1 No Online
clnode2 No Offline
# clrs status

------------- --------- ----- --------------
ora-listener-res clnode1 Online Online
ora-server-res clnode1 Online Online

ora-stor clnode1 Online Online

ora-lh clnode1 Online Online - LogicalHostname

online.
# ps -ef | grep oracle

…
oracle 4251 1 0 04:27:06 ? 0:01 ora_pmon_orcl
root 3685 3669 0 03:15:32 pts/1 0:00 su - oracle
oracle 4259 1 0 04:27:11 ? 0:04 ora_gen0_orcl
oracle 4289 1 0 04:27:16 ? 0:02 ora_mmon_orcl
oracle 4261 1 0 04:27:11 ? 0:00 ora_mman_orcl
oracle 4279 1 0 04:27:14 ? 0:01 ora_smon_orcl
oracle 4267 1 0 04:27:12 ? 0:01 ora_dbrm_orcl
oracle 4273 1 0 04:27:14 ? 0:01 ora_lgwr_orcl
oracle 4315 1 0 04:29:58 ? 0:00 ora_fbda_orcl
oracle 4255 1 3 04:27:09 ? 0:29 ora_vktm_orcl
oracle 4265 1 0 04:27:12 ? 0:00 ora_diag_orcl
oracle 4311 1 0 04:29:45 ? 0:00 ora_smco_orcl
oracle 4319 1 0 04:30:27 ? 0:00 ora_aqpc_orcl
oracle 4291 1 0 04:27:17 ? 0:01 ora_mmnl_orcl
oracle 4271 1 0 04:27:13 ? 0:01 ora_dbw0_orcl
oracle 4287 1 0 04:27:15 ? 0:01 ora_lreg_orcl
oracle 4730 1 2 04:45:04 ? 0:00 ora_j000_orcl
oracle 4281 1 0 04:27:15 ? 0:00 ora_reco_orcl
oracle 4277 1 0 04:27:14 ? 0:01 ora_ckpt_orcl
oracle 4293 1 0 04:27:17 ? 0:00 ora_d000_orcl
oracle 4387 1 0 04:33:37 ? 0:01 ora_w001_orcl
oracle 4313 1 0 04:29:49 ? 0:01 ora_w000_orcl
oracle 4325 1 0 04:30:38 ? 0:01 ora_q002_orcl
oracle 4676 1 0 04:43:50 ? 0:00 oracleorcl
(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle 4321 1 0 04:30:37 ? 0:00 ora_qm02_orcl
oracle 4634 1 0 04:43:37 ? 0:00
/oracledb/12c/db_1/bin/tnslsnr LISTENER -inherit
oracle 4357 1 0 04:32:13 ? 0:01 ora_cjq0_orcl

oracle 4295 1 0 04:27:18 ? 0:00 ora_s000_orcl
oracle 4309 1 0 04:27:53 ? 0:00 ora_tt00_orcl
root 4524 2507 0 04:36:01 ? 0:00
/opt/SUNWscor/oracle_server/bin/oracle_server_monitor64_v10 -R ora-server-res
-
oracle 4327 1 0 04:30:38 ? 0:00 ora_q003_orcl
root 4679 2507 0 04:44:00 ? 0:00 /usr/bin/ksh
/opt/SUNWscor/oracle_listener/bin/oracle_listener_monitor -R ora-l
Task 13: Verify That Oracle Runs Properly in the Cluster

Perform the following steps as indicated to verify that the Oracle software properly runs in the
cluster:
1. Switch to the oracle user on the cluster node currently running the ora-rg resource
group.
# su - oracle
2. Query the database from the cluster node currently running the ora-rg resource group.
$ sqlplus /nolog
SQL> quit
3. As the root user (from any cluster node), switch the resource group to another node.
# clrg switch -n other-node ora-rg
# clrg status
Note: This step might take up to 10 to 15 minutes.

4. Query the database from the cluster node currently running the ora-rg resource group (as
user oracle).
$ sqlplus /nolog
SQL> insert into mytable values (‘daniel’, 34);
SQL> commit;
SQL> quit
5. Verify that the database fails over properly under various failure conditions:
a. Complete single public network failure on the node on which the resource group is
running
b. Complete failure of the node on which the resource group is running
6. After verifying the various failure conditions, power off the following virtual machines one
after the other:
• Sun_ZFS_Storage_7000_DB
Practice 13-2: Install and Configure Oracle Solaris and Oracle Solaris
Cluster Software Using Unified Archives (Demonstration)
Overview
The Oracle Solaris Unified Archives support feature enables you to use the Automated Installer
(AI) server to install Oracle Solaris Cluster and Oracle Solaris on cluster nodes from Oracle
Solaris Unified Archives. Once an archive is created, you can store the archive as a file until it is
needed.
Note: Oracle Solaris Unified Archives are a new native archive type for Oracle Solaris. Unified
Archives allow multiple system instances to be archived in a single unified file format. Unified
Archives may contain one or more archived instances of Oracle Solaris OS from a single host.
An OS instance may be a global zone, a nonglobal zone, or a kernel zone. These individual
systems may be archived independently or bundled together.
In this demonstration, you will observe how to:
• Set up an Automated Installer (AI) server
• Create an Oracle Solaris unified archive file
• Start the scinstall utility to establish the cluster
• Register the client machines on AI server
• Deploy Oracle Solaris OS and Oracle Solaris Cluster to network clients
• Use the pconsole utility to display the Individual console screens
• Verify the cluster nodes
Assumptions
Adobe Flash Player is already installed on the host machine.
Special note for playing the demonstration in a virtual machine:

• To be able to view demonstration controls in the browser, it is recommended that you
switch to full screen.
• To switch to full-screen mode, in the browser window, go to View  Full Screen.
Task
Perform the following steps on your host machine:
1. On your host machine, open a terminal window.
2. Change to the /opt/ora/demo/UnifiedArchivesDemo directory.
# cd /opt/ora/demo/UnifiedArchivesDemo
# ls
UnifiedArchivesDemo.htm
UnifiedArchivesDemo.swf
standard.js
3. Open the UnifiedArchivesDemo.htm file in a web browser.

# firefox UnifiedArchivesDemo.htm &
A browser window with the Flash demonstration is displayed for you to walk through.
4. Close the web browser after you complete viewing the Flash demonstration.
5. Close the terminal window.

D74942GC40 Ag PDF

Caricato da

Informazioni sul documento

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

D74942GC40 Ag PDF

Caricato da

Copyright:

Formati disponibili

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY.

COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Edition 4.0 | February 2016 | D94670

Copyright © 2016, Oracle and/or its affiliates. All rights reserved.

Restricted Rights Notice

U.S. GOVERNMENT RIGHTS

Technical Contributors and Reviewers

This book was published using: Oracle Tutor

Copyright © 2016, Oracle and/or its affiliates. All rights reserved.

Oracle Solaris Cluster 4.x Administration Table of Contents

Oracle Solaris Cluster 4.x Administration Table of Contents

Practices for Lesson 1: Introduction

Copyright © 2016, Oracle and/or its affiliates. All rights reserved.

Practices for Lesson 1: Overview

Copyright © 2016, Oracle and/or its affiliates. All rights reserved.

Practices for Lesson 1: Introduction

Figure 1: Oracle VirtualBox Virtual Machines

The VirtualBox environment consists of the following VMs:

Name of the VM Description

Copyright © 2016, Oracle and/or its affiliates. All rights reserved.

Practices for Lesson 1: Introduction

Name of the VM Description

Logging In to the Practice Environment

Copyright © 2016, Oracle and/or its affiliates. All rights reserved.

Practices for Lesson 1: Introduction

For the Administrative Workstation VM:

Task: Becoming Familiar with Your Practice Environment (Example)

Copyright © 2016, Oracle and/or its affiliates. All rights reserved.

Practices for Lesson 1: Introduction

How to Start the VM?

How to Log In to the VM?

Copyright © 2016, Oracle and/or its affiliates. All rights reserved.

Practices for Lesson 1: Introduction

Copyright © 2016, Oracle and/or its affiliates. All rights reserved.

Practices for Lesson 1: Introduction

How to Power Off the VM?

Copyright © 2016, Oracle and/or its affiliates. All rights reserved.

Practices for Lesson 1: Introduction

Copyright © 2016, Oracle and/or its affiliates. All rights reserved.

Practices for Lesson 1: Introduction

Practices for Lesson 1: Introduction

Copyright © 2016, Oracle and/or its affiliates. All rights reserved.

Practices for Lesson 2

Copyright © 2016, Oracle and/or its affiliates. All rights reserved.

Copyright © 2016, Oracle and/or its affiliates. All rights reserved.

Practices for Lesson 3: Establishing Cluster Node Console Connectivity

Practice for Lesson 3: Overview

Copyright © 2016, Oracle and/or its affiliates. All rights reserved.

Practices for Lesson 3: Establishing Cluster Node Console Connectivity

Practice 3-1: Connecting to the Cluster Node Console

Task 1: Update Host Name Resolution

Copyright © 2016, Oracle and/or its affiliates. All rights reserved.

Practices for Lesson 3: Establishing Cluster Node Console Connectivity

# svcprop system/name-service/switch | grep config/host

Task 2: Install the pconsole Utility

DOWNLOAD PKGS FILES XFER (MB) SPEED

Practices for Lesson 3: Establishing Cluster Node Console Connectivity

Updating package cache 2/2

$ pfexec pkg install ha-cluster/system/manual

DOWNLOAD PKGS FILES XFER (MB) SPEED

$ pkg info ha-cluster/system/manual

Copyright © 2016, Oracle and/or its affiliates. All rights reserved.

Practices for Lesson 3: Establishing Cluster Node Console Connectivity