Sun Cluster 3.2 - Cheat Sheet http://www.datadisk.co.uk/html_docs/sun/sun_cluster_cs.

htm

Sun Cluster Cheat Sheet

This cheatsheet contains common commands and information for both Sun Cluster 3.1 and 3.2, there is some missing information and over time I hope to complete this i.e zones, NAS
devices, etc

Also both versions of Cluster have a text based GUI tool, so don't be afraid to use this, especially if the task is a simple one

scsetup (3.1)
clsetup (3.2)

Also all the commands in version 3.1 are available to version 3.2

Daemons and Processes

At the bottom of the installation guide I listed the daemons and processing running after a fresh install, now is the time explain what these processes do, I have managed to obtain
informtion on most of them but still looking for others.

Versions 3.1 and 3.2

This is used by cluster kernel threads to execute userland commands (such as the run_reserve and dofsck
commands). It is also used to run cluster commands remotely (like the cluster shutdown command).
clexecd This daemon registers with failfastd so that a failfast device driver will panic the kernel if this daemon is
killed and not restarted in 30 seconds.
This daemon provides access from userland management applications to the CCR. It is automatically restarted if it
cl_ccrad is stopped.
The cluster event daemon registers and forwards cluster events (such as nodes entering and leaving the cluster).
cl_eventd There is also a protocol whereby user applications can register themselves to receive cluster events.
The daemon is automatically respawned if it is killed.
cluster event log daemon logs cluster events into a binary log file. At the time of writing for this course, there
cl_eventlogd is no published interface to this log. It is automatically restarted if it is stopped.
This daemon is the failfast proxy server.The failfast daemon allows the kernel to panic if certain essential
failfastd daemons have failed
The resource group management daemon which manages the state of all cluster-unaware applications. A failfast
rgmd driver panics the kernel if this daemon is killed and not restarted in 30 seconds.
This is the fork-and-exec daemon, which handles requests from rgmd to spawn methods for specific data services. A
rpc.fed failfast driver panics the kernel if this daemon is killed and not restarted in 30 seconds.
This is the process monitoring facility. It is used as a general mechanism to initiate restarts and failure action
scripts for some cluster framework daemons (in Solaris 9 OS), and for most application daemons and application
rpc.pmfd fault monitors (in Solaris 9 and10 OS). A failfast driver panics the kernel if this daemon is stopped and not
restarted in 30 seconds.
Public managment network service daemon manages network status information received from the local IPMP daemon
pnmd running on each node and facilitates application failovers caused by complete public network failures on nodes. It
is automatically restarted if it is stopped.

Disk path monitoring daemon monitors the status of disk paths, so that they can be reported in the output of the
cldev status command. It is automatically restarted if it is stopped.
scdpmd Multi-threaded DPM daemon runs on each node. It is automatically started by an rc script when a node boots. It
monitors the availibility of logical path that is visiable through various multipath drivers (MPxIO), HDLM,
Powerpath, etc. Automatically restarted by rpc.pmfd if it dies.

Version 3.2 only

This daemon serves as a proxy whenever any quorum device activity requires execution of some command in userland
qd_userd i.e a NAS quorum device
cl_execd
ifconfig_proxy_serverd
rtreg_proxy_serverd
is a daemon for the public network management (PMN) module. It is started at boot time and starts the PMN service.
cl_pnmd It keeps track of the local host's IPMP state and facilities inter-node failover for all IPMP groups.
scprivipd This daemon provisions IP addresses on the clprivnet0 interface, on behalf of zones.
This daemon monitors the state of Solaris 10 non-global zones so that applications designed to failover between
sc_zonesd zones can react appropriately to zone booting failure
It is used for reconfiguring and plumbing the private IP address in a local zone after virtual cluster is created,
cznetd also see the cznetd.xml file.
This is the "fork and exec" daemin which handles requests from rgmd to spawn methods for specfic data services.
rpc.fed Failfast will hose the box if this is killed and not restarted in 30 seconds
scqdmd The quorum server daemon, this possibly use to be called "scqsd"
pnm mod serverd

File locations

Both Versions (3.1 and 3.2)

man pages /usr/cluster/man
/var/cluster/logs
log files /var/adm/messages
Configuration files (CCR, eventlog, etc) /etc/cluster/
Cluster and other commands /usr/cluser/lib/sc
Version 3.1 Only
sccheck logs /var/cluster/sccheck/report.<date>
Cluster infrastructure file /etc/cluster/ccr/infrastructure
Version 3.2 Only
sccheck logs /var/cluster/logs/cluster_check/remote.<date>
Cluster infrastructure file /etc/cluster/ccr/global/infrastructure
Command Log /var/cluster/logs/commandlog

SCSI Reservations

scsi2:
/usr/cluster/lib/sc/pgre -c pgre_inkeys -d /dev/did/rdsk/d4s2
Display reservation keys
scsi3:

1 of 5 8/25/2010 4:43 AM
Sun Cluster 3.2 - Cheat Sheet http://www.datadisk.co.uk/html_docs/sun/sun_cluster_cs.htm

/usr/cluster/lib/sc/scsi -c inkeys -d /dev/did/rdsk/d4s2

scsi2:
/usr/cluster/lib/sc/pgre -c pgre_inresv -d /dev/did/rdsk/d4s2
determine the device owner
scsi3:
/usr/cluster/lib/sc/scsi -c inresv -d /dev/did/rdsk/d4s2

Command shortcuts

In version 3.2 there are number of shortcut command names which I have detailed below, I have left the full command name in the rest of the document so it is obvious what we are
performing, all the commands are located in /usr/cluster/bin

shortcut
cldevice cldev
cldevicegroup cldg
clinterconnect clintr
clnasdevice clnas
clquorum clq
clresource clrs
clresourcegroup clrg
clreslogicalhostname clrslh
clresourcetype clrt
clressharedaddress clrssa

Shutting down and Booting a Cluster

3.1 3.2
cluster shutdown -g0 -y
##other nodes in cluster
scswitch -S -h <host>
shutdown -i5 -g0 -y
shutdown entire cluster
## Last remaining node
scshutdown -g0 -y

scswitch -S -h <host> clnode evacuate <node>

shutdown single node shutdown -i5 -g0 -y shutdown -i5 -g0 -y
reboot a node into non-cluster mode ok> boot -x ok> boot -x

Cluster information

3.1 3.2
cluster list -v
Cluster scstat -pv cluster show
cluster status
clnode list -v
Nodes scstat –n clnode show
clnode status
cldevice list
Devices scstat –D cldevice show
cldevice status
clquorum list -v
Quorum scstat –q clquorum show
clqorum status
clinterconnect show
Transport info scstat –W
clinterconnect status
clresource list -v
Resources scstat –g clresource show
clresource status

clresourcegroup list -v
scsat -g
Resource Groups scrgadm -pv
clresourcegroup show
clresourcegroup status

clresourcetype list -v
Resource Types clresourcetype list-props -v
clresourcetype show
IP Networking Multipathing scstat –i clnode status -m
Installation info (prints packages and version) scinstall –pv clnode show-rev -v

Cluster Configuration

3.1 3.2
Release cat /etc/cluster/release
Integrity check sccheck cluster check
Configure the cluster (add nodes, add data
scinstall
services, etc) scinstall
Cluster configuration utility (quorum, data
scsetup clsetup
sevices, resource groups, etc)
Rename cluster rename -c <cluster_name>
Set a property cluster set -p <name>=<value>
## List cluster commands
cluster list-cmds

## Display the name of the cluster

cluster list
List
## List the checks
cluster list-checks

## Detailed configuration
cluster show -t global
Status cluster status

2 of 5 8/25/2010 4:43 AM
Sun Cluster 3.2 - Cheat Sheet
Reset the cluster private network settings
Place the cluster into install mode
Add a node
Remove a node
Prevent new nodes from entering
Put a node into maintenance state
Get a node out of maintenance state
Node Configuration
Add a node to the cluster
Remove a node from the cluster
Evacuate a node from the cluster
Cleanup the cluster configuration (used after
removing nodes)
List nodes
Change a nodes property
Status of nodes
Admin Quorum Device
Quorum devices are nodes and disk devices, so the total quorum will be all nodes and devices added together. You can use the scsetup(3.1)/clsetup(3.2) interface to add/remove
quorum devices or use the below commands.
Adding a SCSI device to the quorum
Adding a NAS device to the quorum
Adding a Quorum Server
Removing a device to the quorum
Remove the last quorum device
List
Resetting quorum info
Bring a quorum device into maintenance mode
(3.2 known as enabled)
Bring a quorum device out of maintenance
mode (3.2 known as disabled)
Device Configuration
Check device
Remove all devices from node
Monitoring
Rename
Replicate
3 of 5
scconf –a –T node=&lthost><host>
scconf –r –T node=&lthost><host>
scconf –a –T node=.
scconf -c -q node=<node>,maintstate
Note: use the scstat -q command to verify
that the node is in maintenance mode, the
vote count should be zero for that node.
scconf -c -q node=<node>,reset
Note: use the scstat -q command to verify
that the node is in maintenance mode, the
vote count should be one for that node.
3.1
scswitch -S -h <node>
3.1
scconf –a –q globaldev=d11
Note: if you get the error message "uable to scrub device"
use scgdevs to add device to the global device namespace.
n/a
n/a
scconf –r –q globaldev=d11
## Evacuate all nodes
## Put cluster into maint mode
scconf –c –q installmode
## Remove the quorum device
scconf –r –q globaldev=d11
## Check the quorum devices
scstat –q
scconf –c –q reset
Note: this will bring all offline quorum devices online
## Obtain the device number
scdidadm –L
scconf –c –q globaldev=<device>,maintstate
scconf –c –q globaldev=<device><device>,reset
3.1
http://www.datadisk.co.uk/html_docs/sun/sun_cluster_cs.htm
cluster restore-netprops <cluster_name>
cluster set -p installmode=enabled
clnode add -c <clustername> -n <nodename> -e endpoint1,endpoint2 -e endpoing3,endpoint4
clnode remove
3.2
clnode add [-c <cluster>] [-n <sponsornode>] \
-e <endpoint> \
-e <endpoint>
<node>
## Make sure you are on the node you wish to remove
clnode remove
clnode evacuate <node>
clnode clear <node>
## Standard list
clnode list [+|<node>]
## Destailed list
clnode show [+|<node>]
clnode set -p <name>=<value> [+|<node>]
clnode status [+|<node>]
3.2
clquorum add [-t <type>] [-p <name>=<value>] [+|<devicename>]
clquorum add -t netapp_nas -p filer=<nasdevice>,lun_id=<IDnumdevice nasdevice>
clquorum add -t quorumserver -p qshost<IPaddress>,port=<portnumber> <quorumser
clquorum remove [-t <type>] [+|<devicename>]
## Place the cluster in install mode
cluster set -p installmode=enabled
## Remove the quorum device
clquorum remove <device>
## Verify the device has been removed
clquorum list -v
## Standard list
clquorum list -v [-t <type>] [-n <node>] [+|<devicename>]
## Detailed list
clquorum show [-t <type>] [-n <node>] [+|<devicename>]
## Status
clquorum status [-t <type>] [-n <node>] [+|<devicename>]
clquorum reset
clquorum enable [-t <type>] [+|<devicename>]
clquorum disable [-t <type>] [+|<devicename>]
3.2
cldevice check [-n <node>] [+]
cldevice clear [-n <node>]
## Turn on monitoring
cldevice monitor [-n <node>] [+|<device>]
## Turn off monitoring
cldevice unmonitor [-n <node>] [+|<device>]
cldevice rename -d <destination_device_name>
cldevice replicate [-S <source-node>] -D <destination-node> [+]
8/25/2010 4:43 AM
Sun Cluster 3.2 - Cheat Sheet http://www.datadisk.co.uk/html_docs/sun/sun_cluster_cs.htm

Set properties of a device cldevice set -p default_fencing={global|pathcount|scsi3} [-n <node>] <device>

## Standard display
cldevice status [-s <state>] [-n <node>] [+|<device>]
Status
## Display failed disk paths
cldevice status -s fail

## Standard List
cldevice list [-n <node>] [+|<device>]
Lists all the configured devices including paths
scdidadm –L
across all nodes. ## Detailed list
cldevice show [-n <node>] [+|<device>]
List all the configured devices including paths
scdidadm –l see above
on node only.
Reconfigure the device database, creating cldevice populate
scdidadm –r
new instances numbers if required. cldevice refresh [-n <node>] [+]
Perform the repair procedure for a particular scdidadm –R <c0t0d0s0> - device
cldevice repair [-n <node>] [+|<device>]
path (use then when a disk gets replaced) scdidadm –R 2 - device id

Disks group

3.1 3.2
Create a device group n/a cldevicegroup create -t vxvm -n <node-list> -p failback=tru
Remove a device group n/a cldevicegroup delete <devgrp>
Adding scconf -a -D type=vxvm,name=appdg,nodelist=<host>:<host>,preferenced=true cldevicegroup add-device -d <device> <devgrp>
Removing scconf –r –D name=<disk group> cldevicegroup remove-device -d <device> <devgrp>
Set a property cldevicegroup set [-p <name>=<value>] [+|<devgrp>]

## Standard list
cldevicegroup list [-n <node>] [-t <type>] [+|<devgrp>]
List scstat
## Detailed configuration report
cldevicegroup show [-n <node>] [-t <type>] [+|<devgrp>]

status scstat cldevicegroup status [-n <node>] [-t <type>] [+|<devgrp>]

adding single node scconf -a -D type=vxvm,name=appdg,nodelist=<host> cldevicegroup add-node [-n <node>] [-t <type>] [+|<devgrp>]
Removing single node scconf –r –D name=<disk group>,nodelist=<host> cldevicegroup remove-node [-n <node>] [-t <type>] [+|<devgr
Switch scswitch –z –D <disk group> -h <host> cldevicegroup switch -n <nodename> <devgrp>
Put into maintenance mode scswitch –m –D <disk group> n/a
take out of maintenance mode scswitch -z -D <disk group> -h <host> n/a
onlining a disk group scswitch -z -D <disk group> -h <host> cldevicegroup online <devgrp>
offlining a disk group scswitch -F -D <disk group> cldevicegroup offline <devgrp>
Resync a disk group scconf -c -D name=appdg,sync cldevicegroup syn [-t <type>] [+|<devgrp>]

Transport Cable

3.1 3.2
Add clinterconnect add <endpoint>,<endpoint>
Remove clinterconnect remove <endpoint>,<endpoint>
Enable scconf –c –m endpoint=<host>:qfe1,state=enabled clinterconnect enable [-n <node>] [+|<endpoint>,<endpoint>]
scconf –c –m endpoint=<host>:qfe1,state=disabled
Disable clinterconnect disable [-n <node>] [+|<endpoint>,<endpoint>]
Note: it gets deleted
## Standard and detailed list
List scstat
clinterconnect show [-n <node>][+|<endpoint>,<endpoint>]
Status scstat clinterconnect status [-n <node>][+|<endpoint>,<endpoint>]

Resource Groups

3.1 3.2

Adding (failover) scrgadm -a -g <res_group> -h <host>,<host> clresourcegroup create <res_group>

Adding (scalable) clresourcegroup create -S <res_group>

Adding a node to a resource group clresourcegroup add-node -n <node> <res_group>

## Remove a resource group

clresourcegroup delete <res_group>
Removing scrgadm –r –g <group>
## Remove a resource group and all its resources
clresourcegroup delete -F <res_group>

Removing a node from a resource group clresourcegroup remove-node -n <node> <res_group>

changing properties scrgadm -c -g <resource group> -y <propety=value> clresourcegroup set -p Failback=true + <name=value>
Status scstat -g clresourcegroup status [-n <node>][-r <resource][-s <state>][-t <resourcetype>
Listing scstat –g clresourcegroup list [-n <node>][-r <resource][-s <state>][-t <resourcetype>][
Detailed List scrgadm –pv –g <res_group> clresourcegroup show [-n <node>][-r <resource][-s <state>][-t <resourcetype>][
Display mode type (failover or scalable) scrgadm -pv -g <res_group> | grep 'Res Group mode'

## All resource groups

clresourcegroup offline +

Offlining scswitch –F –g <res_group> ## Individual group

clresourcegroup offline [-n <node>] <res_group>

clresourcegroup evacuate [+|-n <node>]

## All resource groups

clresourcegroup online +
Onlining scswitch -Z -g <res_group>
## Individual groups
clresourcegroup online [-n <node>] <res_group>

4 of 5 8/25/2010 4:43 AM
Sun Cluster 3.2 - Cheat Sheet
Evacuate all resource groups from a node
(used when shutting down a node)
Unmanaging
Managing
Switching
Suspend
Resume
Remaster (move the resource group/s to their
preferred node)
Restart a resource group (bring offline then
online)
Resources
Adding failover network resource
Adding shared network resource
adding a failover apache application and
attaching the network resource
adding a shared apache application and
attaching the network resource
Create a HAStoragePlus failover resource
Removing
changing or adding properties
List
Detailed List
Status
Disable resoure monitor
Enable resource monitor
Disabling
Enabling
Clearing a failed resource
Find the network of a resource
Removing a resource and resource group
Resource Types
Adding (register in 3.2)
Register a resource type to a node
Deleting (remove in 3.2)
Deregistering a resource type from a node
Listing
Listing resource type properties
Show resource types
Set properties of a resource type
5 of 5
http://www.datadisk.co.uk/html_docs/sun/sun_cluster_cs.htm
clresourcegroup evacuate [+|-n <node>]
scswitch –u –g <res_group>
clresourcegroup unmanage <res_group>
Note: (all resources in group must be disabled)
scswitch –o –g <res_group> clresourcegroup manage <res_group>
scswitch –z –g <res_group> –h <host> clresourcegroup switch -n <node> <res_group>
n/a clresourcegroup suspend [+|<res_group>]
n/a clresourcegroup resume [+|<res_group>]
n/a clresourcegroup remaster [+|<res_group>]
n/a clresourcegroup restart [-n <node>] [+|<res_group>]
3.1 3.2
scrgadm –a –L –g <res_group> -l <logicalhost> clreslogicalhostname create -g <res_group> <lh-resource>
scrgadm –a –S –g <res_group> -l <logicalhost> clressharedaddress create -t -g <res_group> <sa-resource>
scrgadm –a –j apache_res -g <res_group> \
-t SUNW.apache -y Network_resources_used = <logicalhost>
-y Scalable=False –y Port_list = 80/tcp \
-x Bin_dir = /usr/apache/bin
scrgadm –a –j apache_res -g <res_group> \
-t SUNW.apache -y Network_resources_used = <logicalhost>
-y Scalable=True –y Port_list = 80/tcp \
-x Bin_dir = /usr/apache/bin
scrgadm -a -g rg_oracle -j hasp_data01 -t SUNW.HAStoragePlus \ clresource create -t HAStorage -g <res_group> \
> -x FileSystemMountPoints=/oracle/data01 \ -p FilesystemMountPoints=<mount-point-list> \
> -x Affinityon=true -p Affinityon=true <rs-hasp>
scrgadm –r –j res-ip
clresource delete [-g <res_group>][-t <resourcetype>][+|<resource>]
Note: must disable the resource first
## Changing
clresource set -t <type> -p <name>=<value> +
scrgadm -c -j <resource> -y <property=value>
## Adding
clresource set -p <name>+=<value> <resource>
clresource list [-g <res_group>][-t <resourcetype>][+|<resource>]
scstat -g
## List properties
clresource list-props [-g <res_group>][-t <resourcetype>][+|<resour
scrgadm –pv –j res-ip
scrgadm –pvv –j res-ip
clresurce show [-n <node>] [-g <res_group>][-t <resourcetype>][+|<r
scstat -g clresource status [-s <state>][-n <node>] [-g <res_group>][-t <reso
scrgadm –n –M –j res-ip clresource monitor [-n <node>] [-g <res_group>][-t <resourcetype>][
scrgadm –e –M –j res-ip clresource unmonitor [-n <node>] [-g <res_group>][-t <resourcetype>
scswitch –n –j res-ip clresource disable <resource>
scswitch –e –j res-ip clresource enable <resource>
scswitch –c –h<host>,<host> -j <resource> -f STOP_FAILED clresource clear -f STOP_FAILED <resource>
scrgadm –pvv –j <resource> | grep –I network
## offline the group ## offline the group
scswitch –F –g rgroup-1 clresourcegroup offline <res_group>
## remove the resource ## remove the resource
scrgadm –r –j res-ip clresource [-g <res_group>][-t <resourcetype>][+|<resource>]
## remove the resource group ## remove the resource group
scrgadm –r –g rgroup-1 clresourcegroup delete <res_group>
3.1 3.2
scrgadm –a –t <resource type> i.e SUNW.HAStoragePlus clresourcetype register <type>
n/a clresourcetype add-node -n <node> <type>
scrgadm –r –t <resource type> clresourcetype unregister <type>
n/a clresourcetype remove-node -n <node> <type>
scrgadm –pv | grep ‘Res Type name’ clresourcetype list [<type>]
clresourcetype list-props [<type>]
clresourcetype show [<type>]
clresourcetype set [-p <name>=<value>] <type>
8/25/2010 4:43 AM