Sei sulla pagina 1di 412

VERITAS Volume Manager

Troubleshooting
IES-410

Student Guide

Sun Microsystems, Inc.


UBRM05-104
500 Eldorado Blvd.
Broomfield, CO 80021
U.S.A.
Revision B
January 13, 2003 5:15 pm
Copyright 2003 Sun Microsystems, Inc., 901 San Antonio Road, Palo Alto, California 94303, U.S.A. All rights reserved.

This product or document is protected by copyright and distributed under licenses restricting its use, copying, distribution, and
decompilation. No part of this product or document may be reproduced in any form by any means without prior written authorization of
Sun and its licensors, if any.

Third-party software, including font technology, is copyrighted and licensed from Sun suppliers.

Sun, Sun Microsystems, the Sun Logo, Solaris, StorEdge, Sun Enterprise, SunSolve, Sun Enterprise Network Array, JumpStart, OpenBoot,
Solstice, Sun BluePrints, and Solstice DiskSuite are trademarks or registered trademarks of Sun Microsystems, Inc. in the U.S. and other
countries.

All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. in the U.S. and
other countries. Products bearing SPARC trademarks are based upon an architecture developed by Sun Microsystems, Inc.

UNIX is a registered trademark in the U.S. and other countries, exclusively licensed through X/Open Company, Ltd.

U.S. Government approval might be required when exporting the product.

RESTRICTED RIGHTS: Use, duplication, or disclosure by the U.S. Government is subject to restrictions of FAR 52.227-14(g)(2)(6/87) and
FAR 52.227-19(6/87), or DFAR 252.227-7015 (b)(6/95) and DFAR 227.7202-3(a).

DOCUMENTATION IS PROVIDED “AS IS” AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS, AND
WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR
NON-INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY
INVALID.

THIS MANUAL IS DESIGNED TO SUPPORT AN INSTRUCTOR-LED TRAINING


(ILT) COURSE AND IS INTENDED TO BE USED FOR REFERENCE PURPOSES IN
CONJUNCTION WITH THE ILT COURSE. THE MANUAL IS NOT A STANDALONE
TRAINING TOOL. USE OF THE MANUAL FOR SELF-STUDY WITHOUT CLASS
ATTENDANCE IS NOT RECOMMENDED.
Export Control Classification Number (ECCN) assigned: 17 July 2002

Please
Recycle
Copyright 2003 Sun Microsystems Inc., 901 San Antonio Road, Palo Alto, California 94303, Etats-Unis. Tous droits réservés.

Ce produit ou document est protégé par un copyright et distribué avec des licences qui en restreignent l’utilisation, la copie, la distribution,
et la décompilation. Aucune partie de ce produit ou document ne peut être reproduite sous aucune forme, par quelque moyen que ce soit,
sans l’autorisation préalable et écrite de Sun et de ses bailleurs de licence, s’il y en a.

Le logiciel détenu par des tiers, et qui comprend la technologie relative aux polices de caractères, est protégé par un copyright et licencié
par des fournisseurs de Sun.

Sun, Sun Microsystems, the Sun Logo, Solaris, StorEdge, Sun Enterprise, SunSolve, Sun Enterprise Network Array, JumpStart, OpenBoot,
Solstice, Sun BluePrints, et Solstice DiskSuite sont des marques de fabrique ou des marques déposées de Sun Microsystems, Inc. aux Etats-
Unis et dans d’autres pays.

Toutes les marques SPARC sont utilisées sous licence sont des marques de fabrique ou des marques déposées de SPARC International, Inc.
aux Etats-Unis et dans d’autres pays. Les produits portant les marques SPARC sont basés sur une architecture développée par Sun
Microsystems, Inc.

UNIX est une marques déposée aux Etats-Unis et dans d’autres pays et licenciée exclusivement par X/Open Company, Ltd.

L’accord du gouvernement américain est requis avant l’exportation du produit.

LA DOCUMENTATION EST FOURNIE “EN L’ETAT” ET TOUTES AUTRES CONDITIONS, DECLARATIONS ET GARANTIES
EXPRESSES OU TACITES SONT FORMELLEMENT EXCLUES, DANS LA MESURE AUTORISEE PAR LA LOI APPLICABLE, Y
COMPRIS NOTAMMENT TOUTE GARANTIE IMPLICITE RELATIVE A LA QUALITE MARCHANDE, A L’APTITUDE A UNE
UTILISATION PARTICULIERE OU A L’ABSENCE DE CONTREFAÇON.

CE MANUEL DE RÉFÉRENCE DOIT ÊTRE UTILISÉ DANS LE CADRE D’UN COURS


DE FORMATION DIRIGÉ PAR UN INSTRUCTEUR (ILT). IL NE S’AGIT PAS D’UN
OUTIL DE FORMATION INDÉPENDANT. NOUS VOUS DÉCONSEILLONS DE
L’UTILISER DANS LE CADRE D’UNE AUTO-FORMATION.

Please
Recycle
Table of Contents
About This Course ........................................................... Preface--xiii
Course Goals...................................................................... Preface--xiii
Course Map......................................................................... Preface--xiv
Topics Not Covered.............................................................Preface--xv
How Prepared Are You?................................................... Preface--xvi
Introductions ..................................................................... Preface--xvii
How to Use Course Materials ........................................ Preface--xviii
Conventions .........................................................................Preface--xix
Icons .............................................................................Preface--xix
Typographical Conventions ......................................Preface--xx
Introducing the VERITAS Volume Manager Software
Architecture ......................................................................................1-1
Objectives ........................................................................................... 1-1
Relevance............................................................................................. 1-2
Additional Resources ........................................................................ 1-3
Introducing Storage Management................................................... 1-4
Host-Based Storage Management........................................... 1-4
Controller-Based Storage Management................................. 1-5
Comparison of Storage Management Methods.................... 1-6
Exploring VxVM Software and Storage Management ................. 1-7
Relationship to the Operating System Environment ........... 1-7
Configuration Database ........................................................... 1-8
Device Discovery Layer (DDL) ............................................... 1-9
Drivers and Daemons............................................................... 1-9
VxVM Software Support Files.............................................. 1-11
Examining VxVM Software Objects.............................................. 1-20
Physical Disks......................................................................... 1-21
VxVM Software Disks ............................................................ 1-21
Disk Groups ............................................................................. 1-23
Subdisks.................................................................................... 1-24
Plexes ........................................................................................ 1-25
Volumes.................................................................................... 1-26
VxVM Software Layered Volume Objects........................... 1-27

Sun Proprietary: Internal Use Only


v
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Resynchronizing Volumes.............................................................. 1-29
Dirty Flag................................................................................. 1-29
Resynchronization Process .................................................. 1-30
Introducing VxVM Software Logging .......................................... 1-31
RAID 5 Logs............................................................................. 1-31
Dirty Region Logs (DRLs)...................................................... 1-32
Examining Plex States ..................................................................... 1-34
Plex State Descriptions ........................................................... 1-34
Plex Kernel States.................................................................... 1-36
General Plex State Cycle ...................................................... 1-37
Other Plex State Transitions .................................................. 1-37
Introducing Supported VxVM Software Version 3.2
Features .......................................................................................... 1-40
Device Discovery .................................................................... 1-40
Device Naming........................................................................ 1-43
Sun StorEdge Traffic Manager Software Support.............. 1-46
Explicit LUN Failover............................................................. 1-46
Ordered Storage Allocation................................................... 1-46
Introducing Unsupported VxVM Software Version 3.2
Features .......................................................................................... 1-47
Disk Group Split, Move, and Join......................................... 1-47
Persistent Fast Resync Function............................................ 1-47
Encapsulating Disks........................................................................ 2-1
Objectives ........................................................................................... 2-1
Relevance............................................................................................. 2-2
Additional Resources ........................................................................ 2-3
Introducing Disk Encapsulation ...................................................... 2-4
Encapsulated Disk Types......................................................... 2-4
Reasons to Encapsulate Disks ................................................. 2-4
Encapsulating Data Disks ................................................................. 2-5
Pre-Encapsulation Configuration Data................................. 2-6
Data Disk Encapsulation Process ........................................... 2-8
Post-Encapsulation Configuration Data .............................. 2-13
Related Post-Encapsulation Files.......................................... 2-18
Encapsulating a Non-Conforming Disk .............................. 2-20
Unencapsulating Data Disks .......................................................... 2-23
Encapsulating Boot Disks ............................................................... 2-31
Booting root Volumes ........................................................... 2-32
Volume Restrictions................................................................ 2-33
Boot Disk Encapsulation Process.......................................... 2-33
Pre-Encapsulation Configuration Data............................... 2-35
Post-Encapsulation Configuration Data .............................. 2-38
Mirroring the Encapsulated Boot Disk ................................ 2-41
Related Post-Encapsulation Files.......................................... 2-45

Sun Proprietary: Internal Use Only


vi VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining Sun Enterprise Services Best Practices for VxVM
Software-Managed Boot Disks.................................................... 2-48
Best-Practice Boot Disk Configuration Guidelines ........... 2-49
Manually Bringing the Boot Disk Under VxVM
Software Management ........................................................... 2-50
Scripted Process Using the EIS CD-ROM............................ 2-56
Unencapsulating Boot Disks .......................................................... 2-57
Unencapsulating a Boot Disk Using the vxunroot
Utility ..................................................................................... 2-57
Manually Unencapsulating a Boot Disk .............................. 2-60
Unencapsulating When Booted From the CD-ROM ........ 2-64
Performing a Basic or Functional Unencapsulation .......... 2-68
Exploring Unencapsulation Issues ................................................ 2-72
Data Disks ................................................................................ 2-72
Boot Disks................................................................................. 2-72
Exercise: Encapsulating Disks........................................................ 2-75
Preparation............................................................................... 2-75
Task 1 – Installing the VxVM Software and
Encapsulating a Boot Disk ................................................. 2-76
Task 2 – Unencapsulating a Boot Disk Using
the vxunroot Utility ........................................................... 2-78
Task 3 – Encapsulating a Boot Disk Using
the vxdiskadm Utility.......................................................... 2-79
Task 4 – Manually Unencapsulating a Boot Disk
When Booted From the CD-ROM...................................... 2-79
Task 5 – Encapsulating a Data Disk Using
the vxdiskadm Utility (Optional) ...................................... 2-80
Task 6 – Unencapsulating a Data Disk (Optional) ............. 2-82
Task 7 – Encapsulating a Non-Conforming Data Disk
(Optional) .............................................................................. 2-83
Task 8 – Lab House Cleaning ................................................ 2-84
Exercise Summary............................................................................ 2-85
Exercise: Encapsulating Disks........................................................ 2-86
Task 1 Solutions....................................................................... 2-86
Task 2 Solutions...................................................................... 2-88
Task 3 Solutions...................................................................... 2-89
Task 4 Solutions....................................................................... 2-89
Task 5 Solutions....................................................................... 2-90
Task 6 Solutions...................................................................... 2-92
Task 7 Solutions...................................................................... 2-93
Managing Dynamic Multi-Pathing ...................................................3-1
Objectives ........................................................................................... 3-1
Relevance............................................................................................. 3-2
Additional Resources ........................................................................ 3-3
Examining VxVM Software and Dynamic Multi-Pathing ........... 3-4

Sun Proprietary: Internal Use Only


vii
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
VxVM Software Architecture and DMP................................ 3-4
Load Balancing .......................................................................... 3-5
Unique Disk Identifier............................................................. 3-6
DMP Device Paths .................................................................... 3-6
Solaris OE Drives and DMP Drives........................................ 3-6
DMP and Device Discovery .................................................... 3-8
Installing and Verifying DMP.......................................................... 3-9
Enabling and Disabling DMP......................................................... 3-10
Disabling DMP in Version 3.0 and Earlier .......................... 3-10
Disabling DMP in Version 3.1 and Later............................. 3-10
Files Related to Disabling and Suppression........................ 3-15
Administrating DMP With the vxdmpadm Command ................ 3-17
The listctrl Option............................................................. 3-17
Viewing Multi-Pathing Status............................................... 3-17
The getsubpaths Option ...................................................... 3-18
The getdmpnode Option ........................................................ 3-19
The disable Opt ion.............................................................. 3-19
The enable Option ................................................................. 3-20
The start restore and stop restore Options............. 3-20
Reviewing Common DMP Problems ............................................ 3-21
Disks Do Not Appear to Be Multi-Pathed........................... 3-21
Serial Number Problems........................................................ 3-21
The product_id and vendor_id Do Not Match .............. 3-22
VxVM Software Does Not See Disk Devices ...................... 3-22
Exercise: Operating DMP................................................................ 3-23
Preparation............................................................................... 3-23
Task 1 – Enabling and Disabling DMP Operations .......... 3-24
Task 2 – Administrating DMP .............................................. 3-25
Task 3 – Using the /etc/vx/diag.d/vxdmping Script .... 3-26
Exercise Summary............................................................................ 3-27
Exercise: Operating DMP................................................................ 3-28
Task 1 Solutions....................................................................... 3-28
Task 2 Solutions....................................................................... 3-29
Task 3 Solutions....................................................................... 3-30
Troubleshooting Tools and Utilities............................................... 4-1
Objectives ........................................................................................... 4-1
Relevance............................................................................................. 4-2
Additional Resources ........................................................................ 4-3
Logging Errors.................................................................................... 4-4
Using the /var/vxvm/vxconfigd.log File ......................... 4-4
Interpreting /var/adm/messages File syslog
Messages.................................................................................. 4-8
The root Mail............................................................................ 4-9
Recovering From Errors.................................................................. 4-10
Using the Debugging Tools and Utilities ..................................... 4-11

Sun Proprietary: Internal Use Only


viii VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
The vxexplorer Utility ......................................................... 4-18
The vxprivutil Utility ......................................................... 4-33
The vxdevwalk Utility............................................................ 4-38
The vxkprint Utility............................................................. 4-40
Using System-Level Debugging Utilities ..................................... 4-41
Exercise: Using the Error Logging and Debugging
Utilities............................................................................................ 4-42
Preparation............................................................................... 4-42
Task 1 – Enabling vxconfigd Debug Logging................... 4-42
Task 2 – Viewing the VxVM Software Messages ............... 4-43
Task 3 – Using VxVM Software Debug Utilities ................ 4-44
Exercise Summary............................................................................ 4-46
Exercise: Using the Error Logging and Debugging
Utilities............................................................................................ 4-47
Task 1 Solutions....................................................................... 4-47
Task 2 Solutions....................................................................... 4-47
Task 3 Solutions....................................................................... 4-49
Recovering Boot and System Processes.......................................5-1
Objectives ........................................................................................... 5-1
Relevance............................................................................................. 5-2
Additional Resources ........................................................................ 5-3
Surveying VxVM Software System Recovery Processes.............. 5-4
Examining the VxVM Software Boot Process ................................ 5-5
Single-User Boot Processing.................................................... 5-6
Multi-User Startup Files......................................................... 5-17
Other Boot Process Failures................................................... 5-18
Troubleshooting Boot Process Failures......................................... 5-20
Bootable Boot Disk.................................................................. 5-20
Valid /etc/system File ......................................................... 5-21
Valid /etc/vfstab File ......................................................... 5-22
Valid rootdg Disk Group...................................................... 5-23
Startable Volumes ................................................................... 5-23
Non-Corrupted and Appropriate Binaries and
Libraries................................................................................. 5-24
Valid /etc/vx/volboot File ............................................... 5-27
VxVM Software Startup Scripts ............................................ 5-27
Troubleshooting VxVM Software Failures................................... 5-29
Exercise: Determining the VxVM Software Problem ................. 5-30
Preparation............................................................................... 5-30
Tasks ........................................................................................ 5-32
Exercise Summary............................................................................ 5-35
Exercise: Determining the VxVM Software Problem ................. 5-36
Task Solutions.......................................................................... 5-36

Sun Proprietary: Internal Use Only


ix
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Recovering Disk, Disk Group, and Volume Failures .................... 6-1
Objectives ........................................................................................... 6-1
Relevance............................................................................................. 6-2
Additional Resources ........................................................................ 6-3
Introducing Recovery Processes ...................................................... 6-4
Identifying Disk Errors ..................................................................... 6-5
The vxprint Command........................................................... 6-5
The vxdisk Command............................................................. 6-6
The vxdiskadm list Option ................................................. 6-7
The vxstat Command............................................................. 6-7
Disk Failure Categories and root Mail ................................. 6-8
Full Disk Failure........................................................................ 6-8
Replacing Disks ................................................................................ 6-10
Replacing Disks Using the vxdiskadm Utility.................... 6-10
Replacing Disks Using the Command Line ........................ 6-10
Recovering a Disk Which VxVM Software Cannot See .... 6-11
Troubleshooting Volume Errors .................................................... 6-13
Listing Unstartable Volumes................................................. 6-13
Restarting a Disabled Volume.............................................. 6-14
Recovering a Mirrored Volume ............................................ 6-14
Recovering a RAID 5 Volume ............................................... 6-15
Forcibly Starting RAID 5 Volumes ....................................... 6-16
Troubleshooting Disk Group Errors ............................................. 6-17
Exercise: Determining the VxVM Software Disk Problem ........ 6-18
Preparation............................................................................... 6-18
Tasks ......................................................................................... 6-20
Exercise Summary............................................................................ 6-23
Exercise: Determining the VxVM Software Disk Problem ........ 6-24
Task Solutions.......................................................................... 6-24
Upgrading the VxVM Software........................................................ 7-1
Objectives ........................................................................................... 7-1
Relevance............................................................................................. 7-2
Additional Resources ........................................................................ 7-3
Surveying the Upgrade Processes and Procedures ...................... 7-4
Upgrading With a Script (VxVM 3.1/3.2/3.5)...................... 7-4
Upgrading Manually (VxVM 3.1/3.2/3.5)............................ 7-6
Upgrade Using pkgadd (VxVM 3.5)...................................... 7-9
Upgrading a Disk Group ....................................................... 7-11
Release Notes.......................................................................... 7-12
Licensing .................................................................................. 7-12
Upgrading the Solaris OE ............................................................... 7-13
Exercise: Upgrading the VxVM Software..................................... 7-14
Preparation............................................................................... 7-14
Tasks ......................................................................................... 7-14
Exercise Summary............................................................................ 7-16

Sun Proprietary: Internal Use Only


x VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
SunSolve INFODOCs....................................................................... A-1
INFODOC 16051 ............................................................................... A-2
INFODOC 24663 ............................................................................... A-4
Disk Encapsulation Processes ...................................................... B-1
Example Five-Slice Boot Disk Encapsulation............................... C-1
Pre-Encapsulation Configuration Data.......................................... C-3
Pre-Encapsulation prtvtoc Command................................ C-3
Pre-Encapsulation format Print Utility ........................... C-4
Pre-Encapsulation df -k Command .................................... C-4
Pre-Encapsulation swap -l Command................................ C-4
Post-Encapsulation Configuration Data ........................................ C-5
Post-Encapsulation prtvtoc /dev/rdsk/c1t2d0s2
Command............................................................................... C-5
Post-Encapsulation format print Utility.......................... C-6
Post-Encapsulation df -k Command................................... C-6
Post-Encapsulation vxprint Command .............................. C-6
Post-Encapsulation /etc/vfstab File.................................. C-7
Post-Encapsulation Mirror Disk prtvtoc Command ........ C-8
Post-Encapsulation Mirror Disk format Print
Utility ..................................................................................... C-9
Example Five-Slice Boot Disk Unencapsulation......................... C-10
Unencapsulating Using the vxunroot Command............ C-10
Manually Unencapsulating .................................................. C-10
The Boot Process ............................................................................ D-1
Configuring the VxVM Software..................................................... E-1
Objectives ........................................................................................... E-1
Relevance.............................................................................................E-2
Additional Resources ........................................................................E-3
Supported RAID Levels ....................................................................E-4
Layered Volumes ......................................................................E-4
Ordered Allocation of Storage ................................................E-4
VxVM Software States.......................................................................E-5
Plex States...................................................................................E-5
Plex and Volume Kernel States ...............................................E-5
Volume States ............................................................................E-5
Disk States ..................................................................................E-6
The vxprint Command....................................................................E-7
Output Header Information ....................................................E-7
Field Descriptions .................................................................... E-8
Stripe Width and Stripe Units ...............................................E-11
Complex RAID Levels.....................................................................E-12
Additional Layout Options ...................................................E-15
Additional Mirror Attributes ............................................... E-17
Disk Layout Practices ......................................................................E-18

Sun Proprietary: Internal Use Only


xi
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Guidelines for RAID Array Layouts ................................... E-19
Guidelines for Complex File System Layouts.....................E-19
Striping Considerations .........................................................E-20
Manipulating Disk Layouts............................................................E-22
Online Re-Layout ....................................................................E-22
Growing File Systems.............................................................E-23
Changing Disk Group Configurations..........................................E-27
Disk Group Reconfiguration Commands............................E-27
Disk Group Reconfiguration Recovery................................E-28
Reconfiguration Considerations ...........................................E-30
Hot-Relocation..................................................................................E-32
Hot-Relocation Process ..........................................................E-32
Hot-Relocation Configuration.............................................. E-36
Unrelocating ........................................................................... E-37
Hot-Spares.........................................................................................E-38
Comparison of Hot-Relocation and Hot-Spares.................E-38
Activating the Hot-Spare Function ..................................... E-39

Sun Proprietary: Internal Use Only


xii VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Preface

About This Course

Course Goals
The VERITAS Volume Manager Troubleshooting course introduces you to the
VERITAS Volume Manager (VxVM) software and its functions.

Upon completion of this course, you should be able to:


● Define information technology (IT) storage management
● Improve the availability of the VxVM software
● Identify and repair common disk and disk group management
problems
● Upgrade the VxVM software
● Resolve VxVM software version and licensing problems

Sun Proprietary: Internal Use Only


Preface-xiii
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Course Map

Course Map
The following course map enables you to see the general topics and the
modules for that topic area in reference to the course goal.

Architecture

The VERITAS
Volume Manager
Software Architecture

Availability Management

Encapsulating Managing Dynamic


Disks Multi-Pathing

Problem Management

Troubleshooting
Tools and Utilities

Recovering Recovering Disk,


Boot and System Disk Group and
Processes Volume Failures

Release Management

Upgrading
the VxVM
Software

Sun Proprietary: Internal Use Only


Preface-xiv VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Topics Not Covered

Topics Not Covered


This course does not cover the following topics. Many of these topics are
covered in other courses offered by Sun Educational Services:
● Storage area networks – Covered in ES-475: Design and Administration
of Storage Area Networks
● Solaris™ Operating Environment (OE) administration – Covered in
SA-288: Solaris 8 System Administration II
● Basic VxVM software administration – Covered in IES-310:
Sun StorEdge™ Volume Manager Administration
● Hitachi Lightning SE9900 administration – Covered in HDS-335:
Hitachi Lightning SE9900 Overview and Configuration
● Sun StorEdge™ T3 administration – Covered in ES-255: Sun
Hardware RAID and T3 Storage Systems Administration

Refer to the Sun Educational Services catalog for specific information and
registration.

Sun Proprietary: Internal Use Only


About This Course Preface-xv
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
How Prepared Are You?

How Prepared Are You?


To be sure you are prepared to take this course, can you answer yes to the
following questions?
● Can you install the VxVM software?
● Can you create the following VxVM software objects from the
command line and Storage Administrator graphical user interface
(GUI)?
● Disks
● Subdisks
● Plexes
● Volumes:
● Striped
● Simple
● RAID 5
● Can you use the vxprint utility to print VxVM software
configuration information?
● Can you mirror the boot disk on a server?
● Can you replace a failed disk using command line utilities and the
Storage Administrator GUI?

Sun Proprietary: Internal Use Only


Preface-xvi VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Introductions

Introductions
Now that you have been introduced to the course, introduce yourself to
the other students and the instructor, addressing the following items:
● Name
● Company affiliation
● Title, function, and job responsibility
● Experience related to topics presented in this course
● Reasons for enrolling in this course
● Expectations for this course

Sun Proprietary: Internal Use Only


About This Course Preface-xvii
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
How to Use Course Materials

How to Use Course Materials


To enable you to succeed in this course, these course materials employ a
learning model that is composed of the following components:
● Goals – You should be able to accomplish the goals after finishing
this course and meeting all of its objectives.
● Objectives – You should be able to accomplish the objectives after
completing a portion of instructional content. Objectives support
goals and can support other higher-level objectives.
● Lecture – The instructor will present information specific to the
objective of the module. This information will help you learn the
knowledge and skills necessary to succeed with the activities.
● Activities – The activities take on various forms, such as an exercise,
self-check, discussion, and demonstration. Activities are used to
facilitate mastery of an objective.
● Visual aids – The instructor might use several visual aids to convey a
concept, such as a process, in a visual form. Visual aids commonly
contain graphics, animation, and video.

Sun Proprietary: Internal Use Only


Preface-xviii VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Conventions

Conventions
The following icons and typographical conventions are used in this course
to represent various training elements and alternative learning resources.

Icons

Additional resources – Indicates additional reference materials are


available.

Discussion – Indicates a small-group or class discussion on the current


topic is recommended at this time.
!
?

Power user – Indicates additional supportive topics, ideas, or other


optional information.

Note – Indicates additional information that can help but is not crucial to
understanding of the concept being described. Examples of notational
information include keyword shortcuts and minor system adjustments.

Caution – Indicates that there is a risk of personal injury from a


nonelectrical hazard, or risk of irreversible damage to data, software, or
the operating system. A caution indicates that the possibility of a hazard
(as opposed to certainty) might happen, depending on the action of the
user.

Caution – Indicates that either personal injury or irreversible damage of


data, software, or the operating system will occur if the user performs this
action. A warning does not indicate potential events; if the action is
performed, catastrophic events will occur.

Sun Proprietary: Internal Use Only


About This Course Preface-xix
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Conventions

Caution – Indicates the risk of injury due to heat or hot surfaces will
result.

Typographical Conventions
Courier is used for the names of command, files, and directories, as well
as on-screen computer output. For example:
Use ls -al to list all files.
system% You have mail.

Courier bold is used for characters and numbers that you type. For
example:
system% su
Password:

Courier italic is used for variables and command-line place-holders


that are replaced with a real name or value. For example:
To delete a file, type the rm filename. command.

Courier italic bold is used to represent variables whose values are to


be entered by the student as part of an activity; for example:
Type chmod a+rwx filename to grant read, write, and execute
rights for filename to world, group, and users.

Palatino italics is used for book titles, new words or terms, or words that
are emphasized. For example:
Read Chapter 6 in the User’s Guide.
You must be root to do this.

Sun Proprietary: Internal Use Only


Preface-xx VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Module 1

Introducing the VERITAS Volume Manager


Software Architecture

Objectives
Upon completion of this module, you should be able to:
● Describe the two storage management methodologies
● Describe the relationship between the VxVM software and the
Solaris™ Operating Environment
● Identify and describe the major components of the VxVM software
configuration database
● Identify and define all the VxVM software objects
● Describe the resynchronization process
● Describe how the VxVM software identifies disks under control
● Describe the different plex states
● List newly supported and unsupported features introduced in the
VxVM software version 3.2

Sun Proprietary: Internal Use Only


1-1
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Relevance

Relevance

Discussion – The following questions are relevant to understanding how


the VxVM software internal architecture supports the management of
! enterprise storage subsystems:
?

● What VxVM software objects are used to build mountable volumes?


● What files are used to control the VxVM software internal
configuration, and where are they located?
● Where are the VxVM software driver and driver configuration files
stored?
● How does device discovery enable adding disks to enterprise storage
subsystems without a reboot of the system?
● What new features are included in the VxVM software version 3.2?

Sun Proprietary: Internal Use Only


1-2 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Additional Resources

Additional Resources

Additional resources – The following references provide additional


information on the topics described in this module:
● VERITAS Volume Manager™ 3.2 Administrator’s Guide. Mountain
View, California: VERITAS Software Corporation, August 2001,
number 30-000392-011, TechPDF ID 240253.
● VERITAS Volume Manager™ 3.2 Installation Guide. Mountain View,
California: VERITAS Software Corporation, August 2001,
number 30-000395-011, TechPDF ID 240256.
● VERITAS Volume Manager™ 3.2 Troubleshooting Guide. Mountain
View, California: VERITAS Software Corporation, August 2001,
number 30-000394-011, TechPDF ID 240255.
● VERITAS Volume Manager™ Storage Administrator 3.2 Administrator’s
Guide. Mountain View, California: VERITAS Software Corporation,
July 2001, number 30-000393-011, TechPDF ID 240257.

For additional information and resources, refer to:


● http://storage.east, http://storage.central, and
http://storage.west

Sun Proprietary: Internal Use Only


Introducing the VERITAS Volume Manager Software Architecture 1-3
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Introducing Storage Management

Introducing Storage Management


Storage provided by local or remote storage subsystems can be managed
using one of two methods. These methods are host-based and controller-
based storage management. The method used depends on the capabilities
of the storage subsystem. This section describes both storage management
methodologies.

Host-Based Storage Management


Storage subsystems such as the Sun StorEdge A5200 array are enclosure-
based systems known as just a bunch of disks (JBODs). These JBOD
storage enclosures must be managed by the VxVM software if additional
capacity and availability beyond single disks is desired. This is illustrated
in Figure 1-1.
Host System

Host Bus
Adapter

Device Drivers

9-GByte
VxVM Software Virtual
Layer Volume
Manager
Data

User
Process

JBOD Enclosure

Interface
Adapter Board

3-GByte
disk/slice/LUN

disk disk disk


disk disk disk
disk disk disk
disk disk disk

Figure 1-1 Host-Based Storage Management

Sun Proprietary: Internal Use Only


1-4 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Introducing Storage Management

JBOD disks also can be found in Sun’s midrange servers, such as the Sun
Enterprise™ 3500 and Sun Enterprise 250 servers. These disks are not
considered by the VxVM software to be enclosure-based JBODs because
they are configured differently. This difference is addressed later in this
module.

Controller-Based Storage Management


Storage subsystems such as the Sun StorEdge T3 array have intelligent
controllers which are capable of internally configuring logical units
(LUNs) for a redundant array of independent disks (RAID) layout. This is
illustrated in Figure 1-2.
Host System

Host Bus
Adapter

Data
Storage Subsystem
Manager
9-GByte
Controller LUN
RAID

User Manager

Process

Cache

RAID
Hardware

3-GByte
Disk/Slice disk disk disk

disk disk disk

disk disk disk

disk disk disk

Figure 1-2 Controller-Based Storage Management

Intelligent controller storage can be managed either by the storage


subsystem’s controller, the VxVM software, or both.

Sun Proprietary: Internal Use Only


Introducing the VERITAS Volume Manager Software Architecture 1-5
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Introducing Storage Management

Comparison of Storage Management Methods


The decision to use either host- or controller-based storage management
ultimately depends on the customer’s storage environment and the
processes and procedures used to manage that storage.

Host-based management has the advantage of providing a standard set of


utilities for configuring storage which is independent of the storage
provider. Host-based management does this at a slight cost to overall
system performance. Since the VxVM software executes in the Solaris OE,
it requires processor cycles to perform storage management operations.
This overhead, typically 1–2 percent, can be an issue for CPU cycle
constrained servers.

Controller-based storage management provides the best overall system


performance since no CPU cycles are used to manage a server’s storage.
This method does have operational issues if the customer has multiple
vendors providing storage. If a customer has storage subsystems from
different vendors installed in the storage environment, the customer’s
administrative staff must learn multiple RAID management applications,
and must track which storage is managed by which RAID management
application on different servers. These issues can lead to confusion and to
mistakes that cause storage availability issues.

If a storage environment contains both intelligent arrays and JBOD


enclosures, administrators must track which storage subsystem is
managed by a RAID manager or by the VxVM software. In this situation,
it can be advantageous to use intelligent arrays to configure small
individual LUNs, which are then used to build larger LUNS using the
VxVM software. A storage management strategy of this type eases
operational issues by turning storage into a black box. This means that
intelligent arrays are configured once using a RAID manager, and the
LUNs are then managed using the VxVM software along with the JBOD
enclosures.

Sun Proprietary: Internal Use Only


1-6 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exploring VxVM Software and Storage Management

Exploring VxVM Software and Storage Management


The VxVM software provides the Solaris OE with the ability to manage
local and remote storage subsystems. The word manage describes the
Solaris OE’s ability to use storage configured in such a fashion that the
Solaris OE capacity and availability is enhanced beyond single, non-
managed storage devices. The VxVM software is one of two products
available for the Solaris OE designed specifically for storage management.
The other product is the Sun Microsystems’ Solstice DiskSuite™ software.
Solstice DiskSuite software concepts and facilities are not described as
part of this course.

This section describes the VxVM software use for storage management.

Relationship to the Operating System Environment


The VxVM software operates as a storage management subsystem
between the Solaris OE and hosted data management subsystems, such as
a database manager or the Solaris OE file system.

The VxVM software storage management subsystem is implemented as a


set of drivers and daemons that are tightly coupled with the Solaris OE.
The VxVM software drivers implement virtual volumes from physical
storage resources that are mounted and accessed by users and
applications hosted by the Solaris OE. Figure 1-3 on page 1-8 displays this
relationship.

Sun Proprietary: Internal Use Only


Introducing the VERITAS Volume Manager Software Architecture 1-7
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exploring VxVM Software and Storage Management

Applications

Data Management Layer


VxVM
File System Database Configuration

Database

VxVM Storage Management Subsystem

DMP Driver

Storage Device Drivers

Kernel

Enclosure/Array vxconfigd

DDL

Figure 1-3 The VxVM Software and Solaris OE Relationship

Configuration Database
The VxVM software configuration database stores all disk and volume
configuration data. The following apply to the configuration database:
● Database access is managed through the /dev/vx/config device.
● Accesses are executed serially.
● Initial volume configurations are downloaded to the kernel through
this device.
● The vxconfigd daemon updates the database to reflect changes to
the configuration of VxVM software objects.

Sun Proprietary: Internal Use Only


1-8 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exploring VxVM Software and Storage Management

● The vxconfigd daemon is the exclusive owner of the


/dev/vx/config device.
● Non-volatile copies of the database are stored in the private region of
a VM disk as follows:
● Configuration copies are replicated within the disk group and
are available to prevent total loss of the database.
● The kernel configuration database is created from the disk
group copies during the system boot process.

Device Discovery Layer (DDL)


The device discovery layer (DDL) was introduced in the VxVM software
version 3.2. It is used to discover all disk devices connected to a system
and their attributes. DDL then stores this information in the dynamic
multi-pathing (DMP) configuration database for use by DMP operations.

Note – DDL concepts and administration are discussed in the section


‘‘Device Discovery’’ on page 1-40.

Drivers and Daemons


The VxVM software storage management subsystem is implemented by
the drivers and daemons described in this section.

Kernel Drivers

VxVM software storage management drivers include the following:


● vxio – The vxio driver manages access to VxVM software virtual
devices. Prior to initiating an input/output (I/O) operation to one of
these virtual devices, the vxio driver consults the VxVM
configuration database. The vxio driver is also responsible for
reporting device errors.
● vxdmp – The vxdmp daemon performs DMP operations on multi-
pathed storage subsystems.
● vxspec – The vxspec software control and status driver is used by
vxconfigd and other VxVM software utilities to communicate with
the Solaris OE kernel.

Sun Proprietary: Internal Use Only


Introducing the VERITAS Volume Manager Software Architecture 1-9
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exploring VxVM Software and Storage Management

VxVM Software Daemons

VxVM software storage management daemons include the following:


● vxconfigd – The vxconfigd software configuration daemon is
responsible for maintaining disk and disk group configuration
information. The vxconfigd daemon performs the following:
● It takes configuration change requests from VxVM software
utilities, communicates those to the kernel, and updates the
VxVM software configuration database.
● During system boot processing, vxconfigd reads the kernel
log to determine the current state of VxVM software objects and
any recovery operations to be performed.
● During disk group imports, vxconfigd scans the private
regions of the disk groups VM disks to find the most current
copy of its configuration database. The daemon then adds that
data to the system’s VxVM software kernel configuration
database.
● It receives cluster related information from the vxclust utility.
In a cluster environment, the different instances of vxconfigd
running on the cluster nodes communicate with each other
across the network.
● It logs any VxVM software object errors.

Note – The vxconfigd logging and error messages are discussed in detail
in Module 4, “Troubleshooting Tools and Utilities.”

● vxrelocd – The vxrelocd daemon performs hot-relocation to


restore redundancy. The vxrelocd daemon performs the following:
● Data located on subdisks that are part of a failed VM disk is
relocated to spare disks that have sufficient free space
configured in the disk group.
● When a relocation operation begins, vxrelocd sends mail to the
local root account.

Sun Proprietary: Internal Use Only


1-10 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exploring VxVM Software and Storage Management

VxVM Software Support Files


The files and directories described in this section are installed by the
vxinstall process to support the VxVM software storage management
operations. This discussion is organized by directory, as follows:
● /kernel/drv
● /kernel/drv/sparcv9
● /sbin
● /usr/sbin
● /etc/init.d
● /etc/rc*.d
● /opt
● /etc/vx
● /var/vxvm
● /dev/vx

The /kernel/drv Directory

The /kernel/drv directory contains the driver binaries and


configuration files for the 32-bit versions of vxiod, vxspec, and vxdmp.

The strategy Sun StorEdge employs to account for different VxVM


software versions of the Solaris OE is important. The driver binary for
vxio is the /kernel/drv/vxio file. There are three additional vxio driver
files, one each for Solaris OE versions 5.6, 5.7 and 5.8. During the
installation process the installer is prompted for the version of Solaris OE
in which this instance of the VxVM software is being installed. The
answer determines which driver version is copied, and that version
becomes the becomes /kernel/drv/vxio file. Notice that the byte count
for /kernel/dev/vxio is identical to /kernel/drv/vxio.SunOS_5.8 as
shown in the following example:
640 -rw-r--r-- 1 root sys 314156 Nov 21 23:03 ./vxdmp
4 -rw-r--r-- 1 root sys 1026 Aug 15 2001 ./vxdmp.conf
608 -rw-r--r-- 1 root sys 297920 Aug 15 2001 ./vxdmp.SunOS_5.6
608 -rw-r--r-- 1 root sys 300980 Aug 15 2001 ./vxdmp.SunOS_5.7
640 -rw-r--r-- 1 root sys 314156 Nov 21 23:03 ./vxdmp.SunOS_5.8
3776 -rw-r--r-- 1 root sys 1920316 Nov 21 23:03 ./vxio
2 -rw-r--r-- 1 root sys 991 Aug 15 2001 ./vxio.conf

Sun Proprietary: Internal Use Only


Introducing the VERITAS Volume Manager Software Architecture 1-11
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exploring VxVM Software and Storage Management

3664 -rw-r--r-- 1 root sys 1860500 Aug 15 2001 ./vxio.SunOS_5.6


3696 -rw-r--r-- 1 root sys 1878432 Aug 15 2001 ./vxio.SunOS_5.7
3776 -rw-r--r-- 1 root sys 1920316 Nov 21 23:03 ./vxio.SunOS_5.8
30 -rw-r--r-- 1 root other 14968 Mar 29 15:41 ./vxspec
4 -rw-r--r-- 1 root sys 1315 Aug 15 2001 ./vxspec.conf
28 -rw-r--r-- 1 root sys 14060 Aug 15 2001 ./vxspec.SunOS_5.6
30 -rw-r--r-- 1 root sys 14388 Aug 15 2001 ./vxspec.SunOS_5.7
30 -rw-r--r-- 1 root sys 14968 Aug 15 2001 ./vxspec.SunOS_5.8
26 -rwxr-xr-x 1 root sys 12384 Jan 8 2000 ./wc
2 -rw-r--r-- 1 root sys 129 Jan 5 2000 ./wc.conf
38 -rwxr-xr-x 1 root sys 18544 Jan 8 2000 ./xbox

If the VxVM software driver binaries are corrupted, copy the proper
architecture version of the binary into the corrupted driver binary file to
correct the problem. This process is presented in Module 5, “Recovering
Boot and System Processes.”

The /kernel/drv/sparcv9 Directory

The /kernel/drv/sparcv9 directory contains the driver binaries for the


64-bit versions of vxio, vxspec, and vxdmp. Notice in this example list
output that the configuration files are not present. When installed in 64-bit
mode (the default for Solaris 8 OE and later), the drivers use the
configuration files ™
800 -rw-r--r-- 1 root sys 393968 Nov 21 23:03 vxdmp
768 -rw-r--r-- 1 root sys 380840 Aug 15 2001 vxdmp.SunOS_5.7
800 -rw-r--r-- 1 root sys 393968 Nov 21 23:03 vxdmp.SunOS_5.8
5328 -rw-r--r-- 1 root sys 2714688 Nov 21 23:03 vxio
5232 -rw-r--r-- 1 root sys 2666256 Aug 15 2001 vxio.SunOS_5.7
5328 -rw-r--r-- 1 root sys 2714688 Nov 21 23:03 vxio.SunOS_5.8
38 -rw-r--r-- 1 root other 18504 Mar 29 15:41 vxspec
36 -rw-r--r-- 1 root sys 17928 Aug 15 2001 vxspec.SunOS_5.7
38 -rw-r--r-- 1 root sys 18504 Aug 15 2001 vxspec.SunOS_5.8

The /sbin Directory

The /sbin directory holds the binaries for vxconfigd and additional
VxVM software commands. Notice the configuration files in this example
list output.
2960 -r-xr-xr-x 1 root sys 1499404 Nov 21 23:03 vxconfigd
2800 -r-xr-xr-x 1 root sys 1421748 Aug 15 2001 vxconfigd.SunOS_5.6
2832 -r-xr-xr-x 1 root sys 1439184 Aug 15 2001 vxconfigd.SunOS_5.7
2960 -r-xr-xr-x 1 root sys 1499404 Nov 21 23:03 vxconfigd.SunOS_5.8
864 -r-xr-xr-x 1 root sys 432736 Nov 21 17:18 vxdg
256 -r-xr-xr-x 1 root sys 116100 Nov 21 17:18 vxdmpadm

Sun Proprietary: Internal Use Only


1-12 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exploring VxVM Software and Storage Management

400 -r-xr-xr-x 1 root bin 195272 Aug 15 2001 vxliccheck


400 -r-xr-xr-x 1 root bin 189800 Aug 15 2001 vxlicense

Note – The vxconfigd file has Solaris OE architectural requirements


similar to the driver binaries in the /kernel/drv directory.

The /usr/sbin Directory

The /usr/sbin directory contains all other VxVM software commands


and utilities, as shown in Table 1-1.

Table 1-1 VxVM Software Commands and Utilities


vxassist vxclust vxconfigd
vxdco vxdctl vxddladm
vxdg vxdisk vxdiskadd
vxdiskadm vxdiskconfig vxdmpadm
vxedit vxibc vxinfo
vxinstall vxiod vxliccheck
vxlicense vxmake vxmend
vxnetd vxnotify vxplex
vxprint vxrecover vxrecover.wrap
vxrelayout vxrlink vxrvg
vxsd vxspcshow vxstart_vvr
vxstat vxtask vxtrace
vxvol

The /etc/init.d Directory

The /etc/init.d directory contains the following VxVM software


startup scripts:
● vmsa-server
● volmgt
● vxnm-host_infod

Sun Proprietary: Internal Use Only


Introducing the VERITAS Volume Manager Software Architecture 1-13
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exploring VxVM Software and Storage Management

● vxnm-vxnetd
● vxvm-reconfig
● vxvm-recover
● vxvm-shutdown
● vxvm-startup1
● vxvm-startup2
● vxvm-sysboot

The /etc/rc*.d Directories

The /etc/rc*.d directories contain the hard links to the VxVM software
startup scripts in /etc/init.d, as follows:
● The /etc/rc0.d directory contains the following links:
● K10vmsa-server
● K99vxvm-shutdown
● The /etc/rc1.d directory contains the K10vmsa-server link.
● The /etc/rcS.d directory contains the following links:
● S35vxvm-startup1
● S85vxvm-startup2
● S86vxvm-reconfig
● The /etc/rc2.d directory contains the following links:
● S94vxnm-host_infod
● S94vxnm-vxnetd
● S95vxvm-recover
● S96vmsa-server

The VxVM software startup sequence during a system boot is covered in


Module 5, “Recovering Boot and System Processes.”

Sun Proprietary: Internal Use Only


1-14 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exploring VxVM Software and Storage Management

The /opt Directory

The /opt directory contains the VxVM software man pages, documents,
the Storage Administrator GUI binaries, and licensing utilities. The /opt
directory contains the following:
VRTS:
man
VRTS/man:
man1m man3x man4 man7
VRTSlic:
bin
VRTSlic/bin:
vxliccheck vxlicense
VRTSvxvm:
docs
VRTSvxvm/docs:
admin.pdf hwnotes.pdf igbook.pdf tshoot.pdf vmsaguide.pdf
admin.ps hwnotes.ps igbook.ps tshoot.ps vmsaguide.ps
VRTSspt:
FS README.VRTSspt VRTSexplorer VVR
VRTSspt/FS:
MetaSave VxBench
VRTSspt/VRTSexplorer:
README fw salr vcs vrtsisp
VRTSexplorer gcm samba vfr vsap
bin.SunOS isis spc visnC vvr
dbed lib spcs visnS vxfs
dbed1 main.SunOS spnas vmsa vxld
edition.dro ndmp sybed1 vnas vxvm
edition.sybed sal txpt vras
VRTSvmsa:
bin jre vmsa
VRTSvmsa/bin:
autostart gen_params pkg_params server_params vmsa vmsa_server
VRTSvmsa/jre:
bin COPYRIGHT jre_config.txt lib LICENSE.ps
VRTSvmsa/vmsa:
java os_properties properties server

Sun Proprietary: Internal Use Only


Introducing the VERITAS Volume Manager Software Architecture 1-15
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exploring VxVM Software and Storage Management

The /etc/vx Directory

The /etc/vx directory holds hundreds of files and subdirectories that are
used to hold key information about the VxVM software and the server
configuration on which it is installed.

Note – Due to the large number of files, this section lists those files and
directories of most importance to the recovery of the VxVM software. A
full list of files is available in Appendix A.

Some of the files and directories in the /etc/vx directory are:


● The /etc/vx/reconfig.d directory holds the files and
subdirectories, shown in the following list output, that are used to
define the VxVM software’s present and prior configurations.
./vx/reconfig.d:
total 18
2 drwxr-xr-x 6 root sys 512 Mar 29 16:06 .
2 drwxr-xr-x 8 root other 512 Mar 29 16:08 ..
2 drwxr-xr-x 3 root other 512 Mar 29 16:01 disk.d
2 -rw-r--r-- 1 root other 18 Mar 29 16:01 enclrs
2 drwxr-xr-x 2 root sys 512 Mar 29 15:41 org.d
2 -rw-r--r-- 1 root other 7 Mar 29 16:02 OTHER_DISKS
2 drwxr-xr-x 3 root root 512 Mar 29 17:15 saveconf.d
2 -rw-r--r-- 1 root other 54 Mar 29 16:02 SENA0
2 drwxr-xr-x 2 root sys 512 Mar 29 17:16 state.d
There are three files in this directory that describe disks under VxVM
software control:
● enclrs – This file lists the files in the directory that identify
disks detected by the VxVM software. An example is shown
here.
::::::::::::::
enclrs
::::::::::::::
OTHER_DISKS
SENA0
● OTHER_DISKS – This file lists all non-enclosure disks, as seen in
this example.
::::::::::::::
OTHER_DISKS
::::::::::::::
c0t1d0

Sun Proprietary: Internal Use Only


1-16 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exploring VxVM Software and Storage Management

● SENA0 – This file lists all disks in the enclosure named SENA0
detected by the VxVM software device discovery. An example
of the SENA0 file is shown here.
::::::::::::::
SENA0
::::::::::::::
c2t25d0
c2t16d0
c2t26d0
c2t10d0
c2t5d0
c2t0d0
c2t19d0
● The /etc/vx/reconfig.d/disk/d directory lists subdirectories that
hold pre- and post-encapsulation information for encapsulated disks,
as seen in these list outputs.
./vx/reconfig.d/disk.d:
total 6
2 drwxr-xr-x 3 root other 512 Mar 29 16:01 .
2 drwxr-xr-x 6 root sys 512 Mar 29 16:06 ..
2 drwxr-xr-x 2 root other 512 Mar 29 16:01 c0t0d0

./vx/reconfig.d/disk.d/c0t0d0:
total 12
2 drwxr-xr-x 2 root other 512 Mar 29 16:01 .
2 drwxr-xr-x 3 root other 512 Mar 29 16:01 ..
2 -rw-r--r-- 1 root other 9 Mar 29 16:01 dmname
2 -rw-r--r-- 1 root other 933 Mar 29 16:01 newpart
2 -rw-r--r-- 1 root other 7 Mar 29 16:01 primary_node
2 -rw-r--r-- 1 root other 452 Mar 29 16:01 vtoc
● The ./c0t0d0 directory contains files that describe present and prior
configuration information about the encapsulated boot disk. The file
named vtoc holds the boot disk’s pre-encapsulation vtoc and can be
used to recover the original configuration for the boot disk if
unencapsulation fails.

Note – The procedure for using the vtoc file to unencapsulate the boot
disk is addressed in Module 2, “Encapsulating Disks.”

Sun Proprietary: Internal Use Only


Introducing the VERITAS Volume Manager Software Architecture 1-17
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exploring VxVM Software and Storage Management

● The /etc/vx/reconfig.d/saveconf.d directory has a single


subdirectory, ./etc, that contains dump device information as well
as a current copy of the /etc/system file. These directories are
shown in list output.
./vx/reconfig.d/saveconf.d:
total 6
2 drwxr-xr-x 3 root root 512 Mar 29 17:15 .
2 drwxr-xr-x 6 root sys 512 Mar 29 16:06 ..
2 drwxr-xr-x 2 root root 512 Mar 29 16:08 etc

./vx/reconfig.d/saveconf.d/etc:
total 14
2 drwxr-xr-x 2 root root 512 Mar 29 16:08 .
2 drwxr-xr-x 3 root root 512 Mar 29 17:15 ..
2 -rw-r--r-- 1 root root 239 Mar 29 16:08 dumpadm.conf.new
2 -rw-r--r-- 1 root root 239 Mar 29 16:08 dumpadm.conf.orig
2 -rw-r--r-- 1 root root 140 Mar 29 16:08 dumpadm.out.orig
4 -rw-r--r-- 1 root other 2002 Mar 29 15:41 system
● The /etc/vx directory also holds a file called /etc/vx/volboot.
This is the VxVM software bootstrap file. It is an ASCII file that
adheres to a very strict format and should not be edited. This file has
the following characteristics:
● It is 512 bytes in length, including padding.
● Update this file using the vxdctl command.
● The volboot file holds the VxVM software host identifier
hostid. This is usually the Solaris OE node name, not the
hardware hostid. Keep in mind the following concepts:
● The VxVM software hostid does not have to match the
server’s node name, which can cause confusion.
● The hostid is used to establish disk and disk group
ownership.
● If two or more servers can access the same disks using the
same bus, the VxVM software hostid ensures that the two
hosts do not interfere with each other when accessing the
VxVM software disks.
● The volboot file can also contain a list of simple disks for
rootdg. Refer to the ‘‘VxVM Software Disks’’ on page 1-21 for
more information.

Sun Proprietary: Internal Use Only


1-18 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exploring VxVM Software and Storage Management

The /var/vxvm Directory

The /var/vxvm directory contains a subdirectory called tempdb that holds


copies of imported disk group configuration databases. There is a file for
each disk group, as shown in this list output.
/var/vxvm
total 6
2 drwxr-xr-x 3 root sys 512 Mar 29 17:15 .
2 drwxr-xr-x 31 root sys 512 Mar 29 15:41 ..
2 drwxr-xr-x 2 root root 512 Mar 29 17:15 tempdb

/var/vxvm/tempdb
total 34
2 drwxr-xr-x 2 root root 512 Mar 29 17:15 .
2 drwxr-xr-x 3 root sys 512 Mar 29 17:15 ..
18 -rw-r--r-- 1 root root 8704 Mar 29 17:15 DGa
6 -rw-r--r-- 1 root root 2560 Mar 29 17:15 DGb
6 -rw-r--r-- 1 root root 3072 Mar 29 17:15 rootdg

Caution – This directory must be present at boot time or the VxVM


software does not start. If the boot disk is under VxVM control, the
system does not boot. Do not remove this directory.

The /dev/vx Directory

The /dev/vx directory contains the logical volume device files that are
used to access the VxVM software objects. These files are shown here in a
list output.
/dev/vx
clust dmp dsk iod rdmp task trace
config dmpconfig info netiod rdsk taskmon

Further explanation of these device files is covered later in the


‘‘Examining VxVM Software Objects’’ on page 1-20.

Sun Proprietary: Internal Use Only


Introducing the VERITAS Volume Manager Software Architecture 1-19
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining VxVM Software Objects

Examining VxVM Software Objects


The VxVM software manipulates physical and virtual devices which
enable the VxVM software storage management operations, as follows:
● Physical objects – Physical objects are physical storage devices that
present raw or block device interfaces to the Solaris OE.
● Virtual objects – The VxVM software builds virtual objects from
physical storage objects that are brought under the VxVM software
control. The VxVM software virtual objects are combined into
volumes which are then formatted and mounted for use by
applications and users.
All objects that are not physical objects are virtual objects.

Figure 1-4 illustrates the relationship between physical and virtual VxVM
software objects.

Volume
Physical Objects Virtual Objects
Plex Plex

Plex Plex
Subdisk Subdisk

Subdisk Subdisk
Disk Group
Physical Disks
VxVM
Software Subdisk
Private

Public

VxVM
Software Subdisk
Private

Public

Figure 1-4 VxVM Software Objects

Sun Proprietary: Internal Use Only


1-20 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining VxVM Software Objects

Physical Disks
Physical disks are storage devices where data is ultimately stored.
Physical disks, or physical objects, are identified by the Solaris OE using a
unique identifier called a ctd number. Valid ctd identifiers are:
● c – The system controller or host bus adapter number
● t – A Small Computer System Interface (SCSI) target identifier
● d – A device or logical unit number
● s – A slice or partition

The VxVM software uses a drive ctd number for identification of the
physical device when it is brought under VxVM software control.

VxVM Software Disks


When a physical disk is brought under VxVM software control, it is either
initialized or encapsulated.

Initialized Disks

Initialized disks are reformatted with either one or two partitions, and all
data is destroyed. The partitions are used to store the VxVM software
configuration and data areas called private and public regions.

The private region is a small partition where disk group configuration


information is stored. The private region has the following characteristics:
● It is usually slice 3.
● Region size starts at 1024 sectors in early versions of the VxVM
software, and 2048 sectors in version 3.2.
Use the vxdisksetup command privlen option to expand the size
of the private region. This is a complicated procedure and is not
recommended. The procedure for expanding the private region is
addressed in a lab exercise.
● It is assigned vtoc tag number 15 for identification purposes.
● This region is not used for data storage.

Sun Proprietary: Internal Use Only


Introducing the VERITAS Volume Manager Software Architecture 1-21
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining VxVM Software Objects

● The private region contains the following information:


● Disk name and indentifier
● Disk group name and identifier
● Disk group configuration copy

The public region uses the remaining space available on the physical disk
to store subdisks. The public region has the following characteristics:
● It is usually slice 4.
● It is used for data storage.
● The region is maintained by the VxVM software commands.
● It is assigned vtoc tag number 14 for identification purposes.

Initialized disks can be configured in one of three methods:


● The sliced configuration – Public and private regions are defined as
separate Solaris OE partitions. This is the preferred method for
initializing a VxVM software disk.
● The simple configuration – Public and private regions are defined as
a single Solaris OE partition.
These are volatile devices and disappear after a reboot if they are not
in use or defined in the /etc/vx/volboot file.
● The nopriv configuration – The disk is configured without a private
region. Configuration information is held and maintained by a sliced
disk acting as a proxy.
These disks are not automatically discovered at bootup unless
defined in the /etc/vx/volboot file.

Note – The nopriv configuration is not supported in future releases.


Avoid using this configuration method.

Sun Proprietary: Internal Use Only


1-22 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining VxVM Software Objects

Encapsulated Disks

Encapsulation brings a physical disk under the VxVM software control


and preserves the data. The /etc/vfstab file is modified to reflect the
new volume names on the disk’s file systems. Encapsulation has the
following characteristics:
● The /etc/vfstab file requires free space on the disk, usually in the
private region, to store configuration information.
● The file size starts at 1024 sectors to the VxVM software version 3.2,
and 2048 sectors in version 3.2 and later.
● Encapsulation fails if there is not enough free space to build a private
region. If space is not available for a private region, use the nopriv
option.
● All partitions on the disk are reassigned to a new public region,
which is usually slice 6.
● Boot disk can be encapsulated and remain bootable.

Note – Avoid nopriv configurations. Support for this configuration is


being phased out.

Encapsulation, including boot disk encapsulation, is covered later in this


course.

Disk Groups
A named collection of VxVM software disks that share a common
configuration is called a disk group. Common configuration refers to a set of
records that provide detailed information about related VxVM software
objects, their connections and attributes. This configuration information is
stored in the private region of the VxVM software disks. A backup copy
for each configured disk group is stored in /var/vxvm/tempdb.

Disk groups are virtual objects and have the following characteristics:
● The default disk group is rootdg.
● Additional disk groups can be created on the fly.
● Disk group names are a maximum of 31 characters in length.
● Disk groups can be renamed.
● Disk groups are versioned.

Sun Proprietary: Internal Use Only


Introducing the VERITAS Volume Manager Software Architecture 1-23
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining VxVM Software Objects

● Disk groups allow grouping of the VxVM software disks into logical
collections.
● Disk groups can be moved from one system to another with an
import and deport process.
● Volumes created within a specific disk group can use only the VxVM
software disks that are a member of that disk group.
● Volumes and disks can be moved among disk groups.

Note – Moving volumes and disks among disk groups using an early
version of the VxVM software was a risky procedure. The VxVM software
version 3.2 has new options for the vxdg command to move the VxVM
software objects among disk groups and to split and join disk groups.
These new options require a special license.

Subdisks
Subdisks are contiguous blocks of space. Subdisks provide the basic
building blocks for the VxVM software plexes and volumes, creating the
VxVM software basic unit of storage allocation. Subdisks are virtual
objects.

The following characteristics apply to subdisks:


● Subdisk storage is allocated from the public region of a VxVM
software disk.
● A VxVM software disk can be subdivided into one or more subdisks.
Multiple subdisks cannot overlap.
● Space on a VxVM software disk not allocated to a subdisk is
considered free space.

Subdisk names are based on the VxVM software disk name where they
reside, appended with an incremental numeric identifier. Figure 1-5 on
page 1-25 illustrates how subdisk names are derived.

Sun Proprietary: Internal Use Only


1-24 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining VxVM Software Objects

Physical Disk VxVM Software Disk Subdisks

Private Region

Public Region disk01-01

disk01-01

/dev/rdsk/c2t3d5 disk01-02 disk01-02

disk01-03

Free Space disk01-03

disk01

Figure 1-5 Subdisk Naming Scheme

Plexes
The VxVM software virtual objects built from subdisks are called plexes. A
plex consists of one or more subdisks located on one or more VxVM
software disks.

Plexes are:
● Also known as submirrors
● Mid-tier building blocks of the VxVM software volumes
● Named based on the name of the volume for which it is a submirror,
plus an appended incremental numeric identifier
● Organized using the following methods:
● Concetanation
● Stripe (RAID 0)
● Mirror (RAID 1)
● Striping with parity (RAID 5)

Sun Proprietary: Internal Use Only


Introducing the VERITAS Volume Manager Software Architecture 1-25
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining VxVM Software Objects

Figure 1-6 illustrates the architectural components of a plex.

VxVM Software Disk Plex

Private Region

Public Region

disk01-01 disk01-01

disk01-02 disk01-02 Subdisks

disk01-03 disk01-03
This is the first
vol01-01 plex of volume
Free Space vol01.

disk01
Volume name
This plex is a
sub-mirror.

Figure 1-6 Plex Components

Note – Plex states are covered in detail in the section ‘‘Resynchronizing


Volumes’’ on page 1-29. Additional plex state information is found in
Appendix E, “Configuring the VxVM Software.”

Volumes
Volumes are virtual devices (virtual objects) that appear to be physical
disks to applications, databases and file systems. Volumes are the VxVM
software’s top-tier virtual objects. Although volumes appear to be
physical disks, they do not share the limitations of physical disks.

Volumes have the following characteristics:


● Volumes are directly addressed by the Solaris OE.
● They consist of one or more plexes. Each plex holds one copy of the
volume’s data.
● Volumes are not restricted to a single disk or specific areas of disks.

Sun Proprietary: Internal Use Only


1-26 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining VxVM Software Objects

● The configuration of a volume can be changed using the VxVM


software utilities. Configuration changes can be accomplished
without disruption to the volume operations.
● Volume names can contain up to 31 characters.
● Volumes can consist of up to 32 plexes.
Each plex can contain multiple subdisks, and all subdisks must be in
the same disk group.

Figure 1-7 illustrates a volume with two plexes.

Plex Volume

disk01-01
disk01-02
disk01-03
disk01-01 disk02-01
vol01-01 disk01-02 disk02-02
disk01-03 disk02-03

disk02-01 vol01-01 vol01-02


disk02-02 vol01
disk02-03
vol01-02

Figure 1-7 Two-Plex Volume

Note – Detailed volume layouts are described in Appendix E,


“Configuring the VxVM Software.”

VxVM Software Layered Volume Objects


With the VxVM software version 3.0, volumes can now be constructed by
mapping a volume’s subdisks to underlying volumes and using those
volumes to build volumes. Volumes constructed in this manner are virtual
objects called layered or professional volumes.

Sun Proprietary: Internal Use Only


Introducing the VERITAS Volume Manager Software Architecture 1-27
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining VxVM Software Objects

Layered volumes are used to implement RAID 1 + 0 or RAID 10 virtual


devices. RAID 1 + 0 devices are striped, mirrored volumes and provide
for a more fault-resilient volume configuration. Implementing these
layered volumes require two new VxVM software objects: subplexes and
subvolumes.

Figure 1-8 illustrates the how a layered volume differs from a standard
volume.

Volume Volume

Plex
Mirror Subvolume

Mirror
Plex Plex
Subplex Subplex
SD SD
SD SD
Stripe

SD SD

SD SD Stripe Subvolume

Mirror
SD SD
Subplex Subplex

SD SD
RAID 0+1
Subvolume

Mirror

Subplex Subplex

SD SD

RAID 1+0

Figure 1-8 Layered Volumes

Layered volumes require more VxVM software objects than standard


volumes. A standard volume requires 9 objects. A layered volume of the
same capacity requires 17 objects. Large disk groups using layered
volumes can exceed the alloted space for the disk group’s private region.

Sun Proprietary: Internal Use Only


1-28 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Resynchronizing Volumes

Resynchronizing Volumes
The VxVM software ensures that all data stored redundantly using
mirrored or RAID 5 volumes remains in a consistent state. Data is written
in parallel to both mirrored and RAID volumes to ensure that data
remains consistent unless there is a system crash or physical disk failure.
If this occurs, data can become inconsistent or unsynchronized.

System failures are not the only reason data can become unsynchronized.
Data can become inconsistent during maintenance procedures when a
mirrored plex or RAID 5 element is taken offline. If data becomes
inconsistent between mirrored plexes or between a RAID 5 volume’s data,
use a volume resynchronization to correct the problem.

Volume resynchronization happens for some volumes during a system


reboot. Other volumes undergo volume resynchronization when they are
started. Plexes that are taken offline for maintenance become stale. Stale
plexes are resynchronized when configured back into a volume.

The components of volume resynchronization are discussed in detail in


this section.

Dirty Flag
The VxVM software keeps track of data synchronization operations using
a flag called the dirty flag. When data is written to a volume, it is marked
as dirty until the volume stops or all writes are completed and the data in
the volume plexes are identical.

Volumes that do not have the dirty flag reset require volume
resynchronization when started or during a system reboot.

Sun Proprietary: Internal Use Only


Introducing the VERITAS Volume Manager Software Architecture 1-29
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Resynchronizing Volumes

Resynchronization Process
The resynchronization (resync) process depends on the type of volume
started.

During resync of a RAID 5 volume, if a log is available, the log is


replayed. If a log is not available, the volume is placed in reconstruct-
recovery mode, and all parity is regenerated. This is a very time
consuming recovery process. Make sure that all RAID 5 volumes have a
log attached to reduce the time it takes to resync a RAID 5 volume.

During resync of a mirrored volume, the volume is placed into recovery


mode (or read-writeback recovery mode). Data is available for use by the
Solaris OE. This type of recovery is executed in the background and is
very time consuming. Attaching a dirty region log (DRL) to mirrored
volumes and using a fast resync operation can help speed up the recovery
process.

The resync process can be expensive and have an adverse impact on


system resources. The VxVM software reduces the impact multiple
resyncs have on the system by managing the recovery operations to avoid
placing stress on specific disks and controllers.

Sun Proprietary: Internal Use Only


1-30 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Introducing VxVM Software Logging

Introducing VxVM Software Logging


The VxVM software transaction log facility provides a method of
decreasing the amount of time it takes to perform volume
resynchronization. This is accomplished by attaching a log plex to a
volume and using that plex to save transaction recovery information.

There are two types of logs: RAID 5 and DRL. This section contains a
detailed description of these logs.

RAID 5 Logs
RAID 5 logs perform the following tasks:
● RAID 5 logs protect data and parity calculations from system
crashes.
Degraded RAID 5 volumes are not restarted during a reboot; they
must be started manually. Attaching a RAID 5 log does not alter this
fact.
● RAID 5 parity is only used when data needs to be rebuilt. It is only
written and never checked.
It is advised to run the vxr5check utility on a regular basis to check
parity.
● RAID 5 logs appear as a plex with a plex state of Log.

Configure the RAID 5 logs as follows:


● Create a RAID 5 log that is large enough to allow multiple
concurrent accesses to the volume. Make the log several times the
stripe size of a RAID 5 plex.
● It is possible to configure a RAID 5 volume without a log, but it is
highly discouraged. It is not a a good practice to configure RAID 5
volumes without logs.
RAID 5 volumes without logs are subject to silent parity errors in the
event of a system crash.
● Sun StorEdge documentation recommends using two RAID 5 logs
per RAID 5 volume.
Using two logs can have a significant performance impact on a
volume type that already has performance problems. Most sites only
use one log.

Sun Proprietary: Internal Use Only


Introducing the VERITAS Volume Manager Software Architecture 1-31
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Introducing VxVM Software Logging

The following is an example of a RAID 5 volume with an attached log:


Disk group: ddg

DG NAME NCONFIG NLOG MINORS GROUP-ID


DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
V NAME USETYPE KSTATE STATE LENGTH READPOL PREFPLEX
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE

v r0 RAID-5 ENABLED ACTIVE 20480 RAID -


pl r0-01 r0 ENABLED ACTIVE 21568 RAID 3/32 RW
sd ddg01-01 r0-01 ddg01 0 10800 0/0 c1t2d0 ENA
sd ddg02-01 r0-01 ddg02 0 10800 1/0 c1t2d1 ENA
sd ddg03-01 r0-01 ddg03 0 10800 2/0 c1t3d0 ENA
pl r0-02 r0 ENABLED LOG 4320 CONCAT - RW
sd ddg04-01 r0-02 ddg04 0 4320 0 c1t4d0 ENA

Dirty Region Logs (DRLs)


Dirty region logging provides a facility that helps speed the process of
resynchronizing a mirrored volume after a system crash. The DRL process
provides a bit map image of the data blocks available in a volume and
sets the dirty bit whenever data is written to a specific block. If a system
crash occurs, only the dirty blocks are resynchronized. If the DRL is not
used, all mirrors must be returned to a consistent state through a full
resynchronization of the volume.

Dirty region logging performs the following tasks:


● It provides faster resynchronization in the event of a system crash or
abnormal system termination only.
In case of a system crash, only the dirty blocks are resynchronized.
● The logging does not perform the same function as the Solstice
DiskSuite software’s dirty mirror region log. Dirty region logging
does not enable partial resynchronization if a plex is taken offline for
maintenance.
Partial resynchronization is handled by the fast mirror resync (FMR)
facility. FMR requires a separate license.
● Dirty region logging requires additional system I/O overhead.
● A DRL is relatively small.

Sun Proprietary: Internal Use Only


1-32 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Introducing VxVM Software Logging

● DRLs are not supported on core system volumes, such as the /, /usr,
and /var volumes.
● Dirty region logging is usually implemented as a log plex.

The following is an example of a mirrored plex with an attached log:


Disk group: ddg

DG NAME NCONFIG NLOG MINORS GROUP-ID


DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
V NAME USETYPE KSTATE STATE LENGTH READPOL PREFPLEX
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE

v m0 fsgen ENABLED ACTIVE 20480 SELECT -


pl m0-01 m0 ENABLED ACTIVE 21600 CONCAT - RW
sd ddg04-02 m0-01 ddg04 4320 21600 0 c1t4d0 ENA
pl m0-02 m0 ENABLED ACTIVE 21600 CONCAT - RW
sd ddg05-01 m0-02 ddg05 4320 21600 0 c1t4d1 ENA
pl m0-03 m0 ENABLED ACTIVE LOGONLY CONCAT - RW
sd ddg04-03 m0-03 ddg04 25920 5 LOG c1t4d0 ENA
v m1 fsgen ENABLED ACTIVE 20480 SELECT -
pl m1-01 m1 ENABLED ACTIVE 21600 CONCAT - RW
sd z m1-01 ddg05 0 4320 LOG c1t4d1 ENA
sd ddg01-02 m1-01 ddg01 10800 21600 0 c1t2d0 ENA
pl m1-02 m1 ENABLED ACTIVE 21600 CONCAT - RW
sd ddg02-02 m1-02 ddg02 10800 21600 0 c1t2d1 ENA

Sun Proprietary: Internal Use Only


Introducing the VERITAS Volume Manager Software Architecture 1-33
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining Plex States

Examining Plex States


Plex states change a part of the normal volume operations and do not
necessarily indicate a problem. When a plex is activated as part of a
volume being started or due to an error-recovery procedure, the targeted
plex progresses through a pre-defined set of plex states. When a plex is
taken offline due to an error and a volume resynchronization is initiated,
all plex data is resynchronized, allowing the volume to return to the
synchronized state. During the resynchronization process, the out-of-sync
plexes move through various plex states until the operation completes.

Plex states are automatically maintained by VxVM software utilities


which:
● Indicate the state of the volumes contents
● Determine if a plex content is valid
● Track whether a plex was actively in use at the time of a system
failure
● Monitor plex operations

Plex states are discussed in detail in this section.

Plex State Descriptions


Plex states refer to the states the plex transitions through to ensure they
contain a complete and consistent copy of a volume’s contents. Plexes
associated with a volume can have one of the following states:
● Empty – When a volume is created and the plex is not initialized, the
plex is in an empty state.
● Clean – A plex is in a clean state when it is known to contain a good
copy (mirror) of the volume. If all the plexes of a volume are clean,
no action is required.
● Active – A plex is in the active state when the volume is started and
the plex fully participates in normal volume I/O (meaning the plex
contents change as the contents of the volume change).
A plex state is also active when the volume is stopped as a result of
a system crash and the plex was active at the moment of the crash. In
this case, a system failure can leave plex contents in an inconsistent

Sun Proprietary: Internal Use Only


1-34 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining Plex States

state. When a volume is started, the VxVM software performs a


recovery action to guarantee that the contents of the plexes are
identical and are marked as active.

Note – The active state is the most common state for plexes on a well-
running system.

● Stale – If there is a possibility a plex does not have the complete and
current volume contents, this plex is placed in a stale state.
Additionally, if I/O errors occur on a plex, the kernel stops using
and updating the plex, and the operation sets the state of the plex to
stale.
● To reattach the stale plex to the volume, synchronize the data
and set the plex to the active state, use the following command:
*vxplex -g (diskgroup) att volumename plexname*
● To force a plex into Stale state, use the following command:
*vxplex -g (diskgroup) det plexname*
● Offline – The following command detaches a plex from a volume
setting and changes the plex state to offline:
*vxmend -g (diskgroup) off plexname*
Although the detached plex is associated with the volume, the
changes to the volume are not reflected to the plex while it is in the
offline state.
● To set the plex state to stale and start to recover data after the
vxvol start operation, use the following command:
*vxplex -g (diskgroup) att volumename plexname*
● Temp – A utility sets the plex state to temp at the start of an
operation, and also sets the plex to an appropriate state at the end of
the operation.
For example, attaching a plex to an enabled volume requires copying
volume contents to the plex before it can be fully attached. If the
system goes down for any reason, a temp plex state indicates the
operation is incomplete; a subsequent vxvol operation starts to
disassociate plexes in the temp state.

Sun Proprietary: Internal Use Only


Introducing the VERITAS Volume Manager Software Architecture 1-35
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining Plex States

● Temprm – A temprm plex state resembles a temp state with the


exception that, upon completion of the operation, the temprm plex is
removed.
If the system goes down for any reason, a temprm plex state
indicates the operation is incomplete; a subsequent vxvol operation
starts to disassociate plexes and remove the temprm plex.
● Temprmsd – The temprmsd plex state is used by vxassist when
attaching new plexes. If the operation does not complete, the plex
and its subdisk are removed.
● Iofail – The iofail plex state is associated with persistent state
logging. On the detection of a failure of an active plex, the
vxconfigd operation places that plex in the iofail state to disqualify
it from the recovery selection process at volume start time.

Plex Kernel States


The plex kernel state indicates the accessibility of a plex to the volume
driver monitoring it. Valid plex kernel states are:
● Disabled – The plex may not be accessed.
● Detached – A write to the volume is not reflected to the plex. A read
request from the volume is never satisfied from the plex.
Plex operations and ioctl functions are accepted.
● Enabled – A write request to the volume is reflected to the plex. A
read request from the volume is satisfied from the plex.

Sun Proprietary: Internal Use Only


1-36 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining Plex States

General Plex State Cycle


Volumes are automatically started at system startup, and all clean plexes
are marked active by the vxvol start command. At system shutdown,
all plexes in the active state are marked clean by the vxvol stop
command. If all plexes are in the clean state, this is indicative of a
controlled system shutdown.

This cycle is illustrated in Figure 1-9.

Start Up (vxvol start)

PS: Clean PS: Active


PKS: Disabled PKS: Enabled

PS = Plex state
PKS = Plex kernel state

Shut Down (vxvol stop)

Figure 1-9 General Plex State Cycle

Other Plex State Transitions


Additional plex states are needed to help transition plexes during
hardware error recovery, abnormal system shutdown, and system
administrator intervention.

Figure 1-10 on page 1-38 illustrates these state transitions.

Sun Proprietary: Internal Use Only


Introducing the VERITAS Volume Manager Software Architecture 1-37
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining Plex States

Create plex

PS: Empty PS: Active


PKS: Disabled PKS: Disabled
After crash
and reboot
(vxvol start)

Initialize plex Start up Recover data


(vxvol init clean) (vxvol start) (vxvol resync)
Take plex offline
(vxmend off)

PS:Clean PS: Active PS: Offline


PKS: Disabled PKS: Enabled PKS: Disabled

Resync data
(vxplex att)
Shut down
(vxvol stop)
Put plex online
(vxmend on)
Uncorrectable
I/O failure

Resync PS: Stale


PS: IOFAIL
fails PKS: Detached
PKS: Detached

PS = Plex state
PKS = Plex kernel state

Figure 1-10 Additional Plex State Transitions

An example of how these additional states are used to manage the


integrity of plexes is as follows:
1. A new plex is created leaving the plex state (PS) as empty and the
plex kernel state (PKS) as disabled.
2. The plex is initialized and transitions to the clean PS. It remains in
the disabled PKS.
3. Once the volume is started, the plex transitions to the active PS and
to the enabled PKS. At this time, the plex is available for use by the
volume and stays in this state until an error occurs or the volume is
stopped.

Sun Proprietary: Internal Use Only


1-38 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining Plex States

When an uncorrectable error happens, the following plex states are used
to manage recovery:
1. An un-correctable I/O failure occurs and the PS transitions to the
iofail state. The PKS transitions to the detached state.
2. Repairs are affected and the plex is reattached to the volume. This
causes the execution of a data resync. Once the data in the plex is
updated, the PS transitions to active, and the PKS transitions to
enabled.
The plex is now usable by the volume.

Use the vxprint utility to view plex states and their transitions. If a small
plex is re-synching, vxprint might not show the transition states as they
may happen too fast for vxprint to report.

The following example shows a VM disk (rootmirror) that has an I/O


failure. The plex rootvol-02 has transitioned to a detached PKS and an
iofail PS.
bash-2.03# vxprint -htg rootdg
DG NAME NCONFIG NLOG MINORS GROUP-ID
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
V NAME RVG KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO

dg rootdg default default 0 1023554924.1025.lowtide

dm rootdisk c1t0d0s2 sliced 3590 17678493 -


dm rootmirror - - - - IOFAIL

v rootvol - ENABLED ACTIVE 4197879 ROUND - root


pl rootvol-01 rootvol ENABLED ACTIVE 4197879 CONCAT - RW
sd rootdisk-B0 rootvol-01 rootdisk 17678492 1 0 c1t0d0 ENA
sd rootdisk-02 rootvol-01 rootdisk 0 4197878 1 c1t0d0 ENA
pl rootvol-02 rootvol DETACHED IOFAIL 4197879 CONCAT - RW
sd rootmirror-01 rootvol-02 rootmirror 0 4197879 0 - RLOC

Sun Proprietary: Internal Use Only


Introducing the VERITAS Volume Manager Software Architecture 1-39
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Introducing Supported VxVM Software Version 3.2 Features

Introducing Supported VxVM Software Version 3.2


Features
The following section describes new features introduced with the release
of the VxVM software version 3.2 that are supported by Sun
Microsystems.

Device Discovery
Device discovery is separated from the base VxVM software functionality
into a separate layer. Previously, the VxVM software discovered block
storage devices by scanning the Solaris OE device tree using the vxiod
daemon. This strategy assumed that the Solaris OE device tree remained
static or changed very little. When changes occurred, command line
utilities were used to update the VxVM software configuration.

Additionally, dynamic multi-pathing (DMP) must be aware of all multi-


pathed storage subsystems and know the type of storage subsystem to
enable DMP support. Prior to the VxVM software version 3.2, support for
new arrays would necessitate changes to the DMP device discovery code.
This required patching the VxVM software and undergoing a probable
system outage.

With the growth of disk subsystem vendors and storage area networks
implemented within storage environments, the previous strategy proved
to be inadequate. The current device discovery facility is designed to
allow the dynamic addition of new storage subsystems without
modification to the VxVM software modules. The VxVM software DDL
discovers all disks that are visible to the Solaris OE.

Device Discovery Layer

The DDL is part of the vxconfigd daemon, and performs the following:
● Discovers all block storage devices connected to a host
● Probes using SCSI commands from user space to determine the
following:
● Type of disks
● Number of paths

Sun Proprietary: Internal Use Only


1-40 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Introducing Supported VxVM Software Version 3.2 Features

● Attributes
● Executes from the vxdisk utility to perform a vxdisk scan
command

Figure 1-11 illustrates DDL components and the relationship to the VxVM
software kernel.

Applications

Data Management Layer


VxVM
File System Database Configuration

Database

VxVM Storage Management Subsystem

DMP Driver

Storage Device Drivers

Kernel

Enclosure/Array vxconfigd

DDL

Figure 1-11 Device Discovery Layer

Addition of New Devices

Libraries are included with the VxVM software for the most popular
brands of arrays. If a new array is created by a vendor, the VxVM
software only needs an updated device discovery library to recognize the
new array. The new library can be added dynamically.

Sun Proprietary: Internal Use Only


Introducing the VERITAS Volume Manager Software Architecture 1-41
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Introducing Supported VxVM Software Version 3.2 Features

Support can be added dynamically for the following types of disk arrays:
● Active/Active
● Active/Passive (Autorepass mode)
● Active/Passive (LUN group failover)
● JBODs

JBOD Support

The device discovery facility can detect multi-pathed disk storage devices
that do not belong to a disk array but are capable of being multi-pathed
by DMP. Use the vxddladm utility to add or remove JBODs.

Disks must have a unique serial number that can be read through a SCSI
inquiry or the mode_sense command to be detected correctly.

DDL Administration

The device discovery facility is managed by the vxddladm utility to


provide a high-level function for device discovery and configuration. It is
used to:
● Add JBODs
● List supported JBODs
● Remove JBODs
● Include arrays
● Exclude arrays
● List supported arrays
● List excluded arrays

A sample output from the execution of the vxddladm command with the
listsupport option is shown here. This option lists all disk enclosures
currently supported by the system’s DDL.
bash-2.03# vxddladm listsupport
LIB_NAME ARRAY_TYPE VID PID
=======================================================================
libvxap.so A/A SUN AP_NODES
libvxatf.so A/A VERITAS ATFNODES
libvxeccs.so A/A ECCS all
libvxemc.so A/A EMC SYMMETRIX
libvxhds.so A/A HITACHI OPEN-*
libvxhitachi.so A/PG HITACHI DF350

Sun Proprietary: Internal Use Only


1-42 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Introducing Supported VxVM Software Version 3.2 Features

libvxhitachi.so A/PG HITACHI DF400


libvxhitachi.so A/PG HITACHI DF500
libvxlsiinf.so A/P LSI INF-01-00
libvxnec.so A/A NEC DS1200
libvxnec.so A/A NEC DS1200F
libvxnec.so A/A NEC DS3000SL
libvxnec.so A/A NEC DS3000SM
libvxnec.so A/A NEC DS3001
libvxnec.so A/A NEC DS3002
libvxnec.so A/A NEC DS1000
libvxnec.so A/A NEC DS1000F
libvxnec.so A/A NEC DS1100
libvxnec.so A/A NEC DS1100F
libvxnec.so A/A NEC DS3011
libvxnec.so A/A NEC DS1230
libvxnec.so A/A NEC DS450
libvxnec.so A/A NEC DS450F
libvxnec.so A/A NEC iStorage 1000
libvxnec.so A/A NEC iStorage 2000
libvxnec.so A/A NEC iStorage 4000
libvxpurple.so A/P SUN T300
libvxrdac.so A/A VERITAS RDACNODES
libvxsena.so A/A SENA all
libvxshark.so A/A IBM 2105
libvxssa.so A/A SSA all
libvxstorcomp.so A/A StorComp OmniForce
libvxveritas.so A/PF VERITAS all
libvxxp256.so A/A HP OPEN-*
libvxfujitsu.so A/A FUJITSU GR710
libvxfujitsu.so A/A FUJITSU GR720
libvxfujitsu.so A/A FUJITSU GR730
libvxfujitsu.so A/A FUJITSU GR740
libvxfujitsu.so A/A FUJITSU GR820
libvxfujitsu.so A/A FUJITSU GR840

Note – See the vxddladm man page for command syntax options.

Device Naming
Enclosure-based and operating system (OS)-independent naming changes
the way storage devices are identified by the VxVM software and
provides the following benefits:
● Naming is independent of the OS.
This mitigates device naming confusion in multi-OS environments.

Sun Proprietary: Internal Use Only


Introducing the VERITAS Volume Manager Software Architecture 1-43
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Introducing Supported VxVM Software Version 3.2 Features

● OS-independent naming relieves problems with some third-party


drivers that do not use the standard Solaris OE c#t#d# device
naming schema.
The VxVM software is now independent of arbitrary device names.
● Enclosure-based naming allows for better location identification of
disks.
This leads to faster fault isolation.
● In cluster environments, disk array names are the same on all nodes
in the cluster.

Device Name Format

The device name format is based on the name of the enclosure. Use the
following format to name devices:
logical_enclosure_name_#

Default disk names are based on the vendor identification (ID). For
example, disks in a Sun StorEdge T3 multi-path array named purple0 by
the VxVM software have the following names:
purple0_1
purple0_2
purple0_3

Device names have the following characteristics:


● Logical names persist across reboots.
● Logical enclosure names can be customized.
● Disk names within enclosures are persistent as long as the disk’s
position within an enclosure remains static.
● All fabric devices are displayed using the enclosure format.
During installation, an option allows the installer to display all non-
fabric devices using the native open profiling standard (OPS) format.

Enabling and Disabling Naming Schemes

Enclosure-based naming is enabled or disabled as follows:


● Enabled or disabled during installation.
● Enabled or disabled using vxdiskadm option 20.

Sun Proprietary: Internal Use Only


1-44 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Introducing Supported VxVM Software Version 3.2 Features

● Enabled by executing the following commands:


# touch /etc/vx/.newnames
# vxconfigd -k
● Disabled by executing the following commands:
# rm /etc/vx/.newnames
# vxconfigd -k

This example shows a system with enclosure-based naming enabled.


bash-2.03# ls /dev/vx/dmp
SENA0_0 SENA0_13s4 SENA0_5s2 c1t0d0 c1t19d0s5 c1t2d0s2
SENA0_0s0 SENA0_13s5 SENA0_5s3 c1t0d0s0 c1t19d0s6 c1t2d0s3
SENA0_0s1 SENA0_13s6 SENA0_5s4 c1t0d0s1 c1t19d0s7 c1t2d0s4
SENA0_0s2 SENA0_13s7 SENA0_5s5 c1t0d0s2 c1t1d0 c1t2d0s5
SENA0_0s3 SENA0_1s0 SENA0_5s6 c1t0d0s3 c1t1d0s0 c1t2d0s6
SENA0_0s4 SENA0_1s1 SENA0_5s7 c1t0d0s4 c1t1d0s1 c1t2d0s7
SENA0_0s5 SENA0_1s2 SENA0_6 c1t0d0s5 c1t1d0s2 c1t3d0
SENA0_0s6 SENA0_1s3 SENA0_6s0 c1t0d0s6 c1t1d0s3 c1t3d0s0
SENA0_0s7 SENA0_1s4 SENA0_6s1 c1t0d0s7 c1t1d0s4 c1t3d0s1
...

Notice that the dmp metanodes now use devices named SENA0_xyz. A
long listing of the directory shows that c1t0d0 and SENA0_0 are the same
device by comparing the major and minor numbers.
bash-2.03# ls -las /dev/vx/dmp
total 12
10 drwxr-xr-x 2 root other 5120 Jul 5 09:17 .
2 drwxr-xr-x 6 root other 512 Jun 8 09:49 ..
0 brw------- 1 root other 68, 2 Jul 5 09:05 SENA0_0
...
0 brw------- 1 root other 68, 2 Jun 8 09:41 c1t0d0
...

An experienced system administrator can take the c#t#d# number and


cross-reference that to a physical device.

A sample vxdisk command listing shows that the VxVM software is


using enclosure-based names.
bash-2.03# vxdisk list
DEVICE TYPE DISK GROUP STATUS
SENA0_0 sliced rootdisk rootdg online
SENA0_1 sliced - - offline
SENA0_2 sliced - - error
SENA0_3 sliced - - online
SENA0_4 sliced - - online
SENA0_5 sliced - - offline
SENA0_6 sliced rootmirror rootdg online
SENA0_7 sliced - - online

Sun Proprietary: Internal Use Only


Introducing the VERITAS Volume Manager Software Architecture 1-45
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Introducing Supported VxVM Software Version 3.2 Features

SENA0_8 sliced datadg02 datadg online


SENA0_9 sliced datadg01 datadg online
SENA0_10 sliced - - online
SENA0_11 sliced - - online
SENA0_12 sliced - - online
SENA0_13 sliced - - online

Sun StorEdge Traffic Manager Software Support


The VxVM software version 3.2 is the first version of this software to
support the Sun StorEdge Traffic Manager software.

Explicit LUN Failover


The LUN failover feature enables systems that are using DMP in place of
the Sun StorEdge Traffic Manager software to manage path and LUN
failover functions.

To enable this support within the Sun StorEdge T3 array, configure the
following modifications in the mp_support file:
● Single-host environment – Set mp_support to rw.
● Multi-host environment – Set mp_support to std.

Ordered Storage Allocation


Ordered storage allocation is a new option to the vxassist utility. This
feature allows a specification in which subdisks are allocated for use when
building a volume. In early versions of the VxVM software (before
version 3.2) and before ordered allocation, an administrator could specify
which disks to use, but the VxVM software would allocate the order in
which those disks were used based on the software’s proximity rules.

The following example command shows how to use the ordered storage
allocation option:
# vxassist -g testdg -o ordered make vol01 1g layout=mirror-stripe ncol=3 disk02 disk04
disk06 disk01 disk03 disk05

Sun Proprietary: Internal Use Only


1-46 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Introducing Unsupported VxVM Software Version 3.2 Features

Introducing Unsupported VxVM Software Version 3.2


Features
The following features, although available in the VxVM software
version 3.2, are not supported by Sun Microsystems. These features
require an additional license.

Disk Group Split, Move, and Join


The disk group split, move, and join facility allows greater flexibility in
disk group management functions. These actions are described as follows:
● Split – Removes objects from an established disk group and creates a
new disk group from those objects
● Move – Moves objects from one disk group to another
● Join – Joins two disk groups

Persistent Fast Resync Function


The fast resync and persistent fast resync functions allow for fast recovery
of a mirrored volume that contains a plex that was taken offline for
maintenance. Fast resync uses data change objects (DCO) to keep track of
changes to a mirror. Persistent fast resync stores resynchronized maps on
a disk rather than in memory, which enables the maps to survive a system
reboot or crash.

Sun Proprietary: Internal Use Only


Introducing the VERITAS Volume Manager Software Architecture 1-47
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Module 2

Encapsulating Disks

Objectives
Upon completion of this module, you should be able to:
● Describe the encapsulation and unencapsulation processes for data
and boot (system) disks
● Encapsulate a data disk
● Unencapsulate a data disk
● Encapsulate a boot disk
● Recover the loss of the encapsulated disk in a boot disk mirrored pair
● Describe best practice partition configurations for boot disk
encapsulation
● Unencapsulate a boot disk, including a compact disc read-only
memory (CD-ROM)
● Perform a successful vxunroot process on a boot disk mirror,
including a mirror that has a replaced encapsulated disk
● Recover an encapsulated boot disk that failed the vxunroot process

Sun Proprietary: Internal Use Only


2-1
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Relevance

Relevance

Discussion – The following questions are relevant to understanding how


the VxVM software supports the preservation (encapsulation) of existing
! data on disks brought under the software’s control:
?

● What is disk encapsulation?


● How is disk encapsulation implemented?
● What are the differences between encapsulating a data disk as
opposed to a system (boot) disk?
● How does a user reverse the encapsulation process without having
to restore data?
● Should the VxVM software be used to encapsulate a system boot
disk?
● What are the supported slice configurations for a boot disk?
● What process recovers an encapsulated boot disk that fails the
vxunroot utility’s attempt at unencapsulation?

Sun Proprietary: Internal Use Only


2-2 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Additional Resources

Additional Resources

Additional resources – The following references provide additional


information on the topics described in this module:
● VERITAS Volume Manager™ 3.2 Administrator’s Guide. Mountain
View, California: VERITAS Software Corporation, August 2001,
number 30-000392-011, TechPDF ID 240253.
● VERITAS Volume Manager™ 3.2 Installation Guide. Mountain View,
California: VERITAS Software Corporation, August 2001,
number 30-000395-011, TechPDF ID 240256.
● VERITAS Volume Manager™ 3.2 Troubleshooting Guide. Mountain
View, California: VERITAS Software Corporation, August 2001,
number 30-000394-011, TechPDF ID 240255.
● VERITAS Volume Manager™ Storage Administrator 3.2 Administrator’s
Guide. Mountain View, California: VERITAS Software Corporation,
July 2001, number 30-000393-011, TechPDF ID 240257.
● http://storage.east, http://storage.central, and
http://storage.west
● SunSolveSM Online Free Info Docs and Free Symptoms and Resolutions,
[http://sunsolve.Sun.COM/pub-cgi/search.pl?mode=
advanced].
● INFODOC 16051 – “How to ‘Encapsulate’ Disks With No Free
Space Using Volume Manager” 22 March 2002. (See
Appendix A, “SunSolve INFODOCs.”)
● INFODOC 24663 – “Full and Basic/Functional Unencapsulation
of a Volume Manager Encapsulated Root Disk While Booted
CDROM” 22 March 2002. (See Appendix A, “SunSolve
INFODOCs.”)
● INFODOCS 13775, 13781, 15838, and 19245.
● Symptom and Resolution Database (SRDB) 19245.
● VERITAS Software Corporation support knowledge base
TechNote ID 244678,
[http://seer.support.veritas.com/nav_bar/index.asp?
content_sURL=%2Fsearch%5Fforms%2Ftechsearch%2Easp].
● Sun BluePrints™ OnLine part number 806-6197-10, Toward a
Reference Configuration for VxVM Managed Boot Disks, August, 2000.
[http://www.sun.com/solutions/blueprints/online.html].

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-3
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Introducing Disk Encapsulation

Introducing Disk Encapsulation


The VxVM software uses two methods to bring disk storage devices
under control: initialization and encapsulation. Initialization is the process
of partitioning a data disk into two partitions for use as public and private
regions. This process destroys all data on the initialized disk. Initialized
disks are then used to build volumes. The encapsulation process provides
the VxVM software with the ability to manage a disk without destroying
data. Each partition on the encapsulated disk becomes a unique VxVM
software volume. Once encapsulated, other initialized VxVM software
disks can be used to provide mirrors for these partitions.

Appendix B, “Disk Encapsulation Processes,” provides an overview of the


different processes which the VxVM software performs involving disk
encapsulation.

Encapsulated Disk Types


There are two types of encapsulated disks, data disks (non-root) and
system (boot or root) disks. Data disks do not contain any system data
needed to boot the operating system. Boot disks do contain data that is
needed to successfully boot the operating system. Rootability is the VxVM
software’s ability to encapsulate system partitions that are usable by the
system. The VxVM software manages data and boot disks differently, as
described in this module.

Reasons to Encapsulate Disks


There are at least four good reasons to encapsulate a disk:
● Increase the fault resiliency of data through mirroring
● Grow the data storage area
● Convert a data disk to a striped volume to increase performance
● Provide a disk to populate the rootdg disk group

Sun Proprietary: Internal Use Only


2-4 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Encapsulating Data Disks

Encapsulating Data Disks


Data disk encapsulation involves changes to the encapsulated disks
partition table, and has the following requirements:
● Two unused partitions in the partition table
● A small amount of unallocated disk space (2 cylinders) at the
beginning or end of the disk
● Slice 2 representing the whole disk

If these requirements are not met, encapsulation is a more difficult


procedure as described in ‘‘Encapsulating a Non-Conforming Disk’’ on
page 2-20.

When the encapsulation process is finished, the encapsulated disk has two
partitions. These are usually partition 6, used for the public region, and
partition 7, used for the private region. Figure 2-1 illustrates this
partitioning.

Data Disk Encapsulated Data Disk

Slice 0
Public
Slice 1 Slice 2
Slice 2 Region
and
(Slice 6)
Slice 3 Slice 6

Slice 4
Slice 7
Free Space Private Region

Figure 2-1 Encapsulation Partition Scheme

Although the VxVM software re-partitions the disk, the original data
resides in the same blocks they occupied prior to encapsulation. The data
is not moved or overwritten. The VxVM software is responsible for
performing the translation necessary to make the data available as VxVM
software volumes.

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-5
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Encapsulating Data Disks

Pre-Encapsulation Configuration Data


The following are example configuration data from a system prior to
installation of the VxVM software and encapsulation of a data disk. The
state of this data prior to the encapsulation process is discussed in this
section.

Note – In some examples of output presented in this module, columns are


aligned and spaces added to aid clarity.

Pre-Encapsulation /etc/vfstab File

The /etc/vfstab file displays the file systems to be encapsulated on a


disk. This example of data disk encapsulation uses disk c1t3d0 as the
target disk.
#device device mount FS fsck mount mount
#to mount to fsck point type pass at boot options
#
#/dev/dsk/c1d0s2 /dev/rdsk/c1d0s2 /usr ufs 1 yes -
fd - /dev/fd fd - no -
/proc - /proc proc - no -
/dev/dsk/c1t0d0s1 - - swap - no -
/dev/dsk/c1t0d0s0 /dev/rdsk/c1t0d0s0 / ufs 1 no logging
swap - /tmp tmpfs - yes -
#
# Storage
#
/dev/dsk/c1t3d0s0 /dev/rdsk/c1t3d0s0 /fs1 ufs 1 yes logging <--\
/dev/dsk/c1t3d0s1 /dev/rdsk/c1t3d0s1 /fs2 ufs 2 yes logging <- Disk to be encapsulated.
/dev/dsk/c1t3d0s3 /dev/rdsk/c1t3d0s3 /fs3 ufs 3 yes logging <- /
/dev/dsk/c1t3d0s4 /dev/rdsk/c1t3d0s4 /fs4 ufs 4 yes logging <--/

Pre-Encapsulation df -k Command

This example output from the df -k command displays the file systems
currently mounted from disk c1t3d0.
# df -k
Filesystem kbytes used avail capacity Mounted on
/dev/dsk/c1t0d0s0 7670973 1536449 6057815 21% /
/proc 0 0 0 0% /proc
fd 0 0 0 0% /dev/fd
mnttab 0 0 0 0% /etc/mnttab
swap 1209520 16 1209504 1% /var/run
swap 1209520 16 1209504 1% /tmp

Sun Proprietary: Internal Use Only


2-6 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Encapsulating Data Disks

/dev/dsk/c1t3d0s1 2055705 2073 1991961 1% /fs2 <-- Mount Points


/dev/dsk/c1t3d0s0 2055705 2073 1991961 1% /fs1 <--/ / /
/dev/dsk/c1t3d0s4 2055705 2073 1991961 1% /fs4 <---/ /
/dev/dsk/c1t3d0s3 2055705 2073 1991961 1% /fs3 <----/

Pre-Encapsulation prtvtoc Command

Executing the prtvtoc command on the target disk shows if there is


space available at the beginning or end of the disk for a private region. If
there is not sufficient space available, the alternative method of
encapsulation must be used. The alternative disk encapsulation process is
described in ‘‘Encapsulating a Non-Conforming Disk’’ on page 2-20.
# prtvtoc /dev/rdsk/c1t3d0s2
* /dev/rdsk/c1t3d0s2 partition map
*
* Dimensions:
* 512 bytes/sector
* 133 sectors/track
* 27 tracks/cylinder
* 3591 sectors/cylinder
* 4926 cylinders
* 4924 accessible cylinders
*
* Flags:
* 1: unmountable
* 10: read-only
*
* Unallocated space:
* First Sector Last
* Sector Count Sector
* 16791516 890568 17682083 <---- Unallocated space at end of disk.
*
* First Sector Last
* Partition Tag Flags Sector Count Sector Mount Directory
0 0 00 0 4197879 4197878 /fs1 <-- Mount Points
1 0 00 4197879 4197879 8395757 /fs2 <--
2 5 01 0 17682084 17682083
3 0 00 8395758 4197879 12593636 /fs3 <--
4 0 00 12593637 4197879 16791515 /fs4 <--

Notice that the beginning sector of unallocated space is the last sector of
partition 4 plus 1.

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-7
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Encapsulating Data Disks

Pre-Encapsulation format Utility Partition Print

The print option on the format utility partition menu shows the present
partition configuration of the disk. Print this information and save a copy
in a safe place to use if the disk needs to be unencapsulated. Note that
there are at least two unused partitions available in the partition table
(partitions 6 and 7).
partition> print
Current partition table (original):
Total disk cylinders available: 4924 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks


0 unassigned wm 0 - 1168 2.00GB (1169/0/0) 4197879
1 unassigned wm 1169 - 2337 2.00GB (1169/0/0) 4197879
2 backup wu 0 - 4923 8.43GB (4924/0/0) 17682084
3 unassigned wm 2338 - 3506 2.00GB (1169/0/0) 4197879
4 unassigned wm 3507 - 4675 2.00GB (1169/0/0) 4197879
5 unassigned wm 0 0 (0/0/0) 0
6 unassigned wm 0 0 (0/0/0) 0
7 unassigned wm 0 0 (0/0/0) 0

Data Disk Encapsulation Process


Following is an example of the encapsulation process for disk c1t3d0.
This process uses the vxdiskadm utility to perform the encapsulation, as
follows:
1. Start the vxdiskadm utility from the shell command line while
logged in as the superuser. Type the following:
# vxdiskadm

Volume Manager Support Operations


Menu: VolumeManager/Disk

1 Add or initialize one or more disks


2 Encapsulate one or more disks
3 Remove a disk
4 Remove a disk for replacement
5 Replace a failed or removed disk
6 Mirror volumes on a disk
7 Move volumes from a disk
8 Enable access to (import) a disk group
9 Remove access to (deport) a disk group
10 Enable (online) a disk device
11 Disable (offline) a disk device
12 Mark a disk as a spare for a disk group
13 Turn off the spare flag on a disk
14 Unrelocate subdisks back to a disk
15 Exclude a disk from hot-relocation use

Sun Proprietary: Internal Use Only


2-8 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Encapsulating Data Disks

16 Make a disk available for hot-relocation use


17 Prevent multipathing/Suppress devices from VxVM's view
18 Allow multipathing/Unsuppress devices from VxVM's view
19 List currently suppressed/non-multipathed devices
20 Change the disk naming scheme
21 Get the newly connected/zoned disks in VxVM view
listList disk information

? Display help about menu


?? Display help about the menuing system
q Exit from menus
2. Select menu item 2, Encapsulate one or more disks, to begin the
encapsulation process. Type the following:
Select an operation to perform: 2

Encapsulate one or more disks


Menu: VolumeManager/Disk/Encapsulate

Use this operation to convert one or more disks to use the Volume Manager.
This adds the disks to a disk group and replaces existing partitions
with volumes. Disk encapsulation requires a reboot for the changes
to take effect.

More than one disk or pattern may be entered at the prompt. Here are
some disk selection examples:

all: all disks


c3 c4t2:all disks on both controller 3 and controller 4, target 2
c3t4d2:a single disk (in the c#t#d# naming scheme)
xyz_0 :a single disk (in the enclosure based naming scheme)
xyz_ :all disks on the enclosure whose name is xyz
3. Enter the disk to be encapsulated using c#t#d# notation, and answer
yes to the Continue operation? prompt. Type the following:
Select disk devices to encapsulate:
[<pattern-list>,all,list,q,?] c1t3d0

Here is the disk selected. Output format: [Device_Name]

c1t3d0

Continue operation? [y,n,q,?] (default: y) y

You can choose to add this disk to an existing disk group or to


a new disk group. To create a new disk group, select a disk group
name that does not yet exist.

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-9
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Encapsulating Data Disks

4. Enter the name of the disk group for this encapsulated disk to join.
In the following example the disk group name is storagedg.
Answer all prompts as appropriate for this particular encapsulation
procedure:
Which disk group [<group>,list,q,?] (default: rootdg) storagedg

There is no active disk group named storagedg.

Create a new group named storagedg? [y,n,q,?] (default: y) y

Use a default disk name for the disk? [y,n,q,?] (default: y) n

A new disk group will be created named storagedg and the selected
disks will be encapsulated and added to this disk group with
disk names that will be specified interactively.

c1t3d0

Continue with operation? [y,n,q,?] (default: y) y

The following disk has been selected for encapsulation.


Output format: [Device_Name]

c1t3d0

Continue with encapsulation? [y,n,q,?] (default: y) y


5. Select a name for the newly encapsulated disk. Although the name is
arbitrary, it is a good idea to follow some naming convention. In the
following example, the disk name is storage01. Answer the
remaining prompts as appropriate for this specific encapsulation
procedure:
Enter disk name for c1t3d0 [<name>,q,?] (default: storage01) storage01

A new disk group storagedg will be created and the disk device c1t3d0 will
be encapsulated and added to the disk group with the disk name storage01.

Use a default private region length for this disk?


[y,n,q,?] (default: y) y
The typical response to the prompt Use a default private
region length for this disk? is yes. If the size of the private
region in the selected disk group was modified, then answer no, and
follow the prompts to adjust the size.

Caution – The size of the private region of all member disks in a disk
group must be the same.

Sun Proprietary: Internal Use Only


2-10 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Encapsulating Data Disks

The c1t3d0 disk has been configured for encapsulation.

The first stage of encapsulation has completed successfully. You


should now reboot your system at the earliest possible opportunity.

The encapsulation will require two or three reboots which will happen
automatically after the next reboot. To reboot execute the command:

shutdown -g0 -y -i6

This will update the /etc/vfstab file so that volume devices are
used to mount the file systems on this disk device. You will need
to update any other references such as backup scripts, databases,
or manually created swap devices.

Encapsulate other disks? [y,n,q,?] (default: n) n


[H[2J
Volume Manager Support Operations
Menu: VolumeManager/Disk

1 Add or initialize one or more disks


2 Encapsulate one or more disks
3 Remove a disk
4 Remove a disk for replacement
5 Replace a failed or removed disk
6 Mirror volumes on a disk
7 Move volumes from a disk
8 Enable access to (import) a disk group
9 Remove access to (deport) a disk group
10 Enable (online) a disk device
11 Disable (offline) a disk device
12 Mark a disk as a spare for a disk group
13 Turn off the spare flag on a disk
14 Unrelocate subdisks back to a disk
15 Exclude a disk from hot-relocation use
16 Make a disk available for hot-relocation use
17 Prevent multipathing/Suppress devices from VxVM's view
18 Allow multipathing/Unsuppress devices from VxVM's view
19 List currently suppressed/non-multipathed devices
20 Change the disk naming scheme
21 Get the newly connected/zoned disks in VxVM view
listList disk information

? Display help about menu


?? Display help about the menuing system
q Exit from menus

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-11
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Encapsulating Data Disks

6. A reboot is necessary to finish the encapsulation process.

This example illustrates how the VxVM software views this disk prior to
completeion of the encapsulation procedure. After the encapsulation
process is completed, compare this example to the same information for
the encapsulated disk.
Select an operation to perform: list

List disk information


Menu: VolumeManager/Disk/ListDisk

Use this menu operation to display a list of disks. You can


also choose to list detailed information about the disk at
a specific disk device address.

Enter disk device or "all" [<address>,all,q,?] (default: all) all

DEVICE DISK GROUP STATUS


c1t0d0 rootdisk rootdg online
c1t1d0 - - error
c1t2d0 - - error
c1t3d0 - - error <--- Disk is in the error state.
c1t4d0 - - online
c1t5d0 - - online
c1t6d0 - - error
c1t16d0 - - error
c1t17d0 - - error
c1t18d0 - - error
c1t19d0 - - error
c1t20d0 - - online
c1t21d0 - - online
c1t22d0 rootmirror rootdg online

Device to list in detail [<address>,none,q,?] (default: none) c1t3d0

Device: c1t3d0s2
devicetag: c1t3d0
type: sliced
flags: online error private autoconfig
errno: Disk is not usable <--- Disk is not usable
Multipathing information:
numpaths: 2
c1t3d0s2state=enabled
c2t3d0s2state=enabled

List another disk device? [y,n,q,?] (default: n) n


[H[2J
Volume Manager Support Operations
Menu: VolumeManager/Disk

Sun Proprietary: Internal Use Only


2-12 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Encapsulating Data Disks

1 Add or initialize one or more disks


2 Encapsulate one or more disks
3 Remove a disk
4 Remove a disk for replacement
5 Replace a failed or removed disk
6 Mirror volumes on a disk
7 Move volumes from a disk
8 Enable access to (import) a disk group
9 Remove access to (deport) a disk group
10 Enable (online) a disk device
.
.
21 Get the newly connected/zoned disks in VxVM view
listList disk information

? Display help about menu


?? Display help about the menuing system
q Exit from menus

Select an operation to perform: q

Goodbye.

A system reboot is required before the disk is available for use by the
VxVM software.

Post-Encapsulation Configuration Data


The following are configuration data from example disk c1t3d0 after
encapsulation.

Post-Encapsulation /etc/vfstab File

In the following example of the /etc/vfstab file, the mount statements


for fs1 through fs4 are modified to use VxVM software volumes.
Additionally, the VxVM software annotated this file with information
about the disk’s prior state. This information is useful if the disk is
unencapsulated.
#device device mount FS fsck mount mount
#to mount to fsck point type pass at boot options
#
#/dev/dsk/c1d0s2 /dev/rdsk/c1d0s2 /usr ufs 1 yes -
fd - /dev/fd fd - no -
/proc - /proc proc - no -
/dev/vx/dsk/swapvol - - swap - no -
/dev/vx/dsk/rootvol /dev/vx/rdsk/rootvol / ufs 1 no logging
swap - /tmp tmpfs - yes -

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-13
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Encapsulating Data Disks

#
# Storage
#
/dev/vx/dsk/storagedg/fs1 /dev/vx/rdsk/storagedg/fs1 /fs1 ufs 1 yes logging
/dev/vx/dsk/storagedg/fs2 /dev/vx/rdsk/storagedg/fs2 /fs2 ufs 2 yes logging
/dev/vx/dsk/storagedg/fs3 /dev/vx/rdsk/storagedg/fs3 /fs3 ufs 3 yes logging
/dev/vx/dsk/storagedg/fs4 /dev/vx/rdsk/storagedg/fs4 /fs4 ufs 4 yes logging
#NOTE: volume rootvol (/) encapsulated partition c1t0d0s0
#NOTE: volume swapvol (swap) encapsulated partition c1t0d0s1
#NOTE: volume fs1 (/fs1) encapsulated partition c1t3d0s0
#NOTE: volume fs2 (/fs2) encapsulated partition c1t3d0s1
#NOTE: volume fs3 (/fs3) encapsulated partition c1t3d0s3
#NOTE: volume fs4 (/fs4) encapsulated partition c1t3d0s4

Post-Encapsulation /etc/vfstab.prevm File

A copy of the /etc/vfstab file before it was modified by the VxVM


software is saved as /etc/vfstab.prevm, as shown in the following
example.
#to mount to fsck point type pass at boot options
#
#/dev/dsk/c1d0s2 /dev/rdsk/c1d0s2 /usr ufs 1 yes -
fd - /dev/fd fd - no -
/proc - /proc proc - no -
/dev/dsk/c1t0d0s1 - - swap - no -
/dev/dsk/c1t0d0s0 /dev/rdsk/c1t0d0s0 / ufs 1 no logging
swap - /tmp tmpfs - yes -
#
# Storage
#
/dev/dsk/c1t3d0s0 /dev/rdsk/c1t3d0s0 /fs1 ufs 1 yes logging
/dev/dsk/c1t3d0s1 /dev/rdsk/c1t3d0s1 /fs2 ufs 2 yes logging
/dev/dsk/c1t3d0s3 /dev/rdsk/c1t3d0s3 /fs3 ufs 3 yes logging
/dev/dsk/c1t3d0s4 /dev/rdsk/c1t3d0s4 /fs4 ufs 4 yes logging

Post-Encapsulation df -k Command

The VxVM software volumes in the following example of the df -k


command are displayed as the file system for fs1 through fs4.
bash-2.03# df -k
Filesystem kbytes used avail capacity Mounted on
/dev/vx/dsk/rootvol 7670973 1646146 5948118 22% /
/proc 0 0 0 0% /proc
fd 0 0 0 0% /dev/fd
mnttab 0 0 0 0% /etc/mnttab
swap 1175152 16 1175136 1% /var/run
swap 1175160 24 1175136 1% /tmp
/dev/vx/dsk/storagedg/fs2
2055705 2073 1991961 1% /fs2

Sun Proprietary: Internal Use Only


2-14 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Encapsulating Data Disks

/dev/vx/dsk/storagedg/fs1
2055705 2073 1991961 1% /fs1
/dev/vx/dsk/storagedg/fs3
2055705 2073 1991961 1% /fs3
/dev/vx/dsk/storagedg/fs4
2055705 2073 1991961 1% /fs4

Post-Encapsulation prtvtoc Command

The following example of the prtvtoc command shows that the c1t3d0
partition table was modified to have only two partitions (partitions 6 and
7). Partition 6 is the public region and partition 7 is the private region.
This partitioning scheme is indicative of encapsulated disks. Even though
this disk partition table was modified, the data from the original four
partitions occupy the same blocks as they did prior to the encapsulation
process. The data was not moved or otherwise disturbed. The data
remains in this configuration unless the volumes are grown or converted
to a striped layout. This is an important distinction to remember if this
disk is ever unencapsulated.
bash-2.03# prtvtoc /dev/rdsk/c1t3d0s2
* /dev/rdsk/c1t3d0s2 partition map
*
* Dimensions:
* 512 bytes/sector
* 133 sectors/track
* 27 tracks/cylinder
* 3591 sectors/cylinder
* 4926 cylinders
* 4924 accessible cylinders
*
* Flags:
* 1: unmountable
* 10: read-only
*
* Unallocated space:
* First Sector Last
* Sector Count Sector
* 17682084 4294963705 17678492
*
* First Sector Last
* Partition Tag Flags Sector Count Sector Mount Directory
2 5 01 0 17682084 17682083
6 14 01 0 17682084 17682083
7 15 01 17678493 3591 17682083

The unencapsulation process for data disks is described in


‘‘Unencapsulating Data Disks’’ on page 2-23.

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-15
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Encapsulating Data Disks

Post-Encapsulation format Utility Partition Table

The following example of the format utility prints the new partition table
for c1t3d0. Notice that partition 6 overlaps the entire disk, including
partition 7, which is the private region.
partition> print
Current partition table (original):
Total disk cylinders available: 4924 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks


0 unassigned wm 0 0 (0/0/0) 0
1 unassigned wm 0 0 (0/0/0) 0
2 backup wu 0 - 4923 8.43GB (4924/0/0) 17682084
3 unassigned wm 0 0 (0/0/0) 0
4 unassigned wm 0 0 (0/0/0) 0
5 unassigned wm 0 0 (0/0/0) 0
6 - wu 0 - 4923 8.43GB (4924/0/0) 17682084
7 - wu 4923 - 4923 1.75MB (1/0/0) 3591

Post-Encapsulation vxdisk list Command

The vxdisk list command in the following example shows that c1t3d0
is under the management of the VxVM software.
bash-2.03# vxdisk list
DEVICE TYPE DISK GROUP STATUS
c1t0d0s2 sliced rootdisk rootdg online
c1t1d0s2 sliced - - error
c1t2d0s2 sliced - - error
c1t3d0s2 sliced storage01 storagedg online
c1t4d0s2 sliced - - online
c1t5d0s2 sliced - - online
c1t6d0s2 sliced - - error
c1t16d0s2 sliced - - error
c1t17d0s2 sliced - - error
c1t18d0s2 sliced - - error
c1t19d0s2 sliced - - error
c1t20d0s2 sliced - - online
c1t21d0s2 sliced - - online
c1t22d0s2 sliced rootmirror rootdg online

Sun Proprietary: Internal Use Only


2-16 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Encapsulating Data Disks

Post-Encapsulation vxdisk list c1t3d0 Command

This example of the vxdisk list command provides detailed


information about the encapsulated disk. Prior to the encapsulation, the
VxVM software was unaware of this disk.
bash-2.03# vxdisk list c1t3d0
Device: c1t3d0s2
devicetag: c1t3d0
type: sliced
hostid: lowtide
disk: name=storage01 id=1019688339.1134.lowtide
group: name=storagedg id=1019688341.1137.lowtide
info: privoffset=1
flags: online ready private autoconfig autoimport imported
pubpaths: block=/dev/vx/dmp/c1t3d0s6 char=/dev/vx/rdmp/c1t3d0s6
privpaths: block=/dev/vx/dmp/c1t3d0s7 char=/dev/vx/rdmp/c1t3d0s7
version: 2.2
iosize: min=512 (bytes) max=2048 (blocks)
public: slice=6 offset=1 len=17678493
private: slice=7 offset=1 len=3590
update: time=1019688342 seqno=0.5
headers: 0 248
configs: count=1 len=2630
logs: count=1 len=398
Defined regions:
config priv 000017-000247[000231]: copy=01 offset=000000 enabled
config priv 000249-002647[002399]: copy=01 offset=000231 enabled
log priv 002648-003045[000398]: copy=01 offset=000000 enabled
Multipathing information:
numpaths: 2
c1t3d0s2 state=enabled
c2t3d0s2 state=enabled

Post-Encapsulation vxprint -ht -g storagedg Command

The vxprint utility in this example displays detailed information about


the encapsulated partitions and their associated VxVM software objects.
bash-2.03# vxprint -ht -g storagedg
DG NAME NCONFIG NLOG MINORS GROUP-ID
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
V NAME RVG KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO

dg storagedg default default 127000 1019688341.1137.lowtide

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-17
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Encapsulating Data Disks

dm storage01 c1t3d0s2 sliced 3590 17678493 -

v fs1 - ENABLED ACTIVE 4197879 ROUND - gen


pl fs1-01 fs1 ENABLED ACTIVE 4197879 CONCAT - RW
sd storage01-B0 fs1-01 storage01 17678492 1 0 c1t3d0 ENA
sd storage01-04 fs1-01 storage01 0 4197878 1 c1t3d0 ENA

v fs2 - ENABLED ACTIVE 4197879 ROUND - gen


pl fs2-01 fs2 ENABLED ACTIVE 4197879 CONCAT - RW
sd storage01-03 fs2-01 storage01 4197878 4197879 0 c1t3d0 ENA

v fs3 - ENABLED ACTIVE 4197879 ROUND - gen


pl fs3-01 fs3 ENABLED ACTIVE 4197879 CONCAT - RW
sd storage01-02 fs3-01 storage01 8395757 4197879 0 c1t3d0 ENA

v fs4 - ENABLED ACTIVE 4197879 ROUND - gen


pl fs4-01 fs4 ENABLED ACTIVE 4197879 CONCAT - RW
sd storage01-01 fs4-01 storage01 12593636 4197879 0 c1t3d0 ENA

Related Post-Encapsulation Files


When a disk is encapsulated, a subdirectory under
/etc/vx/reconfig.d/disk.d is created which contains files that hold
information related to the encapsulation. The following is an example of
files that relate to the encapsulation of c1t3d0.

The File List

A directory list of files specific to c1t3d0 is shown in the following


example.
bash-2.03# pwd
/etc/vx/reconfig.d/disk.d/c1t3d0
bash-2.03# ls -las
total 16
2 drwxr-xr-x 2 root other 512 Apr 24 15:29 .
2 drwxr-xr-x 4 root other 512 Apr 24 15:29 ..
2 -rw-r--r-- 1 root other 10 Apr 24 15:29 dg
2 -rw-r--r-- 1 root other 10 Apr 24 15:29 dmname
4 -rw-r--r-- 1 root other 1156 Apr 24 15:29 newpart
2 -rw-r--r-- 1 root other 7 Apr 24 15:29 primary_node
2 -rw-r--r-- 1 root other 476 Apr 24 15:29 vtoc

The dg File

The dg file in the following example delineates the disk group of which
this encapsulated disk is a member.

Sun Proprietary: Internal Use Only


2-18 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Encapsulating Data Disks

bash-2.03# more *

::::::::::::::
dg
::::::::::::::
storagedg
::::::::::::::

The dmname File

The dmname file for the following example lists the VxVM software disk
name of the encapsulated disk.
::::::::::::::
dmname
::::::::::::::
storage01
::::::::::::::

The newpart File

The following example of the newpart file displays the new partition
information for this encapsulated disk. Additionally, it lists all the VxVM
software commands executed to successfully complete the encapsulation
process.
::::::::::::::
newpart
::::::::::::::
# volume manager partitioning for drive c1t3d0
0 0x0 0x000 0 0
1 0x0 0x000 0 0
2 0x5 0x201 0 17682084
3 0x0 0x000 0 0
4 0x0 0x000 0 0
5 0x0 0x000 0 0
6 0xe 0x201 0 17682084
7 0xf 0x201 17678493 3591
#vxmake vol fs1 plex=fs1-%%00 usetype=gen
#vxmake plex fs1-%%00 sd=storage01-B0,storage01-%%00
#vxmake sd storage01-%%00 disk=storage01 offset=0 len=4197878
#vxmake sd storage01-B0 disk=storage01 offset=17678492 len=1 putil0=Block0
comment="Remap of block 0
#vxvol start fs1
#rename c1t3d0s0 fs1
#vxmake vol fs2 plex=fs2-%%01 usetype=gen
#vxmake plex fs2-%%01 sd=storage01-%%01
#vxmake sd storage01-%%01 disk=storage01 offset=4197878 len=4197879
#vxvol start fs2
#rename c1t3d0s1 fs2
#vxmake vol fs3 plex=fs3-%%02 usetype=gen

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-19
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Encapsulating Data Disks

#vxmake plex fs3-%%02 sd=storage01-%%02


#vxmake sd storage01-%%02 disk=storage01 offset=8395757 len=4197879
#vxvol start fs3
#rename c1t3d0s3 fs3
#vxmake vol fs4 plex=fs4-%%03 usetype=gen
#vxmake plex fs4-%%03 sd=storage01-%%03
#vxmake sd storage01-%%03 disk=storage01 offset=12593636 len=4197879
#vxvol start fs4
#rename c1t3d0s4 fs4

The primary_node File

The primary_node file lists the original c#t#d# name of the encapsulated
disk in the following example.
::::::::::::::
primary_node
::::::::::::::
c1t3d0
::::::::::::::

The vtoc File

The following example of the vtoc file saves the original partition table of
the encapsulated disk. In earlier versions of the VxVM software, this file
was modified and used with the /etc/fmthard command to re-partition
disks during the unencapsulation process. The vxmksdpart command
now performs this function.
::::::::::::::
vtoc
::::::::::::::
#THE PARTITIONING OF /dev/rdsk/c1t3d0s2 IS AS FOLLOWS:
#SLICE TAG FLAGS START SIZE
0 0x0 0x200 0 4197879
1 0x0 0x200 4197879 4197879
2 0x5 0x201 0 17682084
3 0x0 0x200 8395758 4197879
4 0x0 0x200 12593637 4197879
5 0x0 0x000 0 0
6 0x0 0x000 0 0
7 0x0 0x000 0 0

Encapsulating a Non-Conforming Disk


To encapsulate a disk into the VxVM software, the disk is required to have
enough free space for the VxVM software to write a private region to the
disk. The private region is generally smaller than 2 megabytes.

Sun Proprietary: Internal Use Only


2-20 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Encapsulating Data Disks

If there is not enough free space on the disk and additional free space
cannot be configured, the situation can be alleviated by temporarily
encapsulating the non-conforming disk without a private region, then
mirroring the encapsulated disk volumes to another disk. Once the data is
mirrored to a real VxVM software disk, the encapsulated plexes are
detached, leaving the data on the mirror. At that point, mirror the mirror
disk to another VxVM software disk to complete the operation.

Only use this procedure if the data on a non-conforming disk must be


brought under the VxVM software control.

The following procedure is based on SunSolve INFODOC 16051. Refer to


Appendix A, “SunSolve INFODOCs,” for a complete copy of this
document. Check the SunSolve Online Web site to determine if the
INFODOC has been updated.

To encapsulate a non-conforming disk:


1. For each slice on the disk (excluding slice 2), run the following
command to prepare the disk for encapsulation without a private
region. In this example, only slices 5 and 6 have data on them. Type
the following:
vxdisk define c#t#d#s5 type=nopriv
vxdisk define c#t#d#s6 type=nopriv
2. Use the vxdg -g command to add each of the slices of the target disk
to a disk group as though each slice is a disk, assigning the slice a
name. In this example the names NPdisk05 and NPdisk06 are used,
as follows:
vxdg -g <diskgroup> adddisk NPdisk05=c#t#d#s5
vxdg -g <diskgroup> adddisk NPdisk06=c#t#d#s6
3. On each of the new disks, create a simple volume (not a file system)
that spans the entire disk. Then use the vxdisk list command to
determine the maximum size for the volumes to be created:
vxdisk list NPdisk05 | grep public
public: slice=0 offset=0 len=8196096
vxdisk list NPdisk06 | grep public
public: slice=0 offset=0 len=9400320
4. Use the information derived from the vxdisk list command len
value and the vxassist command to create the volumes. The
following example names the volumes NPdisk05vol and
NPdisk06vol.
vxassist -g <diskgroup> make NPdisk05vol 8196096 layout=nostripe alloc="NPdisk05"
vxassist -g <diskgroup> make NPdisk06vol 9400320 layout=nostripe alloc="NPdisk06"

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-21
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Encapsulating Data Disks

5. Mirror the volumes to a disk that has enough space to mirror both
volumes. In the following example, the volumes are mirrored to a
disk named disk01.
vxassist -g <diskgroup> mirror NPdisk05vol layout=nostripe alloc="disk01"
vxassist -g <diskgroup> mirror NPdisk06vol layout=nostripe alloc="disk01"
6. When the mirroring process is complete, remove the original side of
the mirror. This removes the disk that does not have a private region
configured. Type the following:
vxplex -g <diskgroup> -o rm dis NPdisk05vol-01
vxplex -g <diskgroup> -o rm dis NPdisk06vol-01
7. Remove the old disks from the disk group and return them to their
original state:
vxdg -g <diskgroup> rmdisk NPdisk05
vxdg -g <diskgroup> rmdisk NPdisk06
vxdisk rm c0t5d10s5 vxdisk rm c0t5d10s6

This leaves two concatenated volumes named NPdisk05vol and


NPdisk06vol. These volumes contain the data that was originally located
on c#t#d#s5 and c#t#d#s6.

Sun Proprietary: Internal Use Only


2-22 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Unencapsulating Data Disks

Unencapsulating Data Disks


After a data disk is encapsulated there is no easy method to reverse the
process. From a VxVM software point of view, once a disk is
encapsulated, there is no reason to reverse the process. Additionally, if the
encapsulated volume is grown or converted to a stripe volume,
unencapsulation is impossible. The only method to reverse the
encapsulation process after a volume is modified in this way is to back up
the data and restore it to a non-VxVM software-managed disk.

Caution – Be sure to back up all data on the encapsulated disk prior to


performing this procedure.

The following example of the unencapsulation process uses an


encapsulated disk that is a member of the storagedg disk group with a
physical address of c1t3d0.
1. Make sure that no volumes from the encapsulated disk were grown
or had their basic structure modified. Use the vxprint command
and the format utility to verify format and condition of the
encapsulated volumes, as follows:
bash-2.03# vxprint -g storagedg -ht
DG NAME NCONFIG NLOG MINORS GROUP-ID
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
V NAME RVG KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO

dg storagedg default default 127000 1019688341.1137.lowtide

dm storage01 c1t3d0s2 sliced 3590 17678493 -


dm storage02 c1t4d0s2 sliced 3590 17674902 -

v fs1 - ENABLED ACTIVE 4197879 ROUND - gen


pl fs1-01 fs1 ENABLED ACTIVE 4197879 CONCAT - RW
sd storage01-B0 fs1-01 storage01 17678492 1 0 c1t3d0 ENA
sd storage01-04 fs1-01 storage01 0 4197878 1 c1t3d0 ENA
pl fs1-02 fs1 ENABLED ACTIVE 4197879 CONCAT - RW
sd storage02-01 fs1-02 storage02 0 4197879 0 c1t4d0 ENA

v fs2 - ENABLED ACTIVE 4197879 ROUND - gen


pl fs2-01 fs2 ENABLED ACTIVE 4197879 CONCAT - RW
sd storage01-03 fs2-01 storage01 4197878 4197879 0 c1t3d0 ENA

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-23
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Unencapsulating Data Disks

pl fs2-02 fs2 ENABLED ACTIVE 4197879 CONCAT - RW


sd storage02-02 fs2-02 storage02 4197879 4197879 0 c1t4d0 ENA

v fs3 - ENABLED ACTIVE 4197879 ROUND - gen


pl fs3-01 fs3 ENABLED ACTIVE 4197879 CONCAT - RW
sd storage01-02 fs3-01 storage01 8395757 4197879 0 c1t3d0 ENA
pl fs3-02 fs3 ENABLED ACTIVE 4197879 CONCAT - RW
sd storage02-03 fs3-02 storage02 8395758 4197879 0 c1t4d0 ENA

v fs4 - ENABLED ACTIVE 4197879 ROUND - gen


pl fs4-01 fs4 ENABLED ACTIVE 4197879 CONCAT - RW
sd storage01-01 fs4-01 storage01 12593636 4197879 0 c1t3d0 ENA
pl fs4-02 fs4 ENABLED ACTIVE 4197879 CONCAT - RW
sd storage02-04 fs4-02 storage02 12593637 4197879 0 c1t4d0 ENA
bash-2.03#
Note that all volumes on c1t3d0 (fs1–fs4) have a value of
LAYOUT=CONCAT, which indicates that the volumes were not grown
or converted to a striped volume. This means that the
unencapsulation process should work on this disk.
Each volume has two plexes, fs#-01, the primary, and fs#-02, the
mirror. This is verified by printing the partition table of each disk.
Additionally, the B0 subdisk is used as a safety mechanism to protect
the bootblock of encapsulated disks. It can be safely ignored during
this process.
The partition tables for these example disks are as follows –
● Encapsulated disk – The following capture is of disk c1t3d0.
Notice that it has slicing indicative of an encapsulated disk.
This is the target disk for the unencapsulation example.
partition> print
Current partition table (original):
Total disk cylinders available: 4924 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks


0 unassigned wm 0 0 (0/0/0) 0
1 unassigned wm 0 0 (0/0/0) 0
2 backup wu 0 - 4923 8.43GB (4924/0/0) 17682084
3 unassigned wm 0 0 (0/0/0) 0
4 unassigned wm 0 0 (0/0/0) 0
5 unassigned wm 0 0 (0/0/0) 0
6 - wu 0 - 4923 8.43GB (4924/0/0) 17682084 <-- Public
7 - wu 4923 - 4923 1.75MB (1/0/0) 3591 <-- Private
● Mirror disk – This is a partition print of disk c1t4d0. It is the
mirror of c1t3d0 and has the partitioning scheme of a VxVM
software disk. This is the disk from which the plexes are
detached (unmirrored) during the unencapsulation process.

Sun Proprietary: Internal Use Only


2-24 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Unencapsulating Data Disks

partition> print
Current partition table (original):
Total disk cylinders available: 4924 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks


0 unassigned wm 0 0 (0/0/0) 0
1 unassigned wm 0 0 (0/0/0) 0
2 backup wu 0 - 4923 8.43GB (4924/0/0) 17682084
3 - wu 0 - 0 1.75MB (1/0/0) 3591 <-- Private
4 - wu 2 - 4923 8.43GB (4922/0/0) 17674902 <-- Public
5 unassigned wm 0 0 (0/0/0) 0
6 unassigned wm 0 0 (0/0/0) 0
7 unassigned wm 0 0 (0/0/0) 0
2. Unmirror all volumes to be unencapsulated, using either the
vxassist or vxplex command. The following example uses the
vxplex command:
bash-2.03# vxplex -o rm dis fs1-02
bash-2.03# vxplex -o rm dis fs2-02
bash-2.03# vxplex -o rm dis fs3-02
bash-2.03# vxplex -o rm dis fs4-02

Power user – Alternatively, for disks with multiple mirrors, the vxplex
command can be looped:
bash-2.03# for i in 1 2 3 4
> do
> vxplex -o rm dis fs$i-02
> done
3. Use the vxprint command to verify that all mirrors are detached:
bash-2.03# vxprint -g storagedg -ht
DG NAME NCONFIG NLOG MINORS GROUP-ID
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
V NAME RVG KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO

dg storagedg default default 127000 1019688341.1137.lowtide

dm storage01 c1t3d0s2 sliced 3590 17678493 -


dm storage02 c1t4d0s2 sliced 3590 17674902 -

v fs1 - ENABLED ACTIVE 4197879 ROUND - gen


pl fs1-01 fs1 ENABLED ACTIVE 4197879 CONCAT - RW
sd storage01-B0 fs1-01 storage01 17678492 1 0 c1t3d0 ENA
sd storage01-04 fs1-01 storage01 0 4197878 1 c1t3d0 ENA

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-25
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Unencapsulating Data Disks

v fs2 - ENABLED ACTIVE 4197879 ROUND - gen


pl fs2-01 fs2 ENABLED ACTIVE 4197879 CONCAT - RW
sd storage01-03 fs2-01 storage01 4197878 4197879 0 c1t3d0 ENA

v fs3 - ENABLED ACTIVE 4197879 ROUND - gen


pl fs3-01 fs3 ENABLED ACTIVE 4197879 CONCAT - RW
sd storage01-02 fs3-01 storage01 8395757 4197879 0 c1t3d0 ENA

v fs4 - ENABLED ACTIVE 4197879 ROUND - gen


pl fs4-01 fs4 ENABLED ACTIVE 4197879 CONCAT - RW
sd storage01-01 fs4-01 storage01 12593636 4197879 0 c1t3d0 ENA
Notice that the -02 plexes are missing from the vxprint output for
these volumes. This is a clear indication that the mirrors no longer
exist. Also, note the subdisk numbers for the surviving plex of each
volume. This information is used in the next step of the
unencapsulation process.
4. For normal partitions (non-private), recreate the original partitions on
the disk by running the vxmksdpart command on each subdisk, as
shown in this example.

Caution – The following steps must be performed exactly as shown or


loss of data may occur. It is very important that a full understanding of
the pre-encapsulation partitioned layout of the disk is known, or the post-
encapsulation partition layout could be incorrect.

To make sure that the mapping of subdisks to encapsulated


partitions is correct, cross-reference the output of the vxprint
command with the commented lines of the /etc/vfstab file. These
comments were added to the file by the VxVM software during the
encapsulation process. This method provides a one-for-one match
between subdisks and encapsulated partitions.
bash-2.03# /etc/vx/bin/vxmksdpart -g storagedg storage01-04 0 0x00 0x00

partition> print
Current partition table (original):
Total disk cylinders available: 4924 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks


0 unassigned wm 0 - 1168 2.00GB (1169/0/0) 4197879
1 unassigned wm 0 0 (0/0/0) 0
2 backup wu 0 - 4923 8.43GB (4924/0/0) 17682084
3 unassigned wm 0 0 (0/0/0) 0
4 unassigned wm 0 0 (0/0/0) 0
5 unassigned wm 0 0 (0/0/0) 0
6 - wu 0 - 4923 8.43GB (4924/0/0) 17682084
7 - wu 4923 - 4923 1.75MB (1/0/0) 3591

Sun Proprietary: Internal Use Only


2-26 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Unencapsulating Data Disks

Notice that after execution of the vxmksdpart command for the


storage01-04 subdisk, the partition table of c1t3d0 is modified to
reflect the original sliced location of that subdisk’s data.
The process continues by creating partitions for the remaining
subdisks.
bash-2.03# /etc/vx/bin/vxmksdpart -g storagedg storage01-03 1 0x00 0x00
bash-2.03# /etc/vx/bin/vxmksdpart -g storagedg storage01-02 3 0x00 0x00
bash-2.03# /etc/vx/bin/vxmksdpart -g storagedg storage01-01 4 0x00 0x00
When the process is completed, a printout of the partition table for
the c1t3d0 disk shows that all the original partitions were recreated
and in the correct locations.
partition> print
Current partition table (original):
Total disk cylinders available: 4924 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks


0 unassigned wm 0 - 1168 2.00GB (1169/0/0) 4197879
1 unassigned wm 1169 - 2337 2.00GB (1169/0/0) 4197879
2 backup wu 0 - 4923 8.43GB (4924/0/0) 17682084
3 unassigned wm 2338 - 3506 2.00GB (1169/0/0) 4197879
4 unassigned wm 3507 - 4675 2.00GB (1169/0/0) 4197879
5 unassigned wm 0 0 (0/0/0) 0
6 - wu 0 - 4923 8.43GB (4924/0/0) 17682084
7 - wu 4923 - 4923 1.75MB (1/0/0) 3591
5. Unmount the volumes. Type the following:
bash-2.03# df -k
Filesystem kbytes used avail capacity Mounted on
/dev/vx/dsk/rootvol 7670973 1646148 5948116 22% /
/proc 0 0 0 0% /proc
fd 0 0 0 0% /dev/fd
mnttab 0 0 0 0% /etc/mnttab
swap 1174480 16 1174464 1% /var/run
swap 1174496 32 1174464 1% /tmp
/dev/vx/dsk/storagedg/fs2
2055705 2073 1991961 1% /fs2
/dev/vx/dsk/storagedg/fs1
2055705 2073 1991961 1% /fs1
/dev/vx/dsk/storagedg/fs3
2055705 2073 1991961 1% /fs3
/dev/vx/dsk/storagedg/fs4
2055705 2073 1991961 1% /fs4

bash-2.03# umount /fs1


bash-2.03# umount /fs2
bash-2.03# umount /fs3
bash-2.03# umount /fs4

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-27
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Unencapsulating Data Disks

When this step is completed, all volumes hosted by this disk are
unmounted, as shown in the following example:
bash-2.03# df -k
Filesystem kbytes used avail capacity Mounted on
/dev/vx/dsk/rootvol 7670973 1646148 5948116 22% /
/proc 0 0 0 0% /proc
fd 0 0 0 0% /dev/fd
mnttab 0 0 0 0% /etc/mnttab
swap 1176080 16 1176064 1% /var/run
swap 1176096 32 1176064 1% /tmp
6. Stop all applications using data on these volumes prior to executing
the remaining steps.
7. Edit the /etc/vfstab file by changing the mount statements to
reflect partitions instead of volumes:
#device device mount FS fsck mount mount
#to mount to fsck point type pass at boot options
#
#/dev/dsk/c1d0s2 /dev/rdsk/c1d0s2 /usr ufs 1 yes -
fd - /dev/fd fd - no -
/proc - /proc proc - no -
/dev/vx/dsk/swapvol - - swap - no -
/dev/vx/dsk/rootvol /dev/vx/rdsk/rootvol / ufs 1 no logging
swap - /tmp tmpfs - yes -
#
# Storage
#
/dev/dsk/c1t3d0s0 /dev/rdsk/c1t3d0s0 /fs1 ufs 1 yes logging
/dev/dsk/c1t3d0s1 /dev/rdsk/c1t3d0s1 /fs2 ufs 1 yes logging
/dev/dsk/c1t3d0s3 /dev/rdsk/c1t3d0s3 /fs3 ufs 1 yes logging
/dev/dsk/c1t3d0s4 /dev/rdsk/c1t3d0s4 /fs4 ufs 1 yes logging
#NOTE: volume rootvol (/) encapsulated partition c1t0d0s0
#NOTE: volume swapvol (swap) encapsulated partition c1t0d0s1
8. Mount the partitions. Use the mountall command:
bash-2.03# mountall

bash-2.03# df -k
Filesystem kbytes used avail capacity Mounted on
/dev/vx/dsk/rootvol 7670973 1646148 5948116 22% /
/proc 0 0 0 0% /proc
fd 0 0 0 0% /dev/fd
mnttab 0 0 0 0% /etc/mnttab
swap 1174552 16 1174536 1% /var/run
swap 1174568 32 1174536 1% /tmp
/dev/dsk/c1t3d0s1 2055705 2073 1991961 1% /fs2
/dev/dsk/c1t3d0s3 2055705 2073 1991961 1% /fs3
/dev/dsk/c1t3d0s4 2055705 2073 1991961 1% /fs4
/dev/dsk/c1t3d0s0 2055705 2073 1991961 1% /fs1
It is now safe to start applications that use data contained on these
partitions.

Sun Proprietary: Internal Use Only


2-28 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Unencapsulating Data Disks

9. Remove each volume (recursively), as shown in the following


example:
bash-2.03# vxedit -r -f rm fs1
bash-2.03# vxedit -r -f rm fs2
bash-2.03# vxedit -r -f rm fs3
bash-2.03# vxedit -r -f rm fs4

bash-2.03# vxprint -g storagedg -ht


DG NAME NCONFIG NLOG MINORS GROUP-ID
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
V NAME RVG KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO

dg storagedg default default 127000 1019688341.1137.lowtide

dm storage01 c1t3d0s2 sliced 3590 17678493 -


dm storage02 c1t4d0s2 sliced 3590 17674902 -
Notice that all volumes are removed.
10. Remove the disk from the disk group. Type the following:
bash-2.03# vxdg rmdisk storage01

bash-2.03# vxdisk rm c1t3d0

bash-2.03# vxdisk list


DEVICE TYPE DISK GROUP STATUS
c1t0d0s2 sliced rootdisk rootdg online
c1t1d0s2 sliced - - error
c1t2d0s2 sliced - - error
c1t4d0s2 sliced storage02 storagedg online
c1t5d0s2 sliced - - online
c1t6d0s2 sliced - - error
c1t16d0s2 sliced - - error
c1t17d0s2 sliced - - error
c1t18d0s2 sliced - - error
c1t19d0s2 sliced - - error
c1t20d0s2 sliced - - online
c1t21d0s2 sliced - - online
c1t22d0s2 sliced rootmirror rootdg online
The vxdisk list output shows that c1t3d0 was removed from
VxVM software control.

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-29
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Unencapsulating Data Disks

11. Remove the public and private partitions using the format utility.
Type the following:
partition> print
Current partition table (unnamed):
Total disk cylinders available: 4924 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks


0 unassigned wm 0 - 1168 2.00GB (1169/0/0) 4197879
1 unassigned wm 1169 - 2337 2.00GB (1169/0/0) 4197879
2 backup wu 0 - 4923 8.43GB (4924/0/0) 17682084
3 unassigned wm 2338 - 3506 2.00GB (1169/0/0) 4197879
4 unassigned wm 3507 - 4675 2.00GB (1169/0/0) 4197879
5 unassigned wm 0 0 (0/0/0) 0
6 unassigned wu 0 0 (0/0/0) 0
7 unassigned wu 0 0 (0/0/0) 0

The unencapsulation process is now complete.

Sun Proprietary: Internal Use Only


2-30 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Encapsulating Boot Disks

Encapsulating Boot Disks


The VxVM software is able to bring system partitions, such as root, usr,
var, opt, or home, under management. This is called rootability. Rootability
is accomplished through encapsulation. The encapsulation procedure for
system partitions is similar to the procedure used for data (non-root) disks
discussed in the section ‘‘Encapsulating Data Disks’’ on page 2-5.

Encapsulation of system partitions is the process of converting standard


UNIX® file system (UFS) partitions to VxVM software volumes. Once
converted to VxVM software volumes, the encapsulated volumes can be
mirrored to increase the fault resiliency of the system areas of a server’s
file system. A volume that is configured as a swap area is called a swap
volume. A volume that is configured as a root file system is called a root
volume.

Figure 2-2 illustrates partition mapping for an encapsulated boot disk


with two system partitions.

Boot Disk Encapsulated Boot Disk

rootvol
Overlay Partition- / (0) Slice 2
Slice 0 - /
and
Slice 2
Slice 3
Overlay Partition -
swap (1)
Public Region
Slice 1 - swap
swapvol
Slice 4 Private Region
Free Space

Mirror Disk

rootvol
Slice 4 Private Region

Slice 3 Public Overlay Partition - / (0)


Slice 2

Region

Overlay Partition -
swap (1) swapvol

Figure 2-2 Two-Slice Boot Disk

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-31
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Encapsulating Boot Disks

Partition mapping for an encapsulated boot disk with four system


partitions is shown in Figure 2-3.

Boot Disk Encapsulated Boot Disk

rootvol
Slice 0 - / Overlay Partition - /(0) swapvol
Overlay Partition - swap(1) Slice 2
Slice 1 - swap and Slice 6
Slice 2
Overlay Partition - /usr(3) Public
Slice 3 - /usr
Overlay Partition - /var(4)
Region

Slice 4 - /var
Free Space Slice 7 Private Region

Mirror Disk

rootvol
Slice 3 Private Region
swapvol

Overlay Partition -/ (0)


Slice 2
Slice 4 Overlay Partition - swap (1)
Public Region
Overlay Partition - /usr (6)

Overlay Partition - /var (7)

Figure 2-3 Four-Slice Boot Disk

Encapsulating the boot disk brings increased fault resiliency to a server


but also brings with it additional system management issues that must be
addressed by system administrators. These issues are addressed in this
section on boot disk encapsulation.

Booting root Volumes


During boot processing, the root and swap volume areas must be available
prior to the loading of the vxconfigd daemon. This means that these
areas of an encapsulated boot disk must be accessible from contiguous
blocks of space as ordinary partitions. Due to this restriction, root and
swap volumes cannot be striped, concatenated, or spanned. Also, the
mirrors (plexes) of these volumes cannot be striped, concatenated, or
spanned.

Sun Proprietary: Internal Use Only


2-32 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Encapsulating Boot Disks

Volume Restrictions
The rootvol, swapvol, and usr volumes have restrictions that other
encapsulated volumes do not have. These restrictions are:
● The root volume rootvol must be a member of the rootdg disk
group.
● The rootvol, swapvol, and usr volumes must have the following
specific minor numbers:
● rootvol = 0
● swapvol = 2
● usr = 3
● The rootvol, swapvol, usr, and var volumes use restricted mirrors
that have overlay partitions created for them. Overlay partitions
occupy the same disk blocks as the restricted mirror and are used to
boot the system prior to the availability of the vxconfigd daemon.
● The rootvol, swapvol, usr, var, and opt volumes cannot be grown,
spanned or occupy a plex that has multiple non-contiguous
subdisks. All data associated with encapsulated system partitions
must reside in contiguous blocks of space.
● When mirroring the boot disk, the mirror disk must be large enough
to hold all plexes on that disk or mirroring fails for one or more
volumes on the encapsulated boot disk.
● The rootvol, swapvol, and usr volumes cannot have DRL attached
to their volumes.

Boot Disk Encapsulation Process


There are two methods which can be used to encapsulate a boot disk. One
method performs the encapsulation during the vxinstall process. The
second method performs the encapsulation process after the VxVM
software is installed. This requires the use of either the vxdiskadm
(preferred) or vxdiskadd, vxmake, and vxvol utilities. This section uses
the vxdiskadm utility for all encapsulation and mirroring example
procedures.

Appendix C, “Example Five-Slice Boot Disk Encapsulation,” provides a


detailed example of the procedures used to encapsulate a boot disk.

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-33
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Encapsulating Boot Disks

Note – The following processes and procedures are standard VERITAS


Software Corporation recommendations for encapsulating and mirroring
boot disks. Sun best practices for boot disk management are discussed in
‘‘Examining Sun Enterprise Services Best Practices for VxVM Software-
Managed Boot Disks’’ on page 2-48.

A detailed boot disk encapsulation example is not presented in this


section due to the similarity of that process to the encapsulation process
for data disks.

Encapsulating During the VxVM Software Installation

To encapsulate the boot disk during VxVM software installation,


answering yes to the encapsulation prompt during the installation script
process. Once vxinstall is finished, the system is rebooted twice. The
first reboot performs the encapsulation. The second reboot boots the
system using the encapsulated system volumes.

Encapsulating Using the vxdiskadm Utility

Encapsulating the boot disk using the vxdiskadm command requires that
a minimum of one disk be configured into rootdg during the vxinstall
processing. This disk is initalized for use as the mirror for the
encapsulated boot disk. Boot disk encapsulation using the vxdiskadm
command is identical to the process outlined for data disks described in
‘‘Data Disk Encapsulation Process’’ on page 2-8. The difference between
the two processes is in the number of reboots. Data disk encapsulation
requires a single reboot, while boot disk encapsulation performs multiple
reboots.

Note – For more information on boot disk encapsulation see the


following: SunSolve INFODOCs 15838, 19245, 13781, 13775, and 24663,
and SRDBs 19245.

Sun Proprietary: Internal Use Only


2-34 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Encapsulating Boot Disks

Pre-Encapsulation Configuration Data


The following section displays disk partition tables and files associated
with non-VxVM software management of a system boot disk and
associated file systems. This example set depicts a two-slice boot disk.

These examples were captured prior to the installation of the VxVM


software and encapsulation of the boot disk. This system has a boot disk
that is configured to use two slices, / and swap.

Pre-Encapsulation /etc/vfstab File

The following example shows a standard /etc/vfstab file prior to the


installation of the VxVM software and encapsulation of the boot disk.
Notice that the boot disk is configured to use two system partitions, /
and swap.
#device device mount FS fsck mount mount
#to mount to fsck point type pass at boot options
#
#/dev/dsk/c1d0s2 /dev/rdsk/c1d0s2 /usr ufs 1 yes -
fd - /dev/fd fd - no -
/proc - /proc proc - no -
/dev/dsk/c1t0d0s1 - - swap - no -
/dev/dsk/c1t0d0s0 /dev/rdsk/c1t0d0s0 / ufs 1 no logging
swap - /tmp tmpfs - yes -

Pre-Encapsulation df -k Command

A df -k command delineates the example system’s mounted file systems.


At this time, this server does not have any application or data file system
configured.
bash-2.03# df -k
Filesystem kbytes used avail capacity Mounted on
/dev/dsk/c1t0d0s0 7670973 1536449 6057815 21% /
/proc 0 0 0 0% /proc
fd 0 0 0 0% /dev/fd
mnttab 0 0 0 0% /etc/mnttab
swap 1209520 16 1209504 1% /var/run
swap 1209520 16 1209504 1% /tmp

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-35
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Encapsulating Boot Disks

Pre-Encapsulation prtvtoc Command

The prtvtoc command output of the example system boot disk shows
the disk having two partitions and no free unpartitioned space. Boot
disks, unlike data disks, do not require any unpartitioned free space for
encapsulation. In this situation, space for the private region is taken from
the swap partition.

This is a less than ideal configuration for encapsulation. The disk


encapsulates, but it will probably fail mirroring if the mirror disk is the
exact same size as this encapsulated disk. This is because the mirror disk
is an initialized VxVM software disk with 2048 bytes or greater of space
consumed for the private region. This leaves a public region with
insufficient space to mirror both the rootvol and swapvol volumes.
bash-2.03# prtvtoc /dev/rdsk/c1t0d0s2
* /dev/rdsk/c1t0d0s2 partition map
*
* Dimensions:
* 512 bytes/sector
* 133 sectors/track
* 27 tracks/cylinder
* 3591 sectors/cylinder
* 4926 cylinders
* 4924 accessible cylinders
*
* Flags:
* 1: unmountable
* 10: read-only
*
* First Sector Last
* Partition Tag Flags Sector Count Sector Mount Directory
0 2 00 0 15581349 15581348 /
1 3 01 15581349 2100735 17682083
2 5 00 0 17682084 17682083

In this example, the space problem was rectified by reducing the size of
the swap partition, leaving sufficient unpartitioned free space for use by
the encapsulation process. Additional swap space can be configured using
swap files to supplement the reduced swap partition size. The corrected
partition map now shows unallocated space.
bash-2.03# prtvtoc /dev/rdsk/c1t0d0s2
* /dev/rdsk/c1t0d0s2 partition map
*
* Dimensions:
* 512 bytes/sector
* 133 sectors/track
* 27 tracks/cylinder
* 3591 sectors/cylinder
* 4926 cylinders

Sun Proprietary: Internal Use Only


2-36 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Encapsulating Boot Disks

* 4924 accessible cylinders


*
* Flags:
* 1: unmountable
* 10: read-only
*
* Unallocated space: <-- Notice this disk now has unallocated space.
* First Sector Last
* Sector Count Sector
* 15581349 4279385947 4294967295
* 17682084 4292870152 15584939
* 16637103 1041390 17678492
* First Sector Last
* Partition Tag Flags Sector Count Sector Mount Directory
0 2 00 0 15581349 15581348
1 3 01 15584940 1052163 16637102
2 5 00 0 17682084 17682083

Note – It is a VxVM software best practice to always configure boot disks


with free unpartitioned space at the beginning or end of the disk in the
event that the disk is encapsulated at another time.

Pre-Encapsulation eeprom Command

The following truncated example output from the eeprom command


shows the contents of the system’s nvramrc area. Once the boot disk is
encapsulated and mirrored, new device aliases are created to delineate the
root disk and mirror for booting purposes.
bash-2.03# eeprom
disabled-memory-list: data not available.
disabled-board-list: data not available.
memory-interleave=max
.
.
.
boot-device=test1
local-mac-address?=false
ansi-terminal?=true
screen-#columns=80
screen-#rows=34
silent-mode?=false
use-nvramrc?=true
nvramrc=devalias test1 /sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w210000203713fc9f,0:a
devalias test3 /sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w21000020372d5917,0:a
devalias test2 /sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w220000203713f643,0:a
.
.
.

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-37
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Encapsulating Boot Disks

hardware-revision: data not available.


last-hardware-update: data not available.
diag-switch?=false

Post-Encapsulation Configuration Data


The following examples were captured after the VxVM software was
installed and the boot disk encapsulated. This example depicts a two-slice
boot disk.

Post-Encapsulation /etc/vfstab File

The /etc/vfstab file displays the new devices to mount and to fsck
after the encapsulation in this example. Notice that comments at the end
of this example file describe the pre-encapsulation device configuration of
the / and swap partitions.
#device device mount FS fsck mount mount
#to mount to fsck point type pass at boot options
#
#/dev/dsk/c1d0s2 /dev/rdsk/c1d0s2 /usr ufs 1 yes -
fd - /dev/fd fd - no -
/proc - /proc proc - no -
/dev/vx/dsk/swapvol - - swap - no -
/dev/vx/dsk/rootvol /dev/vx/rdsk/rootvol / ufs 1 no logging
swap - /tmp tmpfs - yes -
#NOTE: volume rootvol (/) encapsulated partition c1t0d0s0
#NOTE: volume swapvol (swap) encapsulated partition c1t0d0s1

Post-Encapsulation df -k Command

The following example of the df -k command shows that / is mounted


using the new encapsulated root volume.
bash-2.03# df -k
Filesystem kbytes used avail capacity Mounted on
/dev/vx/dsk/rootvol 7670973 1646089 5948175 22% /
/proc 0 0 0 0% /proc
fd 0 0 0 0% /dev/fd
mnttab 0 0 0 0% /etc/mnttab
swap 1175352 16 1175336 1% /var/run
swap 1175360 24 1175336 1% /tmp

Sun Proprietary: Internal Use Only


2-38 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Encapsulating Boot Disks

Post-Encapsulation swap -l Command

An example of the swap -l command shows that the encapsulated


swapvol is used as the swap device.
bash-2.03# swap -l
swapfile dev swaplo blocks free
/dev/vx/dsk/swapvol 75,5 16 1052144 1052144

Post-Encapsulation prtvtoc Command

Output from the following example of the prtvtoc command clearly


shows the encapsulation slicing scheme. Unlike that of the data disk
encapsulation slicing, encapsulated boot disk partitioning is more
complex. A two-slice boot disk private region uses slice 4, and the public
region is slice 3, which overlaps the entire disk.

Additionally, there are two overlay partitions for the / and swap partitions
that block-for-block match the original / and swap partition locations.
This is an important distinction. The overlay partitions are used during
initial boot processing prior to the initialization of the vxconfigd
daemon. Because the data in an encapsulated disk is not overwritten, the
overlay partitions map to the proper data blocks. This allows you access
to them without having to perform an active vxconfigd process to access
the rootvol and swapvol volumes.
bash-2.03# prtvtoc /dev/rdsk/c1t0d0s2
* /dev/rdsk/c1t0d0s2 partition map
*
* Dimensions:
* 512 bytes/sector
* 133 sectors/track
* 27 tracks/cylinder
* 3591 sectors/cylinder
* 4926 cylinders
* 4924 accessible cylinders
*
* Flags:
* 1: unmountable
* 10: read-only
*
* Unallocated space:
* First Sector Last
* Sector Count Sector
* 15581349 4279385947 4294967295
* 17682084 4292866561 15581348
*
* First Sector Last
* Partition Tag Flags Sector Count Sector Mount Directory
0 2 00 0 15581349 15581348 <-- Overlay partition for /

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-39
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Encapsulating Boot Disks

1 3 01 15584940 2097144 17682083 <-- Overlay partition for swap


2 5 00 0 17682084 17682083 <-- Backup
3 14 01 0 17682084 17682083 <-- Public Region
4 15 01 15581349 3591 15584939 <-- Private Region

Post-Encapsulation format Utility print Option

The following example of the print selection from the format utility
shows another view of the encapsulated disk partitioning scheme.
partition> print
Current partition table (original):
Total disk cylinders available: 4924 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks


0 root wm 0 - 4338 7.43GB (4339/0/0) 15581349 <-- Overlay
1 swap wu 4340 - 4923 1024.00MB (584/0/0) 2097144 <-- Overlay
2 backup wm 0 - 4923 8.43GB (4924/0/0) 17682084
3 - wu 0 - 4923 8.43GB (4924/0/0) 17682084 <-- Public
4 - wu 4339 - 4339 1.75MB (1/0/0) 3591 <-- Private
5 unassigned wm 0 0 (0/0/0) 0
6 unassigned wm 0 0 (0/0/0) 0
7 unassigned wm 0 0 (0/0/0) 0

Post-Encapsulation vxprint -ht (No Mirror) Command

A vxprint command example shows the following:


● A VxVM software disk called rootdisk – The rootdisk is the
default disk name for an encapsulated disk when using the
vxinstall or the vxdiskadm utility to perform the encapsulation.
● Two plexes for the rootvol volume, as follows:
● rootdisk-B0 – A shadow subdisk used to preserve an
encapsulated disk boot block.
● rootdisk-02 – The encapsulated / partition.
● One plex for the swapvol volume – Subdisk rootdisk-01 is the
encapsulated swap partition.
bash-2.03# vxprint -ht
Disk group: rootdg

DG NAME NCONFIG NLOG MINORS GROUP-ID


DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
V NAME RVG KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE

Sun Proprietary: Internal Use Only


2-40 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Encapsulating Boot Disks

SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE


DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO

dg rootdg default default 0 1019673916.1025.lowtide

dm rootdisk c1t0d0s2 sliced 3590 17682083 -

sd rootdiskPriv - rootdisk 15581349 3590 PRIVATE c1t0d0 ENA

v rootvol - ENABLED ACTIVE 15581349 ROUND - root


pl rootvol-01 rootvol ENABLED ACTIVE 15581349 CONCAT - RW
sd rootdisk-B0 rootvol-01 rootdisk 15581348 1 0 c1t0d0 ENA
sd rootdisk-02 rootvol-01 rootdisk 0 15581348 1 c1t0d0 ENA

v swapvol - ENABLED ACTIVE 2097144 ROUND - swap


pl swapvol-01 swapvol ENABLED ACTIVE 2097144 CONCAT - RW
sd rootdisk-01 swapvol-01 rootdisk 15584939 2097144 0 c1t0d0 ENA

Mirroring the Encapsulated Boot Disk


Mirroring an encapsulated boot disk is accomplished using either the
vxdiskadm or vxrootmir utility. The boot disk mirror used in this
example was created using the vxdiskadm utility.

Post-Encapsulation vxprint -ht Command With Mirror

Notice in the following example output that each system volume (root
and swap) is now mirrored using a VxVM software disk called
rootmirror. The example rootmirror disk provides the following
plexes:
● rootvol - plex = rootvol-02 / sd = rootmirror-01
● swapvol - plex = swapvol-02 / sd = rootmirror-02
bash-2.03# vxprint -ht -g rootdg
DG NAME NCONFIG NLOG MINORS GROUP-ID
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
V NAME RVG KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO

dg rootdg default default 0 1019673916.1025.lowtide

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-41
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Encapsulating Boot Disks

dm rootdisk c1t0d0s2 sliced 3590 17678493 -


dm rootmirror c1t22d0s2 sliced 3590 17674902 -

v rootvol - ENABLED ACTIVE 15581349 ROUND - root


pl rootvol-01 rootvol ENABLED ACTIVE 15581349 CONCAT - RW
sd rootdisk-B0 rootvol-01 rootdisk 17678492 1 0 c1t0d0 ENA
sd rootdisk-02 rootvol-01 rootdisk 0 15581348 1 c1t0d0 ENA
pl rootvol-02 rootvol ENABLED ACTIVE 15581349 CONCAT - RW
sd rootmirror-01 rootvol-02 rootmirror 0 15581349 0 c1t22d0 ENA

v swapvol - ENABLED ACTIVE 1052163 ROUND - swap


pl swapvol-01 swapvol ENABLED ACTIVE 1052163 CONCAT - RW
sd rootdisk-01 swapvol-01 rootdisk 15584939 1052163 0 c1t0d0 ENA
pl swapvol-02 swapvol ENABLED ACTIVE 1052163 CONCAT - RW
sd rootmirror-02 swapvol-02 rootmirror 15581349 1052163 0 c1t22d0 ENA

Post-Encapsulation vxdisk list Command

Example output from the vxdisk list command shows both the
encapsulated boot disk (rootdisk) and its mirror (rootmirror). Both
disks must be members of the rootdg disk group.
bash-2.03# vxdisk list
DEVICE TYPE DISK GROUP STATUS
c1t0d0s2 sliced rootdisk rootdg online
c1t1d0s2 sliced - - error
c1t2d0s2 sliced - - error
c1t3d0s2 sliced - - error
c1t6d0s2 sliced - - error
c1t16d0s2 sliced - - error
c1t17d0s2 sliced - - error
c1t18d0s2 sliced - - error
c1t19d0s2 sliced - - error
c1t20d0s2 sliced - - online
c1t21d0s2 sliced - - online
c1t22d0s2 sliced rootmirror rootdg online

Post-Encapsulation Boot Disk prtvtoc Command

The following example of the prtvtoc command output for the boot disk
displays an encapsulation partition scheme with overlay partitions for /
and swap. Notice the starting and stopping sectors for each overlay
partition.
bash-2.03# prtvtoc /dev/rdsk/c1t0d0s2
* /dev/rdsk/c1t0d0s2 partition map
*
* Dimensions:
* 512 bytes/sector
* 133 sectors/track
* 27 tracks/cylinder

Sun Proprietary: Internal Use Only


2-42 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Encapsulating Boot Disks

* 3591 sectors/cylinder
* 4926 cylinders
* 4924 accessible cylinders
*
* Flags:
* 1: unmountable
* 10: read-only
*
* Unallocated space:
* First Sector Last
* Sector Count Sector
* 15581349 4279385947 4294967295
* 17682084 4292870152 15584939
* 16637103 1041390 17678492
*
* First Sector Last
* Partition Tag Flags Sector Count Sector Mount Directory
0 2 00 0 15581349 15581348
1 3 01 15584940 1052163 16637102
2 5 00 0 17682084 17682083
3 14 01 0 17682084 17682083
4 15 01 17678493 3591 17682083

Post-Encapsulation Boot Disk Mirror prtvtoc Command

The following example shows the output from the prtvtoc command for
the root mirror disk. Notice that it not only mirrors the public and private
regions of the boot disk, but additionally creates overlay partitions for /
and swap. The partitions are configured in this manner, so the mirror disk
can be used to boot the system if the primary fails. The difference in the
starting sector for overlay partition 0 is because this is a VxVM software-
initialized disk which has a private region configured within the first two
cylinders of the disk. This causes the offset to sector 7182 for the overlay
of partition 0.
bash-2.03# prtvtoc /dev/rdsk/c1t22d0s2
* /dev/rdsk/c1t22d0s2 partition map
*
* Dimensions:
* 512 bytes/sector
* 133 sectors/track
* 27 tracks/cylinder
* 3591 sectors/cylinder
* 4926 cylinders
* 4924 accessible cylinders
*
* Flags:
* 1: unmountable
* 10: read-only
*
* Unallocated space:

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-43
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Encapsulating Boot Disks

* First Sector Last


* Sector Count Sector
* 3591 3591 7181
* 15588531 4279385947 7181
* 17682084 4292873743 15588530
* 16640694 1041390 17682083
*
* First Sector Last
* Partition Tag Flags Sector Count Sector Mount Directory
0 2 00 7182 15581349 15588530
1 3 01 15588531 1052163 16640693
2 5 00 0 17682084 17682083
3 15 01 0 3591 3590
4 14 01 7182 17674902 17682083

Post-Encapsulation Boot Disk format print Command

The format utility print option for the example rootdisk shows the
boot disk slicing on cylinder boundaries.
partition> print
Current partition table (original):
Total disk cylinders available: 4924 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks


0 root wm 0 - 4338 7.43GB (4339/0/0) 15581349
1 swap wu 4340 - 4632 513.75MB (293/0/0) 1052163
2 backup wm 0 - 4923 8.43GB (4924/0/0) 17682084
3 - wu 0 - 4923 8.43GB (4924/0/0) 17682084
4 - wu 4923 - 4923 1.75MB (1/0/0) 3591
5 unassigned wm 0 0 (0/0/0) 0
6 unassigned wm 0 0 (0/0/0) 0
7 unassigned wm 0 0 (0/0/0) 0

partition>

Post-Encapsulation Boot Disk Mirror format print Command

The following example output from the format utility print option is for
the rootmirror disk. Notice how the private region (slice 3) occupies
cylinder 0. The private region (slice 4) and overlay partition 0 start at
cylinder 2. Contrast this with the output for the encapsulated boot disk.
partition> print
Current partition table (original):
Total disk cylinders available: 4924 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks


0 root wm 2 - 4340 7.43GB (4339/0/0) 15581349
1 swap wu 4341 - 4633 513.75MB (293/0/0) 1052163
2 backup wm 0 - 4923 8.43GB (4924/0/0) 17682084

Sun Proprietary: Internal Use Only


2-44 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Encapsulating Boot Disks

3 - wu 0 - 0 1.75MB (1/0/0) 3591


4 - wu 2 - 4923 8.43GB (4922/0/0) 17674902
5 unassigned wm 0 0 (0/0/0) 0
6 unassigned wm 0 0 (0/0/0) 0
7 unassigned wm 0 0 (0/0/0) 0
partition>

Related Post-Encapsulation Files


When a disk is encapsulated, a subdirectory under
/etc/vx/reconfig.d/disk.d is created which contains files that hold
information related to the encapsulation. The following is an example of
files that relate to the encapsulation of c1t0d0.

File List

A directory list of files specific to c1t0d0 is shown in the following


example.
bash-2.03# pwd
/etc/vx/reconfig.d/disk.d/c1t0d0

bash-2.03# ls -las
total 12
2 drwxr-xr-x 2 root other 512 Apr 26 19:17 .
2 drwxr-xr-x 4 root other 512 Apr 24 15:29 ..
2 -rw-r--r-- 1 root other 9 Apr 26 19:17 dmname
2 -rw-r--r-- 1 root other 827 Apr 26 19:17 newpart
2 -rw-r--r-- 1 root other 7 Apr 26 19:17 primary_node
2 -rw-r--r-- 1 root other 452 Apr 26 19:17 vtoc

The dmname File

The dmname file for this example lists the VxVM software disk name of the
encapsulated disk.
bash-2.03# more *

::::::::::::::
dmname
::::::::::::::
rootdisk
::::::::::::::

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-45
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Encapsulating Boot Disks

The newpart File

The following example of the newpart file displays the new partition
information for this encapsulated disk. Additionally, it lists all the VxVM
software commands executed to successfully complete the encapsulation
process.
::::::::::::::
newpart
::::::::::::::
# volume manager partitioning for drive c1t0d0
0 0x2 0x200 0 15581349
1 0x3 0x201 15584940 1052163
2 0x5 0x200 0 17682084
3 0xe 0x201 0 17682084
4 0xf 0x201 17678493 3591
5 0x0 0x000 0 0
6 0x0 0x000 0 0
7 0x0 0x000 0 0
#vxmake vol rootvol plex=rootvol-%%00 usetype=root logtype=none
#vxmake plex rootvol-%%00 sd=rootdisk-B0,rootdisk-%%00
#vxmake sd rootdisk-%%00 disk=rootdisk offset=0 len=15581348
#vxmake sd rootdisk-B0 disk=rootdisk offset=17678492 len=1 putil0=Block0 comment="Remap
of block 0
#vxvol start rootvol
#rename c1t0d0s0 rootvol
#vxmake vol swapvol plex=swapvol-%%01 usetype=swap
#vxmake plex swapvol-%%01 sd=rootdisk-%%01
#vxmake sd rootdisk-%%01 disk=rootdisk offset=15584939 len=1052163
#vxvol start swapvol
#rename c1t0d0s1 swapvol

The primary_node File

The primary_node file lists the original c#t#d# name of the encapsulated
disk in this example.
::::::::::::::
primary_node
::::::::::::::
c1t0d0
::::::::::::::

Sun Proprietary: Internal Use Only


2-46 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Encapsulating Boot Disks

The vtoc File

The following example of the vtoc file saves the original partition table of
the encapsulated disk. In earlier versions of the VxVM software, this file
was modified and used with the /etc/fmthard command to re-partition
disks during the unencapsulation process. The vxmksdpart command
now performs this function.
::::::::::::::
./vtoc
::::::::::::::
#THE PARTITIONING OF /dev/rdsk/c1t0d0s2 IS AS FOLLOWS:
#SLICE TAG FLAGS START SIZE
0 0x2 0x200 0 15581349
1 0x3 0x201 15584940 1052163
2 0x5 0x200 0 17682084
3 0x0 0x000 0 0
4 0x0 0x000 0 0
5 0x0 0x000 0 0
6 0x0 0x000 0 0
7 0x0 0x000 0 0

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-47
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining Sun Enterprise Services Best Practices for VxVM Software-Managed

Examining Sun Enterprise Services Best Practices for


VxVM Software-Managed Boot Disks
The previous sections describe the processes and procedures to
encapsulate and mirror a system’s boot disk using standard VxVM
software recommendations from the VERITAS Software Corporation. This
type of encapsulation is found in many sites that have not contracted with
Sun Enterprise Services to install the VxVM software and bring the
system’s boot disk under VxVM software management.

The main difference between the standard VxVM software-encapsulated


disk and the Sun Enterprise Services management process is that the Sun
Enterprise Services method does not leave the root disk as an
encapsulated disk. Once the boot disk is mirrored, the encapsulated root
disk is detached, initialized, and mirrored to the original boot disk mirror.
This removes the B0 subdisk and makes the two VxVM disks used to
manage the rootability of the system identical. The complexity of the boot
disk configuration is reduced by providing a consistent slice structure
between the boot disk and its mirrors.

This section addresses the following two methods of bringing a boot disk
under VxVM software management using the Sun Enterprise Services
best practice recommendations:
● Manual process using the command line
● Scripted process using the Sun Enterprise Installation Services (EIS)
CD-ROM

Note – The Sun Enterprise Services best practices for boot disk
management are based on guidelines from the Sun BluePrints OnLine
document Toward a Reference Configuration for VxVM Managed Boot Disks,
part number 806-6197-10.

Sun Proprietary: Internal Use Only


2-48 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining Sun Enterprise Services Best Practices for VxVM Software-Managed

Best-Practice Boot Disk Configuration Guidelines


Use the following design goals to configure boot disk media:
● Simple – Keep the configuration simple. A system administrator
with moderate levels of experience should be able to view the boot
disk and understand the configuration.
● Consistent – Boot disk configurations that are alike across an
enterprise result in simpler recovery operations and installation
procedures. If system administrators can successfully troubleshoot a
boot disk failure on one system, consistency increases the possibility
that the administrator can do so on other systems.
● Resilient – Design boot media so that a single hardware, driver or
device failure does not cause a failure. Do not tolerate a single point
of failure.

The following are recommended guidelines for configuring VxVM


software-managed boot media:
● Use a maximum of three slices for the boot disk.
The suggested best practice is use only the / and swap slices. If
necessary, also configure the /var slice as a discreet slice.
● Never configure /usr as a separate slice.
All the support files and utilities for the VxVM software are located
in the /usr directory. If the /usr volume cannot be mounted, these
support files are unavailable.
● Attach mirrors in geographical, not alphabetical, order.
The vxdiskadm process mirrors volumes in alphabetical order. Do
not use vxdiskadm to mirror the boot media. Mirroring the disk in
geographical order ensures that the mirror disk overlay slicing looks
identical to the original boot disk. Depending on the release of the
VxVM software, this is only an issue if the boot disk is sliced with
partitions other than /, swap, and /var.
● Convert the boot disk from an encapsulated disk to an initialized
disk.
An encapsulated disk is a special case and may be the only
encapsulated device in the system. Replacing the encapsulated boot
disk with an initialized disk insures that the mirror disk and the boot
disk are exact clones of each other. This reduces the complexity of the
VxVM software-managed boot disk configuration.

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-49
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining Sun Enterprise Services Best Practices for VxVM Software-Managed

● Map core operating system volumes to disk slices or partitions.


This ensures that the system is bootable with minimal effort if the
VxVM software does not start.
● Ensure that all mirrors of the boot disk are bootable and that a device
alias is present in the OpenBoot™ programmable read-only memory
(PROM).
● Create a clone disk.
The clone disk must be bootable from slices, not volumes, and be
able to run VxVM software utilities. This disk is used if there is a
complete failure of a VxVM software-managed boot disk which
makes the system un-bootable.

Manually Bringing the Boot Disk Under VxVM Software


Management
Use the following procedure to configure a boot disk according to the
Sun Enterprise Services best practices.

This example uses the following disks and slicing:


● /dev/dsk/c0t0d0 – rootdisk
● /dev/dsk/c1t1d0 – rootmirror

The boot disk has three partitions:


● / – Slice 0
● swap – Slice 1
● /var – Slice 6

Perform the following:


1. Save the boot disk’s vtoc to a file for later reference, if needed. Type
the following:
# mkdir /var/tmp/sysdocs
# prtvtoc /dev/rdsk/c0t0d0s2 > /var/tmp/sysdocs/rootdisk_prevm.vtoc
2. Encapsulate the boot disk, using either the vxinstall or vxdiskadm
utility. Boot disk encapsulation is usually completed during the
installation and initial setup of the VxVM software by using the
vxinstall utility.

Sun Proprietary: Internal Use Only


2-50 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining Sun Enterprise Services Best Practices for VxVM Software-Managed

3. Initialize the disk to be used as the rootmirror and add it to the


rootdg disk group. Type the following:
# /etc/vx/bin/vxdisksetup -i c1t1d0
# vxdg -g rootdg adddisk rootmirror=c1t1d0
4. Attach the mirrors in the order of the slices on the encapsulated boot
disk.

Note – If the boot disk was sliced with swap as the first slice, reverse the
order of mirroring for the / and swap slices.

Perform the following steps:


a. Use the following command to start the procedure to mirror /
to the rootmirror disk:
# /etc/vx/bin/vxrootmir rootmirror&
b. Use the following command to start the procedure to mirror the
swap slice:
# vxassist -g rootdg mirror swapvol rootmirror&
c. Use the following command to start the procedure to mirror the
/var slice:
# vxassist -g rootdg mirror var rootmirror&
Use the following procedure to display the progress of the mirroring
process:
#while true
> do
> vxtask list
> sleep 15
> echo “##################”
> done
TASKID PTID TYPE/STATE PCT PROGRESS
160 ATCOPY/R 84.39% 0/4197879/3542680 PLXATT rootvol rootvol-02
163 ATCOPY/R 24.95% 0/4197879/1047304 PLXATT var var-02
###############
TASKID PTID TYPE/STATE PCT PROGRESS
160 ATCOPY/R 86.00% 0/4197879/3610384 PLXATT rootvol rootvol-02
163 ATCOPY/R 26.57% 0/4197879/1115256 PLXATT var var-02
Wait for the mirroring procedure to complete before continuing this
procedure.
5. Dissociate the rootdisk plexes. Type the following:
# vxplex -g rootdg dis rootvol-01
# vxplex -g rootdg dis swapvol-01
# vxplex -g rootdg dis var-01
# vxedit -fr rm rootvol-01 swapvol-01 var-01

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-51
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining Sun Enterprise Services Best Practices for VxVM Software-Managed

6. Remove rootdisk from the rootdg disk group:


# vxdg -g rootdg rmdisk rootdisk
7. Initialize the rootdisk and add it back to the rootdg disk group:
# /etc/vx/bin/vxdisksetup -i c0t0d0
# vxdg -g rootdg adddisk rootdisk=c0t0d0
8. Attach the mirrors in the order of the slices on the encapsulated boot
disk. This is similar to the process outlined in Step 4.

Note – If the boot disk was sliced with swap as the first slice, reverse the
order of mirroring for the / and swap slices.

Perform the following steps:


a. Use the following command to start the procedure to mirror /
to the rootmirror disk:
# /etc/vx/bin/vxrootmir rootdisk&
b. Use the following command to start the procedure to mirror the
swap slice:
# vxassist -g rootdg mirror swapvol rootdisk&
c. Use the following command to start the procedure to mirror the
/var slice:
# vxassist -g rootdg mirror var rootdisk&
Use the following procedure to display the progress of the mirroring
process:
#while true
> do
> vxtask list
> sleep 15
> echo “##################”
> done
Wait for the mirroring procedure to complete before continuing this
procedure.
9. If needed, create the underlying partitions by creating overlay
partitions for each partition on the boot disk. Use the vxmksdpart
command to create the overlay partitions; vxmksdpart requires the
partition name, flags, and tags as input.

Note – Depending on the version of the VxVM software used, building


overlay partitions for /, swap and /var might not be necessary.

Sun Proprietary: Internal Use Only


2-52 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining Sun Enterprise Services Best Practices for VxVM Software-Managed

A list of valid flags is shown in Table 2-1.

Table 2-1 Partition Flags

Name Flag

Mountable, Read and Write 0x00


Not Mountable 0x01
Mountable, Read Only 0x10

Valid tags are listed in Table 2-2.

Table 2-2 Partition Tags

Name Tag

UNASSIGNED 0x00
BOOT 0x01
ROOT 0x02
SWAP 0x03
USR 0x04
BACKUP 0x05
STAND 0x06
VAR 0x07
HOME 0x08
ALTSCTR 0x09
CACHE 0x0a
VxVM PRIVATE REGION 0x15
VxVM PUBLIC REGION 0x14

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-53
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining Sun Enterprise Services Best Practices for VxVM Software-Managed

Use vxprint and prtvtoc to print partitioning and subdisk


information to identify which overlay partitions must be built. These
commands are shown in the following example.
# vxprint -qhtg rootdg
dg rootdg default default 0 1023554924.1025.lowtide

dm rootdisk c1t0d0s2 sliced 3590 17678493 -


dm rootmirror c1t1d0s2 sliced 3590 17674902 -

v rootvol - ENABLED ACTIVE 4197879 ROUND - root


pl rootvol-01 rootvol ENABLED ACTIVE 4197879 CONCAT - RW
sd rootdisk-B0 rootvol-01 rootdisk 17678492 1 0 c1t0d0 ENA
sd rootdisk-02 rootvol-01 rootdisk 0 4197878 1 c1t0d0 ENA
pl rootvol-02 rootvol ENABLED ACTIVE 4197879 CONCAT - RW
sd rootmirror-01 rootvol-02 rootmirror 0 4197879 0 c1t1d0 ENA

v swapvol - ENABLED ACTIVE 1052163 ROUND - swap


pl swapvol-01 swapvol ENABLED ACTIVE 1052163 CONCAT - RW
sd rootdisk-01 swapvol-01 rootdisk 4197878 1052163 0 c1t0d0 ENA
pl swapvol-02 swapvol ENABLED ACTIVE 1052163 CONCAT - RW
sd rootmirror-02 swapvol-02 rootmirror 4197879 1052163 0 c1t1d0 ENA

v var - ENABLED ACTIVE 4197879 ROUND - fsgen


pl var-01 var ENABLED ACTIVE 4197879 CONCAT - RW
sd rootdisk-03 var-01 rootdisk 9447920 4197879 0 c1t0d0 ENA
pl var-02 var ENABLED ACTIVE 4197879 CONCAT - RW
sd rootmirror-03 var-02 rootmirror 5250042 4197879 0 c1t1d0 ENA

# prtvtoc /dev/rdsk/c1t1d0s2
* /dev/rdsk/c1t1d0s2 partition map
*
* Dimensions:
* 512 bytes/sector
* 133 sectors/track
* 27 tracks/cylinder
* 3591 sectors/cylinder
* 4926 cylinders
* 4924 accessible cylinders
*
* Flags:
* 1: unmountable
* 10: read-only
*
* Unallocated space:
* First Sector Last
* Sector Count Sector
* 3591 3591 7181
* 4205061 4290769417 7181
* 17682084 4281490273 4205060
* 9455103 8226981 17682083
*

Sun Proprietary: Internal Use Only


2-54 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining Sun Enterprise Services Best Practices for VxVM Software-Managed

* First Sector Last


* Partition Tag Flags Sector Count Sector Mount Directory
0 2 00 7182 4197879 4205060
1 3 01 4205061 1052163 5257223
2 5 00 0 17682084 17682083
3 15 01 0 3591 3590
4 14 01 7182 17674902 17682083

Note – In this example, the capture was modified to show the /var slice
as not built. This was done to help show how to use the vxmksdpart
command to create overlay partitions. The VxVM software version 3.2
creates overlay partitions for /, swap and /var, and thus you do not need
to execute any additional commands. If /opt or other non-system
partitions are defined on the boot disk, use vxmksdpart to define those
partitions.

In this example, an overlay partition must be built for the /var slice.
The subdisk information listed in Table 2-3 is needed as input to the
vxmksdpart command.

Table 2-3 Required subdisk Information

Subdisk Slice Tag Flag


rootdisk-03 5 0x07 0x00
rootmirror-03 5 0x07 0x00

To create the underlying partitions for /var, type the following:


# /usr/vx/bin/vxmksdpart -g rootdg rootdisk-03 6 0x07 0x00
# /usr/vx/bin/vxmksdpart -g rootdg rootmirror-03 6 0x07 0x00
This example output from the prtctoc command now shows the
/var partition as an overlay partition.
# prtvtoc /dev/rdsk/c1t1d0s2
* /dev/rdsk/c1t1d0s2 partition map
*
* Dimensions:
* 512 bytes/sector
* 133 sectors/track
* 27 tracks/cylinder
* 3591 sectors/cylinder
* 4926 cylinders
* 4924 accessible cylinders
*
* Flags:
* 1: unmountable
* 10: read-only

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-55
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining Sun Enterprise Services Best Practices for VxVM Software-Managed

*
* Unallocated space:
* First Sector Last
* Sector Count Sector
* 3591 3591 7181
* 4205061 4290769417 7181
* 17682084 4281490273 4205060
* 9455103 8226981 17682083
*
* First Sector Last
* Partition Tag Flags Sector Count Sector Mount Directory
0 2 00 7182 4197879 4205060
1 3 01 4205061 1052163 5257223
2 5 00 0 17682084 17682083
3 15 01 0 3591 3590
4 14 01 7182 17674902 17682083
6 7 00 5257224 4197879 9455102
10. Set the dump device to a non-VxVM software disk, if available. If
such a disk is not available, use the boot disk. Type the following:
# dumpadm -d /dev/dsk/c0t0d0s1
11. Create the OpenBoot PROM device aliases, if needed. Build the
aliases using the eeprom commands nvedit or nvalias at the
OpenBoot PROM prompt.

Note – Depending on the version of VxVM software installed, this step


might not be necessary.

Scripted Process Using the EIS CD-ROM


The EIS CD-ROM contains a script that mirrors the boot disk using the
same process as the manual best practices procedures. It is located on the
EIS CD-ROM at the following location:
/cdrom/eis-cd2/sun-internal/tools/VXTOOLS/K.Vietmeier.scripts/mirrorboot_<rev>.ksh

Caution – This scripts is distributed as is and is not supported by Sun


Microsystems. To gain familiarity with the process, try performing the
manual procedure multiple times prior to using this script.

Sun Proprietary: Internal Use Only


2-56 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Unencapsulating Boot Disks

Unencapsulating Boot Disks


To unencapsulate boot disks (remove rootability), use the vxunroot utility
or perform the process manually. Use the manual method only if the
vxunroot utility fails to unencapsulate the boot disk. The manual method
is similar to the process used to unencapsulate data disks, described in
‘‘Unencapsulating Data Disks’’ on page 2-23.

This section addresses multiple methods of boot disk unencapsulation.


Additional information about boot disk unencapsulation is located in
SunSolve INFODOC 24663 (see Appendix A, “SunSolve INFODOCs”).

Boot disk unencapsulation always requires a reboot of the system.

Caution – Back up all volumes to be unencapsulated. Use these backups


for recovery purposes if the process fails and the encapsulated volumes
are corrupted.

Unencapsulating a Boot Disk Using the vxunroot


Utility
The following process is based on best practice recommendations from the
VERITAS Software Corporation TechNote ID 244678. This example
depicts a two-slice boot disk:
1. Unmount all file systems that are under the control of the VxVM
software except the encapsulated boot disk file systems (/, usr, var,
and opt). Type the following:
bash-2.03# df -n
/ : ufs
/proc : proc
/dev/fd : fd
/etc/mnttab : mntfs
/var/run : tmpfs
/tmp : tmpfs
2. Record and save the current configuration of rootdg using the
vxprint -qhtg rootdg command. This preserves a copy of the
rootdg configuration prior to unencapsulation, as shown in the
following example.
bash-2.03# vxprint -qhtg rootdg >/rootdg.print

bash-2.03# more /rootdg.print


dg rootdg default default 0 1019673916.1025.lowtide

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-57
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Unencapsulating Boot Disks

dm rootdisk c1t0d0s2 sliced 3590 17678493 -


dm rootmirror c1t22d0s2 sliced 3590 17674902 -

v rootvol - ENABLED ACTIVE 15581349 ROUND - root


pl rootvol-01 rootvol ENABLED ACTIVE 15581349 CONCAT - RW
sd rootdisk-B0 rootvol-01 rootdisk 17678492 1 0 c1t0d0 ENA
sd rootdisk-02 rootvol-01 rootdisk 0 15581348 1 c1t0d0 ENA
pl rootvol-02 rootvol ENABLED ACTIVE 15581349 CONCAT - RW
sd rootmirror-01 rootvol-02 rootmirror 0 15581349 0 c1t22d0 ENA

v swapvol - ENABLED ACTIVE 1052163 ROUND - swap


pl swapvol-01 swapvol ENABLED ACTIVE 1052163 CONCAT - RW
sd rootdisk-01 swapvol-01 rootdisk 15584939 1052163 0 c1t0d0 ENA
pl swapvol-02 swapvol ENABLED ACTIVE 1052163 CONCAT - RW
sd rootmirror-02 swapvol-02 rootmirror 15581349 1052163 0 c1t22d0 ENA
3. Capture the current configuration of the disk using the vxdisk
utility:
bash-2.03# vxdisk -o alldgs list > /vxdisk_alldgs.list
4. Detach all plexes associated with the rootmirror disk. Type the
following:
bash-2.03# vxprint -qhtg rootdg -s | grep -i rootmirror | awk ’{print $3}’> /rmsub.plex

bash-2.03# more /rmsub.plex


rootvol-02
swapvol-02

bash-2.03# for i in ‘cat /rmsub.plex‘


> do
> vxplex -g rootdg dis $i
> vxprint -qhtg rootdg -p $i
> done

pl rootvol-02 - DISABLED - 15581349 CONCAT - RW


sd rootmirror-01 rootvol-02 rootmirror 0 15581349 0 c1t22d0 ENA
pl swapvol-02 - DISABLED - 1052163 CONCAT - RW
sd rootmirror-02 swapvol-02 rootmirror 15581349 1052163 0 c1t22d0 ENA
bash-2.03#
Adding the rm option to the vxplex command removes the mirror
plexes in addition to performing the disable operation. The
command syntax is as follows:
# vxplex -g rootdg -o rm dis plex_name
If executed within a loop, the command syntax is –
> vxplex -g rootdg -o rm dis $i
5. Verify that all rootmirror plexes were detached. Use the vxprint
command as follows:

Sun Proprietary: Internal Use Only


2-58 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Unencapsulating Boot Disks

bash-2.03# vxprint -qhtg rootdg


dg rootdg default default 0 1019673916.1025.lowtide

dm rootdisk c1t0d0s2 sliced 3590 17678493 -


dm rootmirror c1t22d0s2 sliced 3590 17674902 -

pl rootvol-02 - DISABLED - 15581349 CONCAT - RW


sd rootmirror-01 rootvol-02 rootmirror 0 15581349 0 c1t22d0 ENA

pl swapvol-02 - DISABLED - 1052163 CONCAT - RW


sd rootmirror-02 swapvol-02 rootmirror 15581349 1052163 0 c1t22d0 ENA

v rootvol - ENABLED ACTIVE 15581349 ROUND - root


pl rootvol-01 rootvol ENABLED ACTIVE 15581349 CONCAT - RW
sd rootdisk-B0 rootvol-01 rootdisk 17678492 1 0 c1t0d0 ENA
sd rootdisk-02 rootvol-01 rootdisk 0 15581348 1 c1t0d0 ENA

v swapvol - ENABLED ACTIVE 1052163 ROUND - swap


pl swapvol-01 swapvol ENABLED ACTIVE 1052163 CONCAT - RW
sd rootdisk-01 swapvol-01 rootdisk 15584939 1052163 0 c1t0d0 ENA
Successful completion of the previous steps is critical to the success
of this process. If all plexes from the mirror are not disabled, the
vxunroot utility fails.
6. Remove rootability using the vxunroot utility as follows:
bash-2.03# /etc/vx/bin/vxunroot

This operation will convert the following file systems from


volumes to regular partitions: root swap usr var opt home

Replace volume rootvol with c1t0d0s0.

This operation will require a system reboot. If you choose to


continue with this operation, system configuration will be updated
to discontinue use of the volume manager for your root and swap
devices.

Do you wish to do this now [y,n,q,?] (default: y) y

Restoring kernel configuration...

A shutdown is now required to install the new kernel.


You can choose to shutdown now, or you can shutdown later, at your
convenience.

Do you wish to shutdown now [y,n,q,?] (default: n) n

Please shutdown before you perform any additional volume manager


or disk reconfiguration. To shutdown your system cd to / and type

shutdown -g0 -y -i6

bash-2.03# init 6

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-59
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Unencapsulating Boot Disks

7. After the reboot completes, check the devices used for the / and
swap partitions. Verify that these partitions use non-VxVM software
objects:
bash-2.03# df -k
Filesystem kbytes used avail capacity Mounted on
/dev/dsk/c1t0d0s0 7670973 1697985 5896279 23% /
/proc 0 0 0 0% /proc
fd 0 0 0 0% /dev/fd
mnttab 0 0 0 0% /etc/mnttab
swap 655784 16 655768 1% /var/run
swap 655792 24 655768 1% /tmp
bash-2.03# swap -l
swapfile dev swaplo blocks free
/dev/dsk/c1t0d0s1 118,145 16 1052144 1052144

Manually Unencapsulating a Boot Disk


The following procedure has much in common with the unencapsulation
process used for data disks. Only use this procedure if the vxunroot
utility fails to unencapsulate the boot disk.

The following procedure is based on VERITAS Software Corporation


TechNote ID 244678:
1. Unmount all the file systems that are under the control of the VxVM
software, except the encapsulated boot disk file systems (/, usr, var,
and opt). Type the following:
bash-2.03# df -n
/ : ufs
/proc : proc
/dev/fd : fd
/etc/mnttab : mntfs
/var/run : tmpfs
/tmp : tmpfs
2. Record the current configuration of rootdg using the
vxprint -qhtg rootdg command. This preserves a copy of the
rootdg configuration prior to unencapsulation. Type the following:
bash-2.03# vxprint -qhtg rootdg >/rootdg.print
3. Capture the current disk configuration using the vxdisk utility:
bash-2.03# vxdisk -o alldgs list > vxdisk_alldgs.list

Sun Proprietary: Internal Use Only


2-60 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Unencapsulating Boot Disks

4. Detach all plexes associated with the rootmirror disk. Use the
following script:
bash-2.03# vxprint -qhtg rootdg -s | grep -i rootmirror | awk ’{print $3}’> /rmsub.plex

bash-2.03# for i in ‘cat ./rmsub.plex‘


> do
> vxplex -g rootdg -o rm dis $i
> done
5. Verify that all rootmirror plexes were detached using the vxprint
command as follows:
bash-2.03# vxprint -qhtg rootdg
dg rootdg default default 0 1019673916.1025.lowtide

dm rootdisk c1t0d0s2 sliced 3590 17678493 -


dm rootmirror c1t22d0s2 sliced 3590 17674902 -

v rootvol - ENABLED ACTIVE 15581349 ROUND - root


pl rootvol-01 rootvol ENABLED ACTIVE 15581349 CONCAT - RW
sd rootdisk-B0 rootvol-01 rootdisk 17678492 1 0 c1t0d0 ENA
sd rootdisk-02 rootvol-01 rootdisk 0 15581348 1 c1t0d0 ENA

v swapvol - ENABLED ACTIVE 1052163 ROUND - swap


pl swapvol-01 swapvol ENABLED ACTIVE 1052163 CONCAT - RW
sd rootdisk-01 swapvol-01 rootdisk 15584939 1052163 0 c1t0d0 ENA
Successful completion of the previous steps are critical to the success
of this process. If all plexes from the mirror are not disabled, the
vxunroot utility fails.
6. Remove rootability using the following manual procedure:
a. Remove the /etc/vx/reconfig.d/state.d/root-done file.
Typethe following:
bash-2.03# rm /etc/vx/reconfig.d/state.d/root-done
Removal of this file tells the VxVM software that the root disk is
no longer encapsulated.
b. Edit the /etc/system and /etc/vfstab files back to their pre-
encapsulation state. Be sure to back up each file prior to editing.
Perform the following tasks:
● Remove the following lines from the /etc/system file –
rootdev:/pseudo/vxio@0:0
set vxio:vol_rootdev_is_volume=1

Caution – Do not comment out these lines, remove them.

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-61
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Unencapsulating Boot Disks

● Restore the /etc/vfstab file to its pre-encapsulation state.


Edit the /etc/vfstab file and modify the mount
statements for /, usr, var, and swap to use the original
physical devices. The original partitions were preserved
through the use of overlay partitions.
Alternatively, copy the /etc/vfstab.prevm file to
/etc/vfstab. This option is valid only if there were no
changes to the system storage configuration since the boot
disk was encapsulated.
The edited file looks like the following example:
#device device mount FS fsck mount mount
#to mount to fsck point type pass at boot options
#
#/dev/dsk/c1d0s2 /dev/rdsk/c1d0s2 /usr ufs 1 yes -
fd - /dev/fd fd - no -
/proc - /proc proc - no -
/dev/dsk/c1t0d0s1 - - swap - no -
/dev/dsk/c1t0d0s0 /dev/rdsk/c1t0d0s0 / ufs 1 no logging
swap - /tmp tmpfs - yes -
c. If non-system (/, usr, var, and swap) partitions exist on the
boot disk and do not have an overlap partition already
configured, use the vxmksdpart command to recreate the
partitions.

Note – This process is similar to the one used to restore partitions for data
disk unencapsulation.

7. Reboot the system.


8. Recursively remove boot volumes from the rootdg disk group:
bash-2.03# vxedit -r -f rm rootvol
bash-2.03# vxedit -r -f rm swapvol
9. Verify that the volumes are gone. Type the following:
bash-2.03# vxprint -qhtg rootdg
dg rootdg default default 0 1019673916.1025.lowtide

dm rootdisk c1t0d0s2 sliced 3590 17678493 -


dm rootmirror c1t22d0s2 sliced 3590 17674902 -
10. Remove the boot disk from VxVM software control. Type the
following:
bash-2.03# vxdg -g rootdg rmdisk rootdisk

Sun Proprietary: Internal Use Only


2-62 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Unencapsulating Boot Disks

Caution – There must be one disk in rootdg, or the VxVM software does
not start. Do not remove the disk that is used as the root mirror.

11. Verify that the boot disk is removed from VxVM software control:
bash-2.03# vxprint -qhtg rootdg
dg rootdg default default 0 1019673916.1025.lowtide

dm rootmirror c1t22d0s2 sliced 3590 17674902 -


12. Delete public and private region partitions by using the format
utility. Type the following:
partition> print
Current partition table (unnamed):
Total disk cylinders available: 4924 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks


0 root wm 0 - 4338 7.43GB (4339/0/0) 15581349
1 swap wu 4340 - 4632 513.75MB (293/0/0) 1052163
2 backup wm 0 - 4923 8.43GB (4924/0/0) 17682084
3 unassigned wu 0 0 (0/0/0) 0
4 unassigned wu 0 0 (0/0/0) 0
5 unassigned wm 0 0 (0/0/0) 0
6 unassigned wm 0 0 (0/0/0) 0
7 unassigned wm 0 0 (0/0/0) 0
13. Run the vxdctl enable command to update the VxVM software
view of installed disks. Type the following:
bash-2.03# vxdctl enable

bash-2.03# vxdisk list


DEVICE TYPE DISK GROUP STATUS
c1t0d0s2 sliced - - error
c1t1d0s2 sliced - - error
c1t2d0s2 sliced - - error
c1t3d0s2 sliced - - online
c1t4d0s2 sliced - - online
c1t5d0s2 sliced - - online
c1t6d0s2 sliced - - online invalid
c1t16d0s2 sliced - - error
c1t17d0s2 sliced - - error
c1t18d0s2 sliced - - error
c1t19d0s2 sliced - - error
c1t20d0s2 sliced - - online
c1t21d0s2 sliced - - online
c1t22d0s2 sliced rootmirror rootdg online

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-63
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Unencapsulating Boot Disks

Unencapsulating When Booted From the CD-ROM


If it is not possible to boot a system into single-user mode, booting from
CD-ROM may be the only option for successful system recovery. The
procedure outlined in this section describes the process of booting a
system from the CD-ROM and unencapsulating the boot disk.

The following procedure is based on SunSolve INFODOC 24663. Refer to


Appendix A, “SunSolve INFODOCs,” for a complete copy of this
document. Check the SunSolve Online web site to determine if the
INFODOC has been updated.

Note – This is a recovery procedure.

To unencapsulate a boot disk when the system is booted from CD-ROM:


1. Bring the system to the ok prompt and insert a Solaris OE compact
disc (CD) into the CD-ROM drive.
2. Boot the system to single-user mode from the CD-ROM, as follows:
ok boot cdrom -s
3. After the system is booted from the CD-ROM, set the terminal type
so that the vi utility works correctly. Type the following:
# TERM=vt100;export TERM
If TERM=sun does not work, try TERM=vt100.
4. Execute an fsck on the root file system. Type the following:
# fsck -y /dev/rdsk/c#t#d#s0
5. If the fsck response is clean, mount slice 0 to /a.
If fsck cannot repair the root file system, determine the source of the
problem and correct it. This guide does not contain explanations of
file system corruption or how to repair it. The fsck response must be
clean to continue this procedure.
6. Mount the root file system to /a. Type the following:
# mount /dev/dsk/c#t#d#s0 /a
7. Make a backup of /a/etc/system.
# cp /a/etc/system /a/etc/system.orig
8. Edit the /etc/system file.
# vi /a/etc/system

Sun Proprietary: Internal Use Only


2-64 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Unencapsulating Boot Disks

9. Completely remove the following lines from the system file:


rootdev:/pseudo/vxio@0:0
set vxio:vol_rootdev_is_volume=1

Note – If the disk is re-encapsulated, these lines are added correctly by the
process, so there is no harm done by removing them.

10. Make a backup of /a/etc/vfstab. Type the following:


# cp /a/etc/vfstab /a/etc/vfstab.orig
11. Edit the vfstab file back to its original state, pointing /, swap, /usr,
and /var to hard partitions on the disk, such as /dev/dsk and
/dev/rdsk, rather than to /dev/vx/ entries. Type the following:
# vi /a/etc/vfstab
12. Temporarily comment out all other /dev/vx volumes from the
/a/etc/vfstab file by using the # character. This includes file
systems like /opt and /export, if they exist.
The original /etc/vfstab looks like the following, assuming root is
c0t0d0.
---------------------------------------------------------------------------
/dev/vx/dsk/swapvol - - swap - no -
/dev/vx/dsk/rootvol /dev/vx/rdsk/rootvol / ufs 1 no -
/dev/vx/dsk/usr /dev/vx/rdsk/usr /usr ufs 1 no -
/dev/vx/dsk/var /dev/vx/rdsk/var /var ufs 1 no -
/dev/vx/dsk/export /dev/vx/rdsk/export /export ufs 2 yes -
swap - /tmp tmpfs - yes -

/dev/vx/dsk/datadg/somevol /dev/vx/rdsk/datadg/somevol /somevol ufs 2 yes -

#NOTE: volume rootvol (/) encapsulated partition c0t0d0s0


#NOTE: volume swapvol (swap) encapsulated partition c0t0d0s1
#NOTE: volume usr (/usr) encapsulated partition c0t0d0s5
#NOTE: volume var (/var) encapsulated partition c0t0d0s6
#NOTE: volume export (/export) encapsulated partition c0t0d0s7
---------------------------------------------------------------------------
Once edited, the vfstab file should look like the following:
---------------------------------------------------------------------------
/dev/dsk/c1t0d0s1 - - swap - no -
/dev/dsk/c1t0d0s0 /dev/rdsk/c0t0d0s0 / ufs 1 no -
/dev/dsk/c1t0d0s5 /dev/rdsk/c0t0d0s5 /usr ufs 1 no -
/dev/dsk/c1t0d0s6 /dev/rdsk/c0t0d0s6 /var ufs 1 no -
#/dev/dsk/c1t0d0s7 /dev/rdsk/c0t0d0s7 /export ufs 2 yes -
swap - /tmp tmpfs - yes -

#/dev/vx/dsk/datadg/somevol /dev/vx/rdsk/datadg/somevol /somevol ufs 2 yes -

#NOTE: volume rootvol (/) encapsulated partition c0t0d0s0

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-65
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Unencapsulating Boot Disks

#NOTE: volume swapvol (swap) encapsulated partition c0t0d0s1


#NOTE: volume usr (/usr) encapsulated partition c0t0d0s5
#NOTE: volume var (/var) encapsulated partition c0t0d0s6
#NOTE: volume export (/export) encapsulated partition c0t0d0s7
---------------------------------------------------------------------------
13. Make sure that the VxVM software does not start during the next
boot.
# touch /a/etc/vx/reconfig.d/state.d/install-db
This is important because if the root disk contains mirrors, and the
system boots, the mirrors are synced; this corrupts the changes just
made.
14. Remove the flag that tells the VxVM software that the root disk is
encapsulated.
# rm /a/etc/vx/reconfig.d/state.d/root-done
15. Reboot the system for changes to take effect. Type the following:
# reboot
Once rebooted, the system comes up in a partially unencapsulated
state with /, /usr, /var, and swap mounted.

Note – The VxVM software does not start. It can be started manually once
the system is booted.

16. Start the VxVM software. Execute the following commands:


# rm /etc/vx/reconfig.d/state.d/install-db
# vxiod set 10
# vxconfigd -m disable
# vxdctl enable
17. Remove the volumes that existed on the encapsulated boot disk.
These are generally rootvol, swapvol, usr, and var. This might also
include home, opt, or other non-standard root partitions. Use the
command vxprint -htg rootdg to list the volumes in rootdg
before removing them.
Then, for each volume, run the following command:
# /usr/sbin/vxedit -rf rm volume_name
18. Remove the rootdisk from the rootdg disk group. The disk name is
usually rootdisk. Type the following:
# /usr/sbin/vxdg -k rmdisk disk_name

Sun Proprietary: Internal Use Only


2-66 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Unencapsulating Boot Disks

19. Re-write the vtoc of the disk so that hard partitions are again
defined for the root file systems.
There are several ways to put the hard partitions back into the vtoc
file, including using the fmthard command on a modified
/etc/vx/reconfig.d/disk.d/c#t#d#/vtoc file, using the format
utility to partition the disk manually, or using the vxmksdpart
command. The simplest method, however, is to use the vxedvtoc
command.
a. When the VxVM software encapsulates a disk, it makes a record
of the old vtoc of the disk. This file is stored for each disk in
/etc/vx/reconfig.d/disk.d/c#t#d#. It is stored in a VxVM
software-specific format, so it cannot be used as an argument
for the fmthard command unless it is modified. The vxedvtoc
command is similar to the fmthard command except that it can
read this vtoc file and write that vtoc to a disk. The command
takes the following form:
vxedvtoc -f filename devicename
b. Assuming that the boot disk is c0t0d0, run the command as
follows:
# /etc/vx/bin/vxedvtoc -f /etc/vx/reconfig.d/disk.d/c0t0d0/vtoc /dev/rdsk/c0t0d0s2
# THE ORIGINAL PARTITIONING IS AS FOLLOWS:
#SLICE TAG FLAGS START SIZE
0 0x0 0x200 0 0
1 0x0 0x200 0 0
2 0x5 0x201 0 8794112
3 0x0 0x200 0 0
4 0x0 0x200 0 0
5 0x0 0x200 0 0
6 0xe 0x201 0 8794112
7 0xf 0x201 8790016 4096
# THE NEW PARTITIONING WILL BE AS FOLLOWS :
#SLICE TAG FLAGS START SIZE
0 0x0 0x200 0 2048000
1 0x0 0x200 2048000 2048000
2 0x5 0x201 0 8794112
3 0x0 0x201 4096000 2048000
4 0x0 0x201 6144000 2048000
5 0x0 0x200 0 0
6 0x0 0x200 0 0
7 0x0 0x200 0 0
DO YOU WANT TO WRITE THIS TO THE DISK ? [Y/N] :y

WRITING THE NEW VTOC TO THE DISK


This partitions the disk back to a pre-encapsulation state.
20. Uncomment the entries for any of the non-root partitions (/, /usr,
/var, and swap) from /etc/vfstab, as well as any data volumes.

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-67
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Unencapsulating Boot Disks

In this example, comments were removed from /export and the data
volume /somevol:
# vi /etc/vfstab
/dev/dsk/c0t0d0s1 - - swap - no -
/dev/dsk/c0t0d0s0 /dev/rdsk/c0t0d0s0 / ufs 1 no -
/dev/dsk/c0t0d0s5 /dev/rdsk/c0t0d0s5 /usr ufs 1 no -
/dev/dsk/c0t0d0s6 /dev/rdsk/c0t0d0s6 /var ufs 1 no -
/dev/dsk/c0t0d0s7 /dev/rdsk/c0t0d0s7 /export ufs 2 yes -
swap - /tmp tmpfs - yes -
/dev/vx/dsk/datadg/somevol /dev/vx/rdsk/datadg/somevol /somevol ufs 2 yes -
21. Start all volumes.
# /usr/sbin/vxvol startall
22. Issue a mountall command to mount the now uncommented
volumes.
# mountall

At this point the root disk is completely free of the VxVM software
control. The VxVM software daemons are started, and all system file
systems should be mounted.

Performing a Basic or Functional Unencapsulation


This section describes the process for performing a temporary
unencapsulation of a system boot disk. Use this procedure only to attempt
recovery of a system that does not boot due to problems with the
encapsulated boot disk. This procedure unencapsulates the boot disk and
allows the system to boot from standard Solaris OE partitions while
leaving the root disk under control of the VxVM software.

This process is a useful boot-issue troubleshooting tool because it


performs a temporary unencapsulation of the boot disk, which can be
reversed once the underlying problem is resolved.

The following procedure is based on SunSolve INFODOC 24663. Refer to


Appendix A, “SunSolve INFODOCs,” for a complete copy of this
document. Check the SunSolve Online Web site to determine if the
INFODOC has been updated.

Note – This is a recovery procedure.

Sun Proprietary: Internal Use Only


2-68 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Unencapsulating Boot Disks

To perform a basic or functional unencapsulation:


1. Bring the system to the ok prompt, and insert a Solaris OE CD into
the CD-ROM drive.
2. Boot system to single-user mode from CD-ROM. Type the following:
ok boot cdrom -s
3. After the system is booted from the CD-ROM, set the terminal type
so that the vi utility works correctly.
# TERM=vt100;export TERM
If TERM=sun does not work, try TERM=vt100.
4. Run the fsck utility on the root file system. Type the following:
# fsck -y /dev/rdsk/c#t#d#s0
5. If the fsck response is clean, mount slice 0 to /a.
If fsck cannot repair the root file system, determine the source of the
problem and correct it. This guide does not contain explanations of
file system corruption or how to repair it. The fsck response must be
clean to continue this procedure.
6. Mount root file system to /a. Type the following:
# mount /dev/dsk/c#t#d#s0 /a
7. Make a backup of /a/etc/system.
# cp /a/etc/system /a/etc/system.orig
8. Edit the /etc/system file. Type the following:
# vi /a/etc/system
9. Comment out the following lines by using double asterisks (**):
**rootdev:/pseudo/vxio@0:0
**set vxio:vol_rootdev_is_volume=1
10. Make a backup of /a/etc/vfstab. Type the following:
# cp /a/etc/vfstab /a/etc/vfstab.orig
11. Edit the vfstab file back to its original state, pointing /, swap, /usr,
and /var to hard partitions on the disk, such as /dev/dsk and
/dev/rdsk, rather than to /dev/vx/ entries. Type the following:
# vi /a/etc/vfstab
12. Temporarily comment out all other /dev/vx volumes from the
/a/etc/vfstab file by using the # character. This includes file
systems like /opt and /export, if they exist.

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-69
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Unencapsulating Boot Disks

The original /etc/vfstab looks like the following, assuming root is


c0t0d0:
---------------------------------------------------------------------------
/dev/vx/dsk/swapvol - - swap - no -
/dev/vx/dsk/rootvol /dev/vx/rdsk/rootvol / ufs 1 no -
/dev/vx/dsk/usr /dev/vx/rdsk/usr /usr ufs 1 no -
/dev/vx/dsk/var /dev/vx/rdsk/var /var ufs 1 no -
/dev/vx/dsk/export /dev/vx/rdsk/export /export ufs 2 yes -
swap - /tmp tmpfs - yes -

/dev/vx/dsk/datadg/somevol /dev/vx/rdsk/datadg/somevol /somevol ufs 2 yes -

#NOTE: volume rootvol (/) encapsulated partition c0t0d0s0


#NOTE: volume swapvol (swap) encapsulated partition c0t0d0s1
#NOTE: volume usr (/usr) encapsulated partition c0t0d0s5
#NOTE: volume var (/var) encapsulated partition c0t0d0s6
#NOTE: volume export (/export) encapsulated partition c0t0d0s7
---------------------------------------------------------------------------
Once edited, the vfstab looks like the following:
---------------------------------------------------------------------------
/dev/dsk/c1t0d0s1 - - swap - no -
/dev/dsk/c1t0d0s0 /dev/rdsk/c0t0d0s0 / ufs 1 no -
/dev/dsk/c1t0d0s5 /dev/rdsk/c0t0d0s5 /usr ufs 1 no -
/dev/dsk/c1t0d0s6 /dev/rdsk/c0t0d0s6 /var ufs 1 no -
#/dev/dsk/c1t0d0s7 /dev/rdsk/c0t0d0s7 /export ufs 2 yes -
swap - /tmp tmpfs - yes -

#/dev/vx/dsk/datadg/somevol /dev/vx/rdsk/datadg/somevol /somevol ufs 2 yes -

#NOTE: volume rootvol (/) encapsulated partition c0t0d0s0


#NOTE: volume swapvol (swap) encapsulated partition c0t0d0s1
#NOTE: volume usr (/usr) encapsulated partition c0t0d0s5
#NOTE: volume var (/var) encapsulated partition c0t0d0s6
#NOTE: volume export (/export) encapsulated partition c0t0d0s7
---------------------------------------------------------------------------
13. Make sure the VxVM software does not start during the next boot.
Type the following:
# touch /a/etc/vx/reconfig.d/state.d/install-db
This is important because if the root disk contains mirrors and the
system boots, the mirrors are synced; this corrupts the changes just
made.
14. Remove the flag that tells the VxVM software that the root disk is
encapsulated.
# rm /a/etc/vx/reconfig.d/state.d/root-done

Sun Proprietary: Internal Use Only


2-70 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Unencapsulating Boot Disks

15. Reboot the system for changes to take effect. Type the following:
# reboot
Once rebooted, the system comes up in an unencapsulated state with
/, /usr, /var, and swap mounted.

At this point a basic or functional unencapsulation is complete. Do not


leave the system in this state system permanently. It is a state that is
useful for troubleshooting and system maintenance. When problems with
the system are resolved and it is ready to be re-encapsulated, perform the
following series of commands:
# touch /etc/vx/reconfig.d/state.d/root-done
# rm /etc/vx/reconfig.d/state.d/install-db
# cp /a/etc/vfstab.orig /a/etc/vfstab
# cp /a/etc/system.orig /a/etc/system
# reboot

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-71
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exploring Unencapsulation Issues

Exploring Unencapsulation Issues


This section addresses various issues that can affect ununcapsulation of
data and boot disks.

Data Disks
Data disks fail unencapsulation if the following is true:
● Any encapsulated partition was grown or has a modified layout.
● The encapsulated disk failed and was replaced.
● The encapsulated disk was non-conforming and was unencapsulated
using the procedure in ‘‘Encapsulating a Non-Conforming Disk’’ on
page 2-20.

If the data on the disk must be removed from the VxVM software control,
back up the data and restore it to a non-VxVM software disk. Data must
be restored because the encapsulated disk’s original mapping of partitions
to blocks within the public region changes when the disk is replaced and
synced. This prevents the vxmksdpart command from properly mapping
subdisks within the public region to physical partitions.

Boot Disks
The issues described in this section affect boot disk encapsulation.

Grown File Systems

Do not grow any encapsulated boot disk file systems. This causes
unencapsulation to fail. The only recovery option is to back up and restore
the boot disk from tape.

Sun Proprietary: Internal Use Only


2-72 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exploring Unencapsulation Issues

Encapsulated Disk Was Replaced

Unlike data disks, if an encapsulated boot disk fails and is replaced, the
disk can be unencapsulated. This is because the /, /usr, /var, and swap
partitions and any other system partitions (such as /opt or /home)
encapsulated on the boot disk are preserved by the mirror. When the
failed boot disk is replaced and re-synced, these preserved and overlay
partitions are copied to the replacement disk and are then available to
successfully unencapsulate the disk.

Depending on the original configuration of the boot disk and the method
of mirroring, the final unencapsulated partition scheme can change from
the pre-encapsulation configuration, as follows:
● The /, /usr, /var, and swap partitions – There should be no
problems with these partitions; the disk unencapsulates successfully
using both scripted and manual unencapsulation methods. In
particular –
● On a two-slice boot disk (/ and swap partitions) – The
unencapsulated partition scheme is identical to pre-
encapsulation except that slice 0 is moved back one or two
cylinders from the start of the disk. Modifications to the
/etc/vfstab file are unnecessary if manually unencapsulated.
● On three- or four-slice boot disks (/, /usr, /var, and swap
partitions) – The unencapsulated partition scheme is different
from the pre-encapsulated scheme. The /usr and /var
partitions are relocated to partition 6 and 7 if originally
configured in partitions 3 and 4.
Scripted unencapsulation using the vxunroot command
successfully unencapsulates boot disk and modifies the
/etc/vfstab file to reflect the new location of the /usr and
/var partitions. Manual unencapsulation requires manual
modification of the /etc/vfstab file to reflect the new
locations of the /usr and /var partitions.

Caution – If the required modifications to the /etc/vfstab are not made,


the reboot fails. Recovery from this error requires booting from CD-ROM,
mounting the root file system, and editing the /etc/vfstab file.

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-73
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exploring Unencapsulation Issues

● The /, /usr, /var, and swap partitions plus /opt or /home – The
partition scheme can change depending on:
● Scripted unencapsulation using vxunroot works with no
problems. The final partition scheme is changed from the
original, which is reflected in the /etc/vfstab file. The system
boots with no problems.
● Manual unencapsulation can be confusing if the original (pre-
encapsulation) partition scheme is not known. Determining
where the original /, /usr, /var, and swap partitions were
located is easy. Output from the format utility delineates those
partitions. The confusing part is determining which of the
preserved system partitions (such as /opt and /home) is which
if the encapsulated boot disk has both.
Additionally, manual unencapsulation procedures must be
modified to reflect the following:
● The vxmksdpart and the vxedvtoc commands are not
necessary because the disk is already partitioned. All
system partitions (/, /usr, /var, and swap) and
encapsulated system partitions (/opt and /home) are
visible. Additionally, the data these commands use is
invalid.
● Use the format utility only to remove the public or private
region partitions.
● Edit the /etc/vfstab file to reflect the new locations for
any relocated partitions. If /usr was originally in slice 4
and now occupies slice 6, this must be reflected in the
/etc/vfstab file prior to a system reboot.

Note – If the Sun Enterprise Services best practices boot disk management
processes are followed, the partitioning concerns described in
‘‘Encapsulated Disk Was Replaced’’ on page 2-73 are eliminated.

Sun Proprietary: Internal Use Only


2-74 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Encapsulating Disks

Exercise: Encapsulating Disks


In this exercise, you perform the processes required to encapsulate and
unencapsulate disks using the VxVM software commands and utilities.
You will complete the following tasks:
● Install the VxVM software
● Encapsulate a boot disk
● Unencapsulate a boot disk using the vxunroot command
● Unencapsulate a boot disk using manual methods
● Unencapsulate a boot disk that has a replaced primary mirror
● Encapsulate a data disk
● Unencapsulate a data disk
● Encapsulate a non-conforming data disk

Preparation
To prepare for this exercise:
● Identify four disks in addition to the boot disk to use as mirror and
data disks.
● Make sure that the boot disk has the /, swap, /usr, /var, and /opt
partitions.
If the boot disk is not configured this way, have the instructor
perform a re-flash installation on your system using the proper boot
disk configuration for this lab.
● Ask your instructor for the location of the VxVM software packages,
patches, and supporting Solaris OE software.
● Have paper and writing instruments for taking notes.

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-75
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Encapsulating Disks

Task 1 – Installing the VxVM Software and


Encapsulating a Boot Disk
Complete the following steps:
1. Open a text editor such as vi and capture the following pre-
encapsulation boot disk information:
● The prtvtoc value
● The format utility partition print
● The df -k output
● Contents of the /etc/vfstab file
Save the file as /bootdisk_capture. The information in this file is
used later in this lab exercise.
2. Install the VxVM software release 3.2 packages.
3. Install all VxVM software release 3.2 patches as indicated by the
instructor.
4. Use vxinstall to configure the system as follows:
a. Do not use enclosure-based naming.
b. Select custom install.
c. Encapsulate the boot disk as follows:
1. Assign it a VxVM software disk name of rootdisk
(default).
2. Accept the default private region size.
d. Leave all other disks alone.
5. Reboot the system.
6. Open /bootdisk_capture using a text editor and capture the
following post-encapsulation boot disk information:
● The prtvtoc value
● The format utility partition print
● The df -k value
● Contents of the /etc/vfstab file
● The vxprint -qhtg rootdg output

Sun Proprietary: Internal Use Only


2-76 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Encapsulating Disks

7. Mirror the encapsulated boot disk using Sun Enterprise Services best
practices procedure (see ‘‘Examining Sun Enterprise Services Best
Practices for VxVM Software-Managed Boot Disks’’ on page 2-48).
Assign the mirror disk a VxVM software disk name of rootmirror.
8. While the boot disk is being mirrored, answer the following
questions:
a. Describe the difference between the pre- and post-encapsulation
boot disk partition configuration. Use the contents of
/bootdisk_capture as a guide.
________________________________________________________
________________________________________________________
________________________________________________________
________________________________________________________
________________________________________________________
b. What directory contains both pre- and post-encapsulation
configuration information about the boot disk?
________________________________________________________
________________________________________________________
________________________________________________________
c. State the purpose of the following files:
● /etc/vx/reconfig.d/state.d/root-done
_____________________________________________________
_____________________________________________________
● /etc/vx/reconfig.d/state.d/install-db
_____________________________________________________
_____________________________________________________
● /etc/vfstab.prevm
_____________________________________________________
_____________________________________________________
9. After the boot disk is mirrored, use vxprint to capture the post-
mirror configuration. Copy this information to the
/bootdisk_capture file for use later in this lab.

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-77
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Encapsulating Disks

Task 2 – Unencapsulating a Boot Disk Using the


vxunroot Utility
Complete the following steps:
1. Unencapsulate the boot disk by using the vxunroot command.
The procedure for this process is found in ‘‘Examining Sun
Enterprise Services Best Practices for VxVM Software-Managed Boot
Disks’’ on page 2-48, but can be successfully used for a five-slice boot
disk as described in ‘‘Unencapsulating a Boot Disk Using the
vxunroot Utility’’ on page 2-57.
To unencapsulate a five-slice boot disk, use the following command
syntax in place of that listed in the procedure to execute the loop to
remove the mirrors:
# for i in ‘cat /rmsub.plex‘
>do
>vxplex -g rootdg -o rm dis $i
>done
This substitution results in full removal of the plexes and their
subdisks.
2. Was the ununcapsulation successful?
_____________________________________________________________
_____________________________________________________________
3. How do you know, and what commands do you use to verify this?
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________
4. Compare the post-unencapsulation partition configuration of the
boot disk with the pre-unencapsulation and describe the differences.
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________

Sun Proprietary: Internal Use Only


2-78 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Encapsulating Disks

Task 3 – Encapsulating a Boot Disk Using the


vxdiskadm Utility
Complete the following steps:
1. Encapsulate the boot disk using the vxdiskadm utility menu
selection 2. Use the following configuration:
a. Assign it a VxVM software disk name of rootdisk (default).
b. Do not configure it as a spare disk.
c. Accept all other defaults.
2. One the encapsulation process is complete and the system has
rebooted, mirror the boot disk using the Sun Enterprise Services best
practice procedure (see ‘‘Examining Sun Enterprise Services Best
Practices for VxVM Software-Managed Boot Disks’’ on page 2-48).
Assign the mirror disk a VxVM software disk name of rootmirror.

Task 4 – Manually Unencapsulating a Boot Disk When


Booted From the CD-ROM
Complete the following steps:
1. Unencapsulate the boot disk using the manual procedure when
booted from CD-ROM described in ‘‘Unencapsulating When Booted
From the CD-ROM’’ on page 2-64. Be sure to recover the
encapsulated /opt partition using the vxedvtoc command outlined
in this procedure.
Be patient—this procedure reboots the system multiple times.
2. Was the ununcapsulation successful?
_____________________________________________________________
_____________________________________________________________
3. If the unencapsulation was not successful, what do you think went
wrong?
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-79
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Encapsulating Disks

4. Is the mistake recoverable?


_____________________________________________________________
_____________________________________________________________
5. Describe the process used to recover and successfully unencapsulate
the boot disk.
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________
6. Re-encapsulate the boot disk using the vxdiskadm utility.
7. After the system reboots, re-mirror the boot disk using the Sun
Enterprise Services best practices procedure (see ‘‘Examining Sun
Enterprise Services Best Practices for VxVM Software-Managed Boot
Disks’’ on page 2-48). Assign the mirror disk a VxVM software disk
name of rootmirror.

Caution – Do not continue the lab without re-mirroring the boot disk. The
next task requires that the boot disk be mirrored.

Task 5 – Encapsulating a Data Disk Using the


vxdiskadm Utility (Optional)
Complete the following steps:
1. Select a disk to be used as a data (non-root) disk for encapsulation.
2. Use the format utility to create the following partition configuration
for this disk –
● Create two partitions minimum (512 megabyte maximum size).
● Leave partitions 6 and 7 unallocated.
3. Build file systems on each of the partitions.

Sun Proprietary: Internal Use Only


2-80 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Encapsulating Disks

4. Create mount points for each of the partitions on the data disk and
verify that the new file systems successfully mounts.
5. Update the /etc/vfstab file to auto-mount these file systems
during system reboot.
6. Verify that the file systems mount from the /etc/vfstab file.
7. Use prtvtoc, the df command, and the format utility to capture
pre-encapsulation and mount information to the
/datadisk_capture file. Also capture the contents of the
/etc/vfstab file. This information is used later in this lab
exercise.
8. Encapsulate this disk using the vxdiskadm utility.
While encapsulating this disk, you are asked for a disk group for this
disk to join. Create a new disk group called datadg, or use rootdg.
9. After the system reboots, verify that the encapsulation was
successful using the df -k and vxprint commands.
10. Mirror the encapsulated data disk using a disk you identified for this
purpose and the vxdiskadm utility.
11. While the data disk is being mirrored, answer the following
questions:
a. What are the differences between the pre- and post-
encapsulation partition configuration for this disk?
________________________________________________________
________________________________________________________
________________________________________________________
b. How many reboots did the system execute?
________________________________________________________
________________________________________________________
________________________________________________________
c. Using output from the execution of a df -k command, contrast
the differences in the devices used to mount the newly
encapsulated file systems.
________________________________________________________
________________________________________________________
________________________________________________________

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-81
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Encapsulating Disks

d. View the /etc/vfstab file and describe the differences


between the pre- and post-data disk encapsulation content.
________________________________________________________
________________________________________________________
________________________________________________________
12. Wait for the mirror process to complete before proceeding to the next
task.

Task 6 – Unencapsulating a Data Disk (Optional)


Complete the following steps:
1. Unencapsulate the data disk using the procedure outlined in
‘‘Unencapsulating Data Disks’’ on page 2-23.
2. Was the unencapsulation successful?
_____________________________________________________________
_____________________________________________________________
3. If the unencapsulation was not successful, what do you think went
wrong?
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________
4. Is the mistake recoverable?
_____________________________________________________________
_____________________________________________________________
5. Describe the process used to recover and successfully unencapsulate
the data disk.
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________

Sun Proprietary: Internal Use Only


2-82 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Encapsulating Disks

6. Once the unencapsulation successfully completes, contrast the pre-


encapsulation and post-unencapsulation partition scheme of this
disk.
Are there any differences?
_____________________________________________________________
_____________________________________________________________
If yes, list them.
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________
7. Unmount and remove all references to the file systems on this disk
from the /etc/vfstab file.

Task 7 – Encapsulating a Non-Conforming Data Disk


(Optional)
Complete the following steps:
1. Select a disk to be used as a data (non-root) disk for encapsulation.
2. Use the format utility to create the following partition configuration
for this disk:
● Create two partitions, minimum, using all the available space
on the disk. Do not leave any free space.
● Leave at least one partition unused.
3. Build file systems on each of the partitions.
4. Create mount points for each of the partitions on the data disk, and
verify that the new file systems successfully mount.
5. Update the /etc/vfstab file to auto-mount these file systems
during system reboot.
6. Verify that the file systems mount from the /etc/vfstab file.

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-83
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Encapsulating Disks

7. Use prtvtoc, the df command, and the format utility to capture


pre-encapsulation and mount information to the
/datadisk_capture file. Also capture the contents of the
/etc/vfstab file. This information is used later in this lab exercise.
8. Encapsulate this disk using the procedure in ‘‘Encapsulating a Non-
Conforming Disk’’ on page 2-20.

Task 8 – Lab House Cleaning


Complete the following
1. Unencapsulate and remove all references, mount points, and
/etc/vfstab entries for the data disk if the data disk encapsulation
lab tasks were completed.
2. Encapsulate the boot disk if it is not already encapsulated.
3. Mirror the boot disk if it is not already mirrored using the Sun
Enterprise Services best practices procedure (see ‘‘Examining Sun
Enterprise Services Best Practices for VxVM Software-Managed Boot
Disks’’ on page 2-48).

Caution – It is critical for the successful completion of labs in other


modules that you complete the lab house-cleaning tasks. When you are
finished, only the rootdg disk group should exist with the boot disk
encapsulated and mirrored.

Sun Proprietary: Internal Use Only


2-84 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise Summary

Exercise Summary

Discussion – Take a few minutes to discuss what experiences, issues, or


discoveries you had during the lab exercise.
!
?
● Experiences
● Interpretations
● Conclusions
● Applications

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-85
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Encapsulating Disks

Exercise: Encapsulating Disks


Following are the answers to the lab exercises.

Task 1 Solutions
Complete the following steps:
1. Open a text editor such as vi and capture the following pre-
encapsulation boot disk information:
● The prtvtoc value
● The format utility partition print
● The df -k output
● Contents of the /etc/vfstab file
Save the file as /bootdisk_capture. The information in this file is
used later in this lab exercise.
2. Install the VxVM software release 3.2 packages.
3. Install all VxVM software release 3.2 patches as indicated by the
instructor.
4. Use vxinstall to configure the system as follows:
a. Do not use enclosure-based naming.
b. Select custom install.
c. Encapsulate the boot disk as follows:
1. Assign it a VxVM software disk name of rootdisk
(default).
2. Accept the default private region size.
d. Leave all other disks alone.
5. Reboot the system.
6. Open /bootdisk_capture using a text editor and capture the
following post-encapsulation boot disk information:
● The prtvtoc value
● The format utility partition print
● The df -k value

Sun Proprietary: Internal Use Only


2-86 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Encapsulating Disks

● Contents of the /etc/vfstab file


● The vxprint -qhtg rootdg output
7. Mirror the encapsulated boot disk using Sun Enterprise Services best
practices procedure (see ‘‘Examining Sun Enterprise Services Best
Practices for VxVM Software-Managed Boot Disks’’ on page 2-48).
Assign the mirror disk a VxVM software disk name of rootmirror.
8. While the boot disk is being mirrored, answer the following
questions:
a. Describe the difference between the pre- and post-encapsulation
boot disk partition configuration. Use the contents of
/bootdisk_capture as a guide.

There are two new disk partitions for the private and public regions. Additionally,
/opt was encapsulated into the disk’s public region and is no longer visible. The
/, swap, /usr, and /var partitions are still visible as overlay partitions.
b. What directory contains both pre- and post-encapsulation
configuration information about the boot disk?

The /etc/vx/reconfig.d/disk.d/c#t#d# directory.


c. State the purpose of the following files:
● /etc/vx/reconfig.d/state.d/root-done

This file defines that the root disk was encapsulated.


● /etc/vx/reconfig.d/state.d/i/etc/vx/reconfig.d/sta
te.d/install-db

This file defines by its presence that vxinstall was not executed. It prevents the
VxVM software daemons from starting.
● /etc/vfstab.prevm

This file holds a copy of the pre-boot disk encapsulation /etc/vfstab file
contents.
9. After the boot disk is mirrored, use vxprint to capture the post-
mirror configuration. Copy this information to the
/bootdisk_capture file for use later in this lab.

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-87
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Encapsulating Disks

Task 2 Solutions
Complete the following steps:
1. Unencapsulate the boot disk by using the vxunroot command.
The procedure for this process is found in ‘‘Examining Sun
Enterprise Services Best Practices for VxVM Software-Managed Boot
Disks’’ on page 2-48, but can be successfully used for a five-slice boot
disk as described in ‘‘Unencapsulating a Boot Disk Using the
vxunroot Utility’’ on page 2-57.”
To unencapsulate a five-slice boot disk, use the following command
syntax in place of that listed in the procedure to execute the loop to
remove the mirrors:
# for i in ‘cat /rmsub.plex‘
>do
>vxplex -g rootdg -o rm dis $i
>done
This substitution results in full removal of the plexes and their
subdisks.
2. Was the ununcapsulation successful?

It should be.
3. How do you know, and what commands do you use to verify this?

The system reboots. Use the following commands:


● df -k
● vxprint
● mount
4. Compare the post-unencapsulation partition configuration of the
boot disk with the pre-unencapsulation and describe the differences.

Partitions were restored to pre-encapsulation configuration. The public and


private region partitions were removed.

Sun Proprietary: Internal Use Only


2-88 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Encapsulating Disks

Task 3 Solutions
Complete the following steps:
1. Encapsulate the boot disk using the vxdiskadm utility menu
selection 6. Use the following configuration:
a. Assign it a VxVM software disk name of rootdisk (default).
b. Do not configure it as a spare disk.
c. Accept all other defaults.
2. One the encapsulation process is complete and the system has
rebooted, mirror the boot disk using the Sun Enterprise Services best
practice procedure (see ‘‘Examining Sun Enterprise Services Best
Practices for VxVM Software-Managed Boot Disks’’ on page 2-48).
Assign the mirror disk a VxVM software disk name of rootmirror.

Task 4 Solutions
Complete the following steps:
1. Unencapsulate the boot disk using the manual procedure when
booted from CD-ROM described in ‘‘Unencapsulating When Booted
From the CD-ROM’’ on page 2-64. Be sure to recover the
encapsulated /opt partition using the vxedvtoc command outlined
in this procedure.
Be patient—this procedure reboots the system multiple times.
2. Was the ununcapsulation successful?

It should be.

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-89
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Encapsulating Disks

3. If the unencapsulation was not successful, what do you think went


wrong?

It depends on the problem.


4. Is the mistake recoverable?

It depends on the problem.


5. Describe the process used to recover and successfully unencapsulate
the boot disk.

It depends on the problem.


6. Re-encapsulate the boot disk using the vxdiskadm utility.
7. After the system reboots, re-mirror the boot disk using the Sun
Enterprise Services best practices procedure (see ‘‘Examining Sun
Enterprise Services Best Practices for VxVM Software-Managed Boot
Disks’’ on page 2-48). Assign the mirror disk a VxVM software disk
name of rootmirror.

Caution – Do not continue the lab without re-mirroring the boot disk. The
next task requires that the boot disk be mirrored.

Task 5 Solutions
Complete the following steps:
1. Select a disk to be used as a data (non-root) disk for encapsulation.
2. Use the format utility to create the following partition configuration
for this disk –
● Create two partitions minimum (512 megabyte maximum size).
● Leave partitions 6 and 7 unallocated.
3. Build file systems on each of the partitions.
4. Create mount points for each of the partitions on the data disk and
verify that the new file systems successfully mounts.
5. Update the /etc/vfstab file to auto-mount these file systems
during system reboot.

Sun Proprietary: Internal Use Only


2-90 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Encapsulating Disks

6. Verify that the file systems mount from the /etc/vfstab file.
7. Use prtvtoc, the df command, and the format utility to capture
pre-encapsulation and mount information to the
/datadisk_capture file. Also capture the contents of the
/etc/vfstab file. This information is used later in this lab
exercise.
8. Encapsulate this disk using the vxdiskadm utility.
While encapsulating this disk, you are asked for a disk group for this
disk to join. Create a new disk group called datadg, or use rootdg.
9. After the system reboots, verify that the encapsulation was
successful using the df -k and vxprint commands.
10. Mirror the encapsulated data disk using a disk you identified for this
purpose and the vxdiskadm utility.
11. While the data disk is being mirrored, answer the following
questions:
a. What are the differences between the pre- and post-
encapsulation partition configuration for this disk?

The post-encapsulated disk has only partitions 6 and 7. The original partitions
were encapsulated in slice 6, the public region. Slice 7 is the private region.
b. How many reboots did the system execute?

One.
c. Using output from the execution of a df -k command, contrast
the differences in the devices used to mount the newly-
encapsulated file systems.

Original devices were normal Solaris OE devices which use


/dev/dsk/c#t#d#s# addresses. Post-encapsulation devices use the VxVM
software volume names. These are /dev/vx/dsk/data1 or other volume names.
d. View the /etc/vfstab file and describe the differences
between the pre- and post-data disk encapsulation content.

The /etc/vfstab file now uses the VxVM software volumes as devices to
mount and fsck. The original device information was saved as comments at the
end of the file.
12. Wait for the mirror process to complete before proceeding to the next
task.

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-91
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Encapsulating Disks

Task 6 Solutions
Complete the following steps:
1. Unencapsulate the data disk using the procedure outlined in
‘‘Unencapsulating Data Disks’’ on page 2-23.
2. Was the unencapsulation successful?

It should be.
3. If the unencapsulation was not successful, what do you think went
wrong?

It depends on the problem.


4. Is the mistake recoverable?

It depends on the problem.


5. Describe the process used to recover and successfully unencapsulate
the data disk.

It depends on the problem.


6. Once the unencapsulation successfully completes, contrast the pre-
encapsulation and post-unencapsulation partition scheme of this
disk.
Are there any differences?

No.
If yes, list them.

There should not be any differences.


7. Unmount and remove all references to the file systems on this disk
from the /etc/vfstab file.

Sun Proprietary: Internal Use Only


2-92 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Encapsulating Disks

Task 7 Solutions
Complete the following steps:
1. Select a disk to be used as a data (non-root) disk for encapsulation.
2. Use the format utility to create the following partition configuration
for this disk:
● Create two partitions, minimum, using all the available space
on the disk. Do not leave any free space.
● Leave at least one partition unused.
3. Build file systems on each of the partitions.
4. Create mount points for each of the partitions on the data disk, and
verify that the new file systems successfully mount.
5. Update the /etc/vfstab file to auto-mount these file systems
during system reboot.
6. Verify that the file systems mount from the /etc/vfstab file.
7. Use prtvtoc, the df command, and the format utility to capture
pre-encapsulation and mount information to the
/datadisk_capture file. Also capture the contents of the
/etc/vfstab file.
8. Encapsulate this disk using the procedure in ‘‘Encapsulating a Non-
Conforming Disk’’ on page 2-20.

Sun Proprietary: Internal Use Only


Encapsulating Disks 2-93
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Module 3

Managing Dynamic Multi-Pathing

Objectives
Upon completion of this module, you should be able to:
● Define and explain how the dynamic multi-pathing (DMP) functions
enhance the availability and accessibility of VxVM software-
managed storage devices
● Explain how DMP identifies disks in both pre- and post-version 3.2
of the VxVM software
● Describe how to install and verify DMP
● Enable and disable multi-pathing to selected disks and controllers
● Administrate DMP by using the vxdmpadm utility
● Perform start restore and stop restore functions
● Identify common DMP problems

Sun Proprietary: Internal Use Only


3-1
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Relevance

Relevance

Discussion – The following questions are relevant to understanding what


DMP and DMP administration is all about:
!
?
● What are the key features of DMP?
● How does DMP manage load balancing?
● How does DMP improve the fault resiliency of system storage?
● How is DMP functionality enabled and disabled on a controller and
on individual disks?
● How does DMP recognize disks?
● How does DMP interface with the DDL function introduced in the
VxVM software version 3.2?
● Is DMP compatible with other Sun Microsystems multi-pathing
applications?
● Is DMP compatible with the Sun Microsystems dynamic
reconfiguration (DR) software?
● How does DMP interface with Sun StorEdge T3 storage arrays, and
how is this different than DMP support of the Sun StorEdge A5000
series of array storage subsystems?

Sun Proprietary: Internal Use Only


3-2 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Additional Resources

Additional Resources

Additional resources – The following references provide additional


details on the topics discussed in this module:
● VERITAS Volume Manager™ 3.2 Administrator’s Guide. Mountain
View, California: VERITAS Software Corporation, August 2001,
number 30-000392-011, TechPDF ID 240253.
● VERITAS Volume Manager™ 3.2 Installation Guide. Mountain View,
California: VERITAS Software Corporation, August 2001,
number 30-000395-011, TechPDF ID 240256.
● VERITAS Volume Manager™ 3.2 Troubleshooting Guide. Mountain
View, California: VERITAS Software Corporation, August 2001,
number 30-000394-011, TechPDF ID 240255.
● VERITAS Volume Manager™ Storage Administrator 3.2 Administrator’s
Guide. Mountain View, California: VERITAS Software Corporation,
July 2001, number 30-000393-011, TechPDF ID 240257.
● http://storage.east, http://storage.central, and
http://storage.west
● SunSolveSM Online SRDBs and INFODOC 18314,
[http://sunsolve.Sun.COM/pub-cgi/search.pl?mode=
advanced].
● VERITAS Software Corporation support knowledge base,
[http://seer.support.veritas.com/nav_bar/index.asp?
content_sURL=%2Fsearch%5Fforms%2Ftechsearch%2Easp].
● Man pages for the vxdmpadm command.

Sun Proprietary: Internal Use Only


Managing Dynamic Multi-Pathing 3-3
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining VxVM Software and Dynamic Multi-Pathing

Examining VxVM Software and Dynamic Multi-Pathing


DMP allows the Solaris OE to see only disks and not the multiple paths to
the disks. It handles the I/O across these multiple paths transparently to
the Solaris OE and the user. Additionally, DMP manages load balancing
for I/Os across the multiple paths.

This feature is only available for Active/Active DMP arrays; it is not


available for active/passive DMP arrays. DMP also handles the failover
from one path to another, transparent to the Solaris OE and the user.

DMP is an important part of the VxVM software’s overall storage fault


resiliency strategy and is a mandatory part of the installation of the VxVM
software version 3.1.x and later. This section describes the VxVM software
and how it implements DMP functions.

VxVM Software Architecture and DMP


Figure 3-1 illustrates the relationship of the DMP driver within the system
software stack, in which the DMP driver is a component of the kernel
level of the software stack.

Applications
User I/O
Kernel
File System

VxVM software (vxio)


/dev/vx/dmp/c1t1d0s0
(DMP meta node)
DMP

/dev/dsk/c1t1d0s0 /dev/dsk/c2t1d0s0
Target Disk Driver

Target HBA Driver


Physical

Path 1 H i ta c h i D a t a S y s te m s

R E A D YW A R N I N G A L A R MP O W E R
SD
Path 2

Figure 3-1 System Software Stack

Sun Proprietary: Internal Use Only


3-4 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining VxVM Software and Dynamic Multi-Pathing

Load Balancing
Load balancing is the function which attempts to maximize I/O
throughput by using the full bandwidth of all paths. Although the goal is
the same, load balancing is implemented differently depending on which
version of the VxVM software is used.

Load Balancing in Software Version 2.5.x

Load balancing in the VxVM software version 2.5.x is achieved by routing


random I/O across all available paths in a round-robin fashion. For the
Sun StorEdge A5000 series of array storage subsystems, any I/O that is
within 64 kilobytes on either side of the last I/O is sent down the same
path, which results in more effective use of cache.

Early versions of the VxVM software (before 2.5.4 for example) allow the
user to disable DMP. Version 2.5.4 of the VxVM software does not allow
the user to disable DMP. While some systems can disable DMP, this causes
vxconfigd to fail.

Load Balancing in Software Version 3.0

Load balancing in the VxVM software version 3.0 is achieved by using the
balanced path policy for the active/active-type disk arrays only, and not for
active/passive-type disk arrays.

The method used to balance random I/Os across all the available paths is
to send all I/Os that start within a single 64-kilobyte block range down
the same path. I/Os that start within other 64-kilobyte ranges on the disk
are routed through a different path.

If uneven distribution of I/Os between the controllers occurs, it may be


caused by a contiguous read of data from the disk within a single
64-kilobyte block range. The term load balancing is misleading in this case
because it refers to I/O distribution, depending on the location of the data
on the disk.

Sun Proprietary: Internal Use Only


Managing Dynamic Multi-Pathing 3-5
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining VxVM Software and Dynamic Multi-Pathing

Unique Disk Identifier


The Solaris OE requires a unique means to identify each disk drive. To
that end, disk manufacturers provide a serial number along with other
information in the standard inquiry data for each disk. It is possible to
view this information by using the vxdmpinq script. The following
example shows output from the vxdmpinq script.
bash-2.03# cd /etc/vx/diag.d
bash-2.03# ./vxdmpinq /dev/rdsk/c1t0d0s2
Vendor id : SEAGATE
Product id : ST19171FCSUN9.0G
Revision : 177E
Serial Number : 9831X06256

DMP Device Paths


The DMP implemented in the VxVM software version 2.5.x supports up
to 42 paths. On the VxVM software version 3.0.x and 3.2, DMP supports
an unlimited number of paths.

For each device path, the Solaris OE creates an entry in the dev_info tree.
DMP scans the dev_info tree and creates metanodes in the /dev/vx tree.
The VxVM software uses the metanodes to access the disk.

Solaris OE Drives and DMP Drives


The Solaris OE sees all the paths to each drive and lists them separately in
the /dev/dsk tree. An example illustrates two links, c1t1d0s0 and
c2t1d0s0, each pointing to the same drive (3f643).
bash-2.03# ls -las /dev/dsk/*t0d0s0
2 lrwxrwxrwx 1 root root 70 Apr 21 15:22 /dev/dsk/c1t0d0s0 ->
../../devices/sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w210000203713f643,0:a
2 lrwxrwxrwx 1 root root 70 Apr 21 15:22 /dev/dsk/c2t0d0s0 ->
../../devices/sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w220000203713f643,0:a

Conversely, DMP creates one metanode per LUN. That is, DMP identifies
LUNs and creates a metanode for each LUN. The result is that a disk with
multiple paths is seen as a single disk by the VxVM software.

Sun Proprietary: Internal Use Only


3-6 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining VxVM Software and Dynamic Multi-Pathing

A LUN can be a physical disk as in the individual disks of a Sun StorEdge


A5000 series of array storage subsystems, or it may be a more complex
configuration of multiple physical disks as in a software RAID device
such as a Sun StorEdge T3 storage array. The operating system (and
therefore the user) sees no difference between a LUN and a disk.

For example, an array with five LUNs has five metanodes. If there are
four paths from the host to the array, the host has 20 entries under
/dev/dsk tree, and only five entries under /dev/vx/dmp tree.

The following example illustrates multiple paths to a single disk in a Sun


StorEdge A5000 Sun Network Storage Array (SENA), with two interface
boards (IBs) and two gigabit interface converters (GBICs).
bash-2.03# ls -las /dev/rdsk/*t0d0s0
2 lrwxrwxrwx 1 root root 74 Apr 21 15:22 /dev/rdsk/c1t0d0s0 ->
../../devices/sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w210000203713f643,0:a,raw
2 lrwxrwxrwx 1 root root 74 Apr 21 15:22 /dev/rdsk/c2t0d0s0 ->
../../devices/sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w220000203713f643,0:a,raw

In the following example, the same disks are managed by DMP, which
arbitrarily chooses one of the two paths and creates a single device entry.
To see the multiple paths to individual disks, use the vxdmpadm command.
bash-2.03# ls -las /dev/vx/rdmp/*t0d0s0
0 crw------- 1 root other 74, 0 May 21 15:34 /dev/vx/rdmp/c1t0d0s0

The following example shows output from the vxdisk list command to
illustrate how the VxVM software sees the same disk shown in the
previous examples.
bash-2.03# vxdisk list
DEVICE TYPE DISK GROUP STATUS
c1t0d0s2 sliced rootdisk rootdg online
c1t3d0s2 sliced - - error
c1t4d0s2 sliced - - online
c1t5d0s2 sliced - - online
c1t6d0s2 sliced - - error
c1t16d0s2 sliced - - error
c1t18d0s2 sliced - - error
c1t19d0s2 sliced - - error
c1t22d0s2 sliced rootmirror rootdg online

Sun Proprietary: Internal Use Only


Managing Dynamic Multi-Pathing 3-7
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining VxVM Software and Dynamic Multi-Pathing

DMP and Device Discovery


In early versions of the VxVM software, DMP was responsible for device
discovery. With the release of the VxVM software version 3.2 and
implementation of the DDL feature, DMP is no longer responsible for
device discovery. This is now handled by the DDL feature. This process is
discussed in Appendix E, “Configuring the VxVM Software."

Sun Proprietary: Internal Use Only


3-8 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Installing and Verifying DMP

Installing and Verifying DMP


The DMP driver is installed into the in /kernel/drv directory during the
installation of the VRTSvxvm package. During installation, a force load is
added to the /etc/system file which force loads the drivers at boot time to
allow the system to boot from a disk under VxVM software control.

With the release of the VxVM software version 3.1, DMP is always
installed and cannot be disabled or the VxVM software does not start. The
process of selectively disabling DMP on a controller or drive basis is
described in ‘‘Enabling and Disabling DMP’’ on page 3-10.

To verify the presence of the drivers, use the modinfo | grep vx


command as seen in this example.
bash-2.03# modinfo | grep vx
19 101e9025 fe998 75 1 vxio (VxVM 3.2t_p1.6 I/O driver)
21 102d43c8 17308 74 1 vxdmp (VxVM 3.2t_p1.6: DMP Driver)
22 102e96f0 837 79 1 vxspec (VxVM 3.2t control/status driver)

One way to verify that the DMP driver is installed correctly and that it is
not corrupt is to compare the size of the DMP driver file to that of the
driver_OSversion, which is in the same directory. Both drivers should
be the same size as seen in this example. The /kernel/drv file holds the
32-bit drivers and /kernel/drv/sparcv9 holds the 64-bit driver.
bash-2.03# pwd
/kernel/drv
bash-2.03# ls -las vxdmp*
640 -rw-r--r-- 1 root sys 314156 Nov 21 2001 vxdmp
608 -rw-r--r-- 1 root sys 297920 Aug 15 2001 vxdmp.SunOS_5.6
608 -rw-r--r-- 1 root sys 300980 Aug 15 2001 vxdmp.SunOS_5.7
640 -rw-r--r-- 1 root sys 314156 Nov 21 2001 vxdmp.SunOS_5.8
4 -rw-r--r-- 1 root sys 1026 Aug 15 2001 vxdmp.conf

bash-2.03# pwd
/kernel/drv/sparcv9
bash-2.03# ls -las vxdmp*
800 -rw-r--r-- 1 root sys 393968 Nov 21 2001 vxdmp
768 -rw-r--r-- 1 root sys 380840 Aug 15 2001 vxdmp.SunOS_5.7
800 -rw-r--r-- 1 root sys 393968 Nov 21 2001 vxdmp.SunOS_5.8

Sun Proprietary: Internal Use Only


Managing Dynamic Multi-Pathing 3-9
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Enabling and Disabling DMP

Enabling and Disabling DMP


In early versions of the VxVM software, DMP was disabled for a number
of different reasons. The most notable reason for disabling DMP was that
the Sun Microsystems alternate pathing (AP) application was installed.
With the release of the VxVM software version 3.1, the incompatibilities
between DMP and AP are resolved, and DMP is no longer disabled.

Caution – AP must be at level 2.3.1 with patch 110722-01 installed.

Additionally, with the release of the VxVM software version 3.1, DMP is a
mandatory component of the VxVM software and must be operational or
the VxVM software is disabled. This is due to new features which are
introduced in the VxVM software version 3.2, such as enclosure-based
naming, which require DMP to function.

Disabling DMP in Version 3.0 and Earlier


With early versions of the VxVM software, DMP could be disabled by
locating the /kernel/drv/ap file prior to the installation of the VxVM
software. If the VxVM software identified the /kernel/drv/ap file, it
would not install. The procedure to disable DMP after installing the
VxVM software is more complex. Refer to SunSolve INFODOC 18314 for
more information.

The disadvantage of this all or none implementation of DMP is that disks


that are not under AP or other third-party multi-pathing software cannot
take advantage of DMP. The release of the VxVM software version 3.1
implementation of DMP changed this to allow selective disabling of the
function for controllers or individual disk devices.

Disabling DMP in Version 3.1 and Later


Disabling DMP in version 3.1 and 3.2 of the VxVM software is
accomplished during the software installation process using the
vxinstall utility or, post-installation, using the vxdiskadm utility. The
disabling or enabling of multi-pathing through DMP is achieved on a
controller-by-controller or disk-by-disk basis. This is a granular, efficient
process that addresses the all or none strategy of previous versions of the
VxVM software.

Sun Proprietary: Internal Use Only


3-10 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Enabling and Disabling DMP

One of the challenges of implementing DMP in this fashion is that, if


multi-pathing is disabled or prevented for a disk, the VxVM software sees
both paths for that disk. This can lead to administrative confusion. To
resolve this problem the VxVM software suppresses the second path
through options provided in the vxinstall and vxdiskadm utilities.

DMP Terminology

Both vxinstall and vxdiskadm use the following terminology to


describe enabling and disabling DMP on selected devices:
● Exclude – Excludes a disk or controller from DMP control. Excludes
have the following options:
● Prevent – Prevents multi-pathing operations
● Suppress – Suppresses the VxVM software operations on the
secondary paths to a device
● Include – Enables DMP operations. Includes have the following
options:
● Allow – Allows multi-pathing operations
● Un-Suppress – Makes secondary paths visible to the VxVM
software operations

When you are disabling DMP on a device or group of devices, the prevent
operation executes first, and the suppress operation executes second.

When you are enabling DMP, the un-suppress operation occurs first and
the allow operation occurs second.

Disabling DMP Using the vxinstall Utility

When you are installing the VxVM software, the vxinstall utility
performs the initial VxVM software setup tasks. One of these tasks is to
prevent multi-pathing and suppress devices from the VxVM software’s
view.

To disable DMP using the vxinstall utility, use option 3, Prevent Multi-
pathing/Suppress devices from VxVMs view, from the main vxinstall
menu. An example shows the process used in vxinstall.
Volume Manager Installation

Menu: VolumeManager/Install

Sun Proprietary: Internal Use Only


Managing Dynamic Multi-Pathing 3-11
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Enabling and Disabling DMP

You will now be asked if you wish to use Quick Installation or Custom
Installation. Custom Installation allows you to select how the Volume
Manager will handle the installation of each disk attached to your
system.

Quick Installation examines each disk attached to your system and


attempts to create volumes to cover all disk partitions that might be
used for file systems or for other similar purposes.

If you want to exclude any devices from being seen by VxVM or not be
multipathed by VxDMP then use the Prevent multipathing/Suppress
devices from VxVMs view option, before you choose Custom
Installation
or Quick Installation.

If you do not wish to use some disks with the Volume Manager, or if
you wish to reinitialize some disks, use the Custom Installation
option. Otherwise, we suggest that you use the Quick Installation
option.

Hit RETURN to continue.

Press Return to continue.


1 Quick Installation
2 Custom Installation
3 Prevent multipathing/Suppress devices from VxVMs view
? Display help about menu
?? Display help about menuing system
q Exit
from menus

When option 3 is selected, the following menu is displayed:


Volume Manager Device Operations

Menu: VolumeManager/Install/Exclude Devices

1 Suppress all paths through a controller from VxVMs view


2 Suppress a path from VxVMs view
3 Suppress disks from VxVMs view by specifying a VID:PID combination
4 Suppress all but one path to a disk
5 Prevent multipathing of all disks on a controller by VxVM
6 Prevent multipathing of a disk by VxVM
7 Prevent multipathing of disks by specifying a VID:PID combination
8 List currently suppressed/non-multipathed devices
? Display help about menu
??Display help about the menuing system
q Exit from menus

Sun Proprietary: Internal Use Only


3-12 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Enabling and Disabling DMP

Options 1 through 4 on the vxinstall utility main menu are used for
suppression operations. Options 6 through 8 are used for prevention
operations. Option 8 lists suppressed or non-multi-pathed devices. To use
this options, select the menu option for the specific operation and follow
the scripted prompts.

Whenever multi-pathing is prevented and secondary paths are


suppressed, the system must be rebooted for these changes to take effect.

Disabling DMP Using the vxdiskadm Utility

To disable multi-pathing or secondary paths after installing the software,


use the vxdiskadm utility. The terminology and procedures used to
exclude and include multi-pathing in the vxinstall utility apply to
vxdiskadm utility.

This example shows the menus used to accomplish multi-pathing and


suppression activities from the vxdiskadm utility.
Volume Manager Support Operations
Menu: VolumeManager/Disk

1 Add or initialize one or more disks


2 Encapsulate one or more disks
3 Remove a disk
4 Remove a disk for replacement
5 Replace a failed or removed disk
6 Mirror volumes on a disk
7 Move volumes from a disk
8 Enable access to (import) a disk group
9 Remove access to (deport) a disk group
10 Enable (online) a disk device
11 Disable (offline) a disk device
12 Mark a disk as a spare for a disk group
13 Turn off the spare flag on a disk
14 Unrelocate subdisks back to a disk
15 Exclude a disk from hot-relocation use
16 Make a disk available for hot-relocation use
17 Prevent multipathing/Suppress devices from VxVM’s view
18 Allow multipathing/Unsuppress devices from VxVM’s view
19 List currently suppressed/non-multipathed devices
20 Change the disk naming scheme
21 Get the newly connected/zoned disks in VxVM view
list List disk information

Sun Proprietary: Internal Use Only


Managing Dynamic Multi-Pathing 3-13
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Enabling and Disabling DMP

? Display help about menu


?? Display help about the menuing system
q Exit from menus

Select an operation to perform:

Use the vxdiskadm utility menu items 17 and 18 to include and exclude
DMP operations. Selecting option 17, Prevent multipathing/Suppress
devices from VxVM’s view, produces the following output:
Exclude Devices
Menu: VolumeManager/Disk/ExcludeDevices

This operation might lead to some devices being suppressed from VxVM’s view
or prevent them from being multipathed by vxdmp (This operation can be
reversed using the vxdiskadm command).

Do you want to continue ? [y,n,q,?] (default: y)

Volume Manager Device Operations


Menu: VolumeManager/Disk/ExcludeDevices

1 Suppress all paths through a controller from VxVM’s view


2 Suppress a path from VxVM’s view
3 Suppress disks from VxVM’s view by specifying a VID:PID combination
4 Suppress all but one paths to a disk
5 Prevent multipathing of all disks on a controller by VxVM
6 Prevent multipathing of a disk by VxVM
7 Prevent multipathing of disks by specifying a VID:PID combination
8 List currently suppressed/non-multipathed devices
? Display help about menu
?? Display help about the menuing system
q Exit from menus

Select an operation to perform:

To use these options, select the menu option for the specific operation and
follow the scripted prompts. A system reboot is required to activate the
changes.

Selecting the vxdiskadm utility menu option 18, Allow


multi-pathing/Unsuppressed devices from VxVM’s view, produces the
following output:
Include Devices
Menu: VolumeManager/Disk/IncludeDevices

The devices selected in this operation will become visible to VxVM and/or
will be multipathed by vxdmp again. Only those devices which were previously
excluded can be included again.

Sun Proprietary: Internal Use Only


3-14 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Enabling and Disabling DMP

Do you want to continue ? [y,n,q,?] (default: y)

Volume Manager Device Operations


Menu: VolumeManager/Disk/IncludeDevices

1 Unsuppress all paths through a controller from VxVM’s view


2 Unsuppress a path from VxVM’s view
3 Unsuppress disks from VxVM’s view by specifying a VID:PID combination
4 Remove a pathgroup definition
5 Allow multipathing of all disks on a controller by VxVM
6 Allow multipathing of a disk by VxVM
7 Allow multipathing of disks by specifying a VID:PID combination
8 List currently suppressed/non-multipathed devices
? Display help about menu
?? Display help about the menuing system
q Exit from menus

Select an operation to perform:

To use these options, select the menu option for the specific operation and
follow the scripted prompts. A system reboot is required to activate the
changes.

Files Related to Disabling and Suppression


The files in the following list are updated by vxinstall and vxdiskadm
when multi-pathing is disabled and secondary paths suppressed:
bash-2.03# ls -las /etc/vx/*exclude
2 -rw-r--r-- 1 root other 59 May 21 15:34 /etc/vx/vxdmp.exclude
2 -rw-r--r-- 1 root other 59 May 21 15:34 /etc/vx/vxvm.exclude

When vxinstall or vxdiskadm performs any exclude or include


operations, these files are modified. The /etc/vx/vxdmp.exclude file is
updated for DMP prevent or allow operations. The
/etc/vx/vxvm.exclude file is updated for suppress and un-suppress
operations.

The following example shows the /etc/vx/vxdmp.exclude after multi-


pathing is disabled on disk c1t0d0.
vxdmp.exclude
:::::::::::::::
exclude_all 0
paths
c1t0d0s2 /sbus@2,0/SUNW,socal@d,10000/sf@0,0/ssd@w210000203713fc9f,0
c2t0d0s2 /sbus@2,0/SUNW,socal@d,10000/sf@1,0/ssd@w220000203713fc9f,0

Sun Proprietary: Internal Use Only


Managing Dynamic Multi-Pathing 3-15
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Enabling and Disabling DMP

#
controllers
#
product
#
pathgroups
#

The following example shows the /etc/vx/vxvm.exclude file when the


secondary path (c2t0d0) for disk c1t0d0 is suppressed.
vxvm.exclude
::::::::::::::
exclude_all 0
paths
c2t0d0s2 /sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w220000203713fc9f,0
#
controllers
#
product
#
pathgroups
#

Caution – Do not edit these files manually. Use vxdiskadm to make all
updates to these files. If it is necessary to edit these file manually be very
careful and only delete the lines needed to get an inoperable system into
service.

Sun Proprietary: Internal Use Only


3-16 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Administrating DMP With the vxdmpadm Command

Administrating DMP With the vxdmpadm Command


The vxdmpadm command is the primary administrative interface to the
DMP facility. Use the vxdmpadm command to enable and disable DMP on
an interface-by-interface basis, display multi-pathing information, and
perform many other functions that affect the operation of DMP or display
DMP status.

To view help for the vxdmpadm command, use the man pages or enter:
# vxdmpadm help

This section describes vxdmpadm options.

The listctrl Option


The vxdmpadm command listctrl option lists all controllers seen by the
Solaris OE. This allows the user to determine which controller is not
communicating with the system. An example follows.
# vxdmpadm listctlr all
CTLR-NAME ENCLR-TYPE STATE ENCLR-NAME
=====================================================
c1 SENA ENABLED SENA0
c2 SENA ENABLED SENA0

Viewing Multi-Pathing Status


Use the vxdmpadm listexclude command to list disks that are excluded
from DMP operations.

An example of the vxdmpadm listexclude command shows that path


c2t0d0 (the secondary path to c1t0d0) is suppressed, and paths c1t0d0
and c2t0d0 are excluded from DMP operations.
# vxdmpadm listexclude

Devices excluded from VxVM:


--------------------------
Paths : c2t0d0

Controllers : None

VID:PID : None

Sun Proprietary: Internal Use Only


Managing Dynamic Multi-Pathing 3-17
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Administrating DMP With the vxdmpadm Command

Devices excluded from multipathing by vxdmp:


-------------------------------------------
Paths : c1t0d0 c2t0d0

VID:PID : None

Pathgroups : None
----------

An example of the vxdisk list command shows that c1t0d0s2 is still


usable by the VxVM software but the secondary path c2t1d0s2 is
suppressed and not visible to the software.
.bash-2.03# vxdisk list
DEVICE TYPE DISK GROUP STATUS
c1t0d0s2 sliced rootdisk rootdg online
c1t3d0s2 sliced - - error
c1t4d0s2 sliced - - online
c1t5d0s2 sliced - - online
c1t6d0s2 sliced - - error
c1t16d0s2 sliced - - error
c1t18d0s2 sliced - - error
c1t19d0s2 sliced - - error
c1t22d0s2 sliced rootmirror rootdg online

The getsubpaths Option


The getsubpaths ctlr option of the vxdmpadm command lists all
subpaths through a specific controller. This allows the system
administrator to determine which DMP nodes (devices) on the path are
not functioning. An example is:
# vxdmpadm getsubpaths ctlr=c1
NAME STATE PATH-TYPE DMPNODENAME ENCLR-TYPE ENCLR-NAME
======================================================================
c1t0d0s2 ENABLED - c1t0d0s2 SENA SENA0
c1t21d0s2 ENABLED - c1t21d0s2 SENA SENA0
c1t17d0s2 ENABLED - c1t17d0s2 SENA SENA0
c1t3d0s2 ENABLED - c1t3d0s2 SENA SENA0
c1t4d0s2 ENABLED - c1t4d0s2 SENA SENA0
c1t6d0s2 ENABLED - c1t6d0s2 SENA SENA0
c1t1d0s2 ENABLED - c1t1d0s2 SENA SENA0
c1t18d0s2 ENABLED - c1t18d0s2 SENA SENA0
c1t20d0s2 ENABLED - c1t20d0s2 SENA SENA0
c1t22d0s2 ENABLED - c1t22d0s2 SENA SENA0
c1t16d0s2 ENABLED - c1t16d0s2 SENA SENA0
c1t2d0s2 ENABLED - c1t2d0s2 SENA SENA0
c1t5d0s2 ENABLED - c1t5d0s2 SENA SENA0
c1t19d0s2 ENABLED - c1t19d0s2 SENA SENA0

Sun Proprietary: Internal Use Only


3-18 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Administrating DMP With the vxdmpadm Command

It is also possible to list the subpaths to a particular node using the


getsubpaths dmpnodename option. This helps to focus on a specific node
by eliminating unwanted information and showing the status of the paths
for that node, as seen in this example.
# vxdmpadm getsubpaths dmpnodename=c1t1d0s2
NAME STATE PATH-TYPE CTLR-NAME ENCLR-TYPE ENCLR-NAME
====================================================================
c1t1d0s2 ENABLED - c1 SENA SENA0
c2t1d0s2 ENABLED - c2 SENA SENA0

The getdmpnode Option


To obtain DMP node information about disks in a specific enclosure, use
the vxdmpadm command getdmpnode option. The getdmpnode option
shows devices within a specific enclosure and the status of the paths to
each node within that enclosure. This example shows how to further
refine the information to a specific node.
# bash-2.03# vxdmpadm getdmpnode enclosure=SENA0
NAME STATE ENCLR-TYPE PATHS ENBL DSBL ENCLR-NAME
=========================================================================
c1t0d0s2 ENABLED SENA 2 2 0 SENA0
c1t21d0s2 ENABLED SENA 2 2 0 SENA0
c1t17d0s2 ENABLED SENA 2 2 0 SENA0
c1t3d0s2 ENABLED SENA 2 2 0 SENA0
c1t4d0s2 ENABLED SENA 2 2 0 SENA0
c1t6d0s2 ENABLED SENA 2 2 0 SENA0
c1t1d0s2 ENABLED SENA 2 2 0 SENA0
c1t18d0s2 ENABLED SENA 2 2 0 SENA0
c1t20d0s2 ENABLED SENA 2 2 0 SENA0
c1t22d0s2 ENABLED SENA 2 2 0 SENA0
c1t16d0s2 ENABLED SENA 2 2 0 SENA0
c1t2d0s2 ENABLED SENA 2 2 0 SENA0
c1t5d0s2 ENABLED SENA 2 2 0 SENA0
c1t19d0s2 ENABLED SENA 2 2 0 SENA0

The disable Opt ion


The vxdmpadm command disable option disables one of the paths to a
disk. This can be advantageous when you are using DMP for multi-
pathing disks on a DR-capable system to disable a path when a system
board is removed. An example shows the results of using the disable
option.
# vxdmpadm disable ctlr=c1
Jul 8 12:11:54 lowtide vxdmp: NOTICE: vxvm:vxdmp: disabled controller
/sbus@3,0/SUNW,socal@0,0/sf@0,0 connected to disk array 5080020000034908

Sun Proprietary: Internal Use Only


Managing Dynamic Multi-Pathing 3-19
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Administrating DMP With the vxdmpadm Command

An example of a vxdmpadm listctlr all command shows the path


disabled.
# vxdmpadm listctlr all
bash-2.03# vxdmpadm listctlr all
CTLR-NAME ENCLR-TYPE STATE ENCLR-NAME
=====================================================
c1 SENA DISABLED SENA0
c2 SENA ENABLED SENA0

The enable Option


The enable option of the vxdmpadm command enables a controller to a
disk. This option can be used on a DR-capable system after a system
board is replaced, to re-enable a path disabled during the DR remove
operation. The following example shows the result of using the enable
option.
# vxdmpadm enable ctlr=c1
Jul 8 12:15:30 lowtide vxdmp: NOTICE: vxvm:vxdmp: enabled controller
/sbus@3,0/SUNW,socal@0,0/sf@0,0 connected to disk array 5080020000034908

An example of a vxdmpadm listctlr command all shows the paths


enabled.
# vxdmpadm listctlr all
CTLR-NAME ENCLR-TYPE STATE ENCLR-NAME
=====================================================
c1 SENA ENABLED SENA0
c2 SENA ENABLED SENA0

The start restore and stop restore Options


The DMP restore daemon thread is a background process that monitors
paths that fail. If the paths are available for use, the daemon thread can be
stopped and re-started using the following syntax:
vxdmpadm start restore interval=seconds

The interval option specifies the time interval after which the daemon
thread checks the failed paths, as specified in seconds. The default is
300 seconds (5 minutes). To not wait for the command to time out, use the
following command:
# vxdctl enable

Sun Proprietary: Internal Use Only


3-20 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Reviewing Common DMP Problems

Reviewing Common DMP Problems


This section provides information on some common problems that might
occur with DMP functions.

Disks Do Not Appear to Be Multi-Pathed


If disks on a system do not appear to be multi-pathed, there can be a
licensing problem. Check the following:
● For the VxVM software version 2.5.x, EMC Corporation disk arrays
require a license.
● Use vxdmpdebug and check the following:
DEBUG: is_module_present: vxdmp module is present
DEBUG: is_module_present: vxdmp module is loaded
DEBUG: is_module_present: vxdmp module is installed
DEBUG: dmp_is_link: /dev/vx/dmp/ is NOT a link
DEBUG: dmp_is_link: /dev/vx/rdmp/ is NOT a link
DEBUG: Have Simple disk DMP license
DEBUG: No SSA DMP license
DEBUG: dmp_set_lic: Start
DEBUG: dmp_set_lic: Done

Serial Number Problems


If a disk manufacturer does not provide unique serial numbers for disks
and a simple disk license is present on a system running the VxVM
software version 2.5.x, DMP does not see the disks correctly on the
system. This causes unexpected behavior, such as seeing multiple
independent paths to multiple disks.

The vxdmpinq command shows serial number information about DMP


disks. See the ‘‘Unique Disk Identifier’’ on page 3-6 for more information.

Sun Proprietary: Internal Use Only


Managing Dynamic Multi-Pathing 3-21
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Reviewing Common DMP Problems

The product_id and vendor_id Do Not Match


Check the following characteristics in the values for product_id and
vendor_id do not match:
● The disks are not multi-pathed.
● The OTHER category is used.

VxVM Software Does Not See Disk Devices


Check the following if the VxVM software does not see disks:
● Make sure that the operating system can see these disks.
● Make sure that I/O can be performed on the disks.

Use these DMP utilities to gather more information:


● vxdmpinq
● vxdmpdebug

Note – Problems seeing disks are addressed in Module 6, “Recovering


Disk, Disk Group, and Volume Failures.” DMP debugging utilities and
troubleshooting suggestions are discussed in Module 4, “Troubleshooting
Tools and Utilities.”

Sun Proprietary: Internal Use Only


3-22 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Operating DMP

Exercise: Operating DMP


In this exercise, you complete the following tasks:
● Configure DMP to exclude a disk device from multi-pathing
operations using the vxdiskadm utility.
● Suppress a path from a DMP-excluded disk using the vxdiskadm
utility.
● Unsuppress a path from a DMP-excluded disk using the vxdiskadm
utility.
● Configure DMP to include a disk device for DMP operations using
the vxdiskadm utility.
● View DMP status using the vxdmpadm utility.
● Locate files used to disable and enable DMP.

Preparation
To prepare for this exercise:
● The VxVM software 3.2 must be installed and operational.
● The boot disk must be encapsulated and mirrored.
● A second disk group must exist other than rootdg with a configured
volume that is formatted and mounted.
● Open a window, and execute an iostat while performing the lab
exercises to see the effect on pathing as paths are removed and
added. Use the following script to run the iostat command:
# while true
> do
> iostat -xcnm
> sleep 2
> done

Note – Re-size the window to fit the output of the iostat display.

● Generate looping I/Os to one of the non-root volumes created in the


exercises in Module 2, “Encapsulating Disks.”

Sun Proprietary: Internal Use Only


Managing Dynamic Multi-Pathing 3-23
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Operating DMP

Task 1 – Enabling and Disabling DMP Operations


Complete the following steps:
1. Use the vxdiskadm utility to disable multi-pathing to one VxVM
software disk on your lab teams system. Do not use the boot disk for
this task.
What vxdiskadm main menu option did you use?
_____________________________________________________________
How did you activate the change you made?
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________
2. Use the vxdiskadm utility to suppress the secondary paths to the
disk on which you just prevented DMP operations.
What vxdiskadm main menu option did you use?
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________
Did you have to reboot to activate this operation?
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________
3. View the DMP multi-pathing status to verify that you were
successful in the execution of the previous tasks.
What commands did you use?
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________
4. What two files were modified to enable the previous operation?
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________

Sun Proprietary: Internal Use Only


3-24 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Operating DMP

View the contents of these files to see the changes that were made.
_____________________________________________________________
5. Un-suppress and allow DMP operations on the disk excluded in the
first two tasks of this lab exercise.
What commands did you use, and in what order did you execute
them?
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________
View the contents of the two exclude files to see what changes were
made.
_____________________________________________________________

Task 2 – Administrating DMP


Complete the following steps:
1. Use the vxdmpadm utility to list all controllers seen by the Solaris OE.
What format of the vxdmpadm command did you use?
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________
2. Use the vxdmpadm utility to list all subpaths to a specific node. The
DMP node you select depends on the configuration of your system.
What format of the vxdmpadm command did you use?
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________
3. Disable a storage controller using the vxdmpadm utility.
What format of the vxdmpadm command did you use?
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________

Sun Proprietary: Internal Use Only


Managing Dynamic Multi-Pathing 3-25
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Operating DMP

4. Verify that you were successful.


_____________________________________________________________
What format of the vxdmpadm command did you use?
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________
5. Enable the storage controller you just disabled using the vxdmpadm
utility.
What format of the vxdmpadm command did you use?
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________
6. Verify that you were successful.
What format of the vxdmpadm command did you use?
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________

Task 3 – Using the /etc/vx/diag.d/vxdmping Script


Use the vxdmping script to view the serial number of a disk of your
choice.

What command syntax did you use?

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

What is the serial number of the disk you selected?

__________________________________________________________________

__________________________________________________________________

Sun Proprietary: Internal Use Only


3-26 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise Summary

Exercise Summary

Discussion – Take a few minutes to discuss what experiences, issues, or


discoveries you had during the lab exercise.
!
?
● Experiences
● Interpretations
● Conclusions
● Applications

Sun Proprietary: Internal Use Only


Managing Dynamic Multi-Pathing 3-27
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Operating DMP

Exercise: Operating DMP


Following are the solutions to this lab exercise.

Task 1 Solutions
Complete the following steps:
1. Use the vxdiskadm utility to disable multi-pathing to one VxVM
software disk on your lab teams system. Do not use the boot disk for
this task.
What vxdiskadm main menu option did you use?
_____________________________________________________________
How did you activate the change you made?

Reboot.
2. Use the vxdiskadm utility to suppress the secondary paths to the
disk on which you just prevented DMP operations.
What vxdiskadm main menu option did you use?

Use option 17, Prevent multi-pathing/Suppress devices from VxVM’s view.


Did you have to reboot to activate this operation?

No.
3. View the DMP multi-pathing status to verify that you were
successful in the execution of the previous tasks.
What commands did you use?

Use the following:


● Issue a vxdisk list command.
● Issue a vxdmpadm listexclude command.
4. What two files were modified to enable the previous operation?

The files /etc/vx/vxdmp.exclude and /etc/vx/vxvm.exclude.


View the contents of these files to see the changes that were made.

Sun Proprietary: Internal Use Only


3-28 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Operating DMP

5. Un-suppress and allow DMP operations on the disk excluded in the


first two tasks of this lab exercise.
What commands did you use and in what order did you execute
them?

Use vxdiskadm menu option 18 to unsuppress the secondary path.

Use vxdiskadm menu option 18 to include both disk paths in DMP operations.
View the contents of the two exclude files to see what changes were
made.

Task 2 Solutions
Complete the following steps:
1. Use the vxdmpadm utility to list all controllers seen by the Solaris OE.
What format of the vxdmpadm command did you use?

Use vxdmpadm listctlr all.


2. Use the vxdmpadm utility to list all subpaths to a specific node. The
DMP node you select depends on the configuration of your system.
What format of the vxdmpadm command did you use?

Use vxdmpadm getsubpaths dmpnodename=c#t#d#s2.


3. Disable a storage controller using the vxdmpadm utility.
What format of the vxdmpadm command did you use?

Use vxdmpadm disable ctlr=full_Solaris_OE_device_path-name.


4. Verify that you were successful.
_____________________________________________________________
What format of the vxdmpadm command did you use?

Use vxdmpadm listctlr all.


5. Enable the storage controller you just disabled using the vxdmpadm
utility.
What format of the vxdmpadm command did you use?

Use vxdmpadm enable ctlr=full_Solaris_OE_device_path-name.

Sun Proprietary: Internal Use Only


Managing Dynamic Multi-Pathing 3-29
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Operating DMP

6. Verify that you were successful.


What format of the vxdmpadm command did you use?

Use vxdmpadm listctlr all.

Task 3 Solutions
Use the vxdmping script to view the serial number of a disk of your
choice.

What command syntax did you use?

Use /etc/vx/diag.d/vxdmpinq /dev/rdsk/c#t#d#s2.

What is the serial number of the disk you selected?

Depends on disk selected.

Sun Proprietary: Internal Use Only


3-30 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Module 4

Troubleshooting Tools and Utilities

Objectives
Upon completion of this module, you should be able to:
● Describe the troubleshooting tools and utilities available for the
VxVM software
● Enable vxconfigd logging from the command line and the VxVM
software startup scripts
● Reference a VxVM failure to the error messages section of the VxVM
software troubleshooting manual
● Use the VxVM software debugging and information-gathering tools
and utilities

Sun Proprietary: Internal Use Only


4-1
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Relevance

Relevance

Discussion – The following questions are relevant to understanding how


to use the VxVM software tools and utilities to troubleshoot VxVM
! software failures:
?

● What tools and utilities are available to system administrators to use


in debugging VxVM software problems?
● How is vxvonfigd logging enabled?
● What system-level information gathering tools are available for
capturing VxVM software-specific information to use in debugging
problems?

Sun Proprietary: Internal Use Only


4-2 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Additional Resources

Additional Resources

Additional resources – The following references provide additional


information on the topics described in this module:
● Service Support (IT Infrastructure Library Series). Stationary Office
Books, February 2002, ISBN 0113300158.
● CCTA Staff. Service Delivery. Stationary Office Books, April 2001,
ISBN 0113300174.
● Ivor Macfarlane and Colin Rudd. IT Service Management Version 2.
United Kingdom: itSMF Ltd, March 2001, ISBN 0-9524706-1-6.
● VERITAS Volume Manager™ 3.2 Administrator’s Guide. Mountain
View, California: VERITAS Software Corporation, August 2001,
number 30-000392-011, TechPDF ID 240253.
● VERITAS Volume Manager™ 3.2 Installation Guide. Mountain View,
California: VERITAS Software Corporation, August 2001,
number 30-000395-011, TechPDF ID 240256.
● VERITAS Volume Manager™ 3.2 Troubleshooting Guide. Mountain
View, California: VERITAS Software Corporation, August 2001,
number 30-000394-011, TechPDF ID 240255.
● VERITAS Volume Manager™ Storage Administrator 3.2 Administrator’s
Guide. Mountain View, California: VERITAS Software Corporation,
July 2001, number 30-000393-011, TechPDF ID 240257.
● http://storage.east, http://storage.central, and
http://storage.west.

Sun Proprietary: Internal Use Only


Troubleshooting Tools and Utilities 4-3
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Logging Errors

Logging Errors
The VxVM software provides extensive logging capabilities that include
the following error logging mechanisms:
● vxconfigd logging
● /var/adm/messages file
● root mail

Using the /var/vxvm/vxconfigd.log File


The VxVM software provides extensive logging capabilities, including the
ability to redirect all the VxVM software-related console messages to a
VxVM software-specific log file and to syslog. This saves for post-crash
analysis any messages generated by the VxVM software prior to a system
crash.

All logging is managed by the vxconfigd daemon. The vxconfigd


logging reports errors detected by the vxconfigd daemon; these errors
are not limited to disk errors. Any error detected by the vxconfigd
daemon can be logged. Logging to the vxconfigd.log is disabled by
default.

There are nine levels of debug logging. Level 1 is minimal logging, and
level 9 logging involves writing all debug and error messages to the
selected log file. Logging is enabled by running the vxconfigd command
from the command line, or by modifying the
/etc/init.d/vxvm-sysboot script.

Caution – Enabling vxconfigd logging at the most verbose level, level 9,


produces excessive amounts of data and can negatively impact system
performance. Disable this type of logging until it is needed to debug a
VxVM software problem.

To view output of the log in real time, tail the /var/vxvm/configd.log


file. Use the following syntax:
# tail -f /var/vxvm/vxconfigd.log

Sun Proprietary: Internal Use Only


4-4 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Logging Errors

Manually Enabling and Disabling Logging

To enable logging from the command line, vxconfigd must first be


stopped. To stop the vxconfigd daemon, use the following command
syntax:
# pkill -9 vxconfigd; vxconfigd -x 9 -x log -x syslog > /dev/null 2>&1 &

This enables maximum vxconfigd logging while suppressing vxconfigd


log messages to the standard out and standard error outputs.

To enable logging selectively from the command line, do one of the


following:
● Enable maximum logging of error and debug messages to
/var/vxvm/vxconfigd.log. Type the following:
# vxconfigd -x 9 -x log
● Enable syslog logging of the VxVM software console messages. Run
the following command:
# vxconfigd -x syslog
This version of the vxconfigd command logs all Error, Fatal Error,
Warning and Notice messages to the syslog. Debug messages are
not logged.
● Enable maximum logging to the vxconfigd log and the syslog file.
Type the following:
# vxconfigd -x 9 -x log -x syslog

To disable vxconfigd logging after it is enabled, use the following syntax:


# pkill -9 vxconfigd; vxconfigd -m enable

Automatically Enabling Logging

To enable logging automatically during the boot process, edit the


/etc/init.d/vxvm-sysboot file and un-comment the lines that
correspond to the logging desired for a specific server or group of servers,
as shown in the following example:
# comment-out or uncomment any of the following lines to enable or
# disable the corresponding feature in vxconfigd.

#opts="$opts -x syslog" # use syslog for console messages


#opts="$opts -x log" # messages to /var/vxvm/vxconfigd.log
#opts="$opts -x logfile=/foo/bar" # specify an alternate log file
#opts="$opts -x timestamp" # timestamp console messages

Sun Proprietary: Internal Use Only


Troubleshooting Tools and Utilities 4-5
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Logging Errors

# to turn on debugging console output, uncomment the following line.


# The debug level can be set higher for more output. The highest debug
# level is 9.

#debug=1 # enable debugging console ouptut

To activate logging after the vxvm-sysboot file is modified, you must


reboot the system.

Example vxconfigd Logging Output

This example of the /var/vxvm/vxconfigd.log shows maximum


logging enabled and a fiber cable disconnected.
07/03 09:01:16: DEBUG: dmp_get_events: Start
07/03 09:01:17: DEBUG: dmp_get_events: Number of events = 2

07/03 09:01:17: DEBUG: dmp_get_events: Event type = 0x1


07/03 09:01:17: DEBUG: dmp_get_events: event type = DMP_PATH_DISABLE_EVENT
07/03 09:01:17: DEBUG: dmp_get_events: path 118/48, dmpnode 68/48
07/03 09:01:17: DEBUG: dmp_get_events: Event type = 0x1
07/03 09:01:17: DEBUG: dmp_get_events: event type = DMP_PATH_DISABLE_EVENT
07/03 09:01:17: DEBUG: dmp_get_events: path 118/32, dmpnode 68/0
07/03 09:01:17: DEBUG: dmp_get_events: Done
07/03 09:01:17: DEBUG: request_loop: No. of DMP events = 2

07/03 09:01:17: DEBUG: request_loop: notify_clients 0

07/03 09:01:17: DEBUG: request_loop: notify_clients 1

The vxconfigd logging Debug messages are not described in the


VERITAS Volume Manager™ 3.2 Troubleshooting Guide. To make these
messages useful, decode any device-related major and minor number
information included in the Debug message.

Sun Proprietary: Internal Use Only


4-6 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Logging Errors

In the previous example, the following error is displayed:


07/03 09:01:17: DEBUG: dmp_get_events: event type = DMP_PATH_DISABLE_EVENT
07/03 09:01:17: DEBUG: dmp_get_events: path 118/48, dmpnode 68/48

This message is decoded as follows:


1. Look at the event type:
event type = DMP_PATH_DISABLE_EVENT
This indicates a path error to one or more disk devices.
2. The part of the message following the event type defines the disk
device affected. The major and minor number of that device are
listed.
path 118/48
To find the cxtxdx number, do a long list on the /dev/dsk directory
and search for the major and minor number pair listed in the error
message. Use the following form of the ls command:
# ls -laRL /dev/dsk | grep “118, 48”
brw-r----- 1 root sys 118, 48 Jun 7 23:47 c2t1d0s0
3. Using this information, the experienced system administrator maps
this to a failing piece of hardware. From the following example and
the ls -l /dev/dsk/c2t1d0s0 command, the hardware address for
device c2t1d0s0s is:
/sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w210000203713f96d,0
This address can now be cross-references to a specific I/O bus, card
slot, port and cable. Information about bus mapping is available on
the SunSolve Online Web site.
4. The remaining part of the message following the event type points to
the DMP metanode. The major and minor number of that device are
listed:
dmpnode 68/48
This device can also be found by using the ls command to list the
/dev/vx/dmp directory:
# ls -laRL /dev/vx/dmp | grep “68, 48”
brw------- 1 root other 68, 48 Jun 8 09:41 c1t1d0s0
This clearly defines the DMP metanode device as c1t1d0s0.

Note – DMP devices, metanodes, and other information about DMP were
described in Module 3, “Managing Dynamic Multi-Pathing.”

Sun Proprietary: Internal Use Only


Troubleshooting Tools and Utilities 4-7
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Logging Errors

Interpreting /var/adm/messages File syslog


Messages
The VxVM software logging to the system syslog facility is enabled
automatically. The software writes errors detected by vxconfigd to the
/var/adm/messages file. Use the -x sys log option when starting
vxconfigd to write all console messages created by vxconfigd to the
syslog as well.

The following shows vxconfigd messages logged to the


/var/adm/messages file with a failed Fiber Channel cable to a disk array:
Jul 3 11:11:56 lowtide ssdrestart transport failed (fffffffe)
Jul 3 11:11:56 lowtide vxdmp: [ID 619769 kern.notice] NOTICE: vxdmp: Path failure on
118/0x48
Jul 3 11:11:56 lowtide vxdmp: [ID 997040 kern.notice] NOTICE: vxvm:vxdmp: disabled
path 118/0x48 belonging to the dmpnode 68/0x48

Cross-reference this message to the section in the VERITAS Volume


Manager™ 3.2 Troubleshooting Guide on error messages generated by the
VxVM software. Use the following procedure to troubleshoot this
problem:
1. Go to section 3 in the VERITAS Volume Manager™ 3.2 Troubleshooting
Guide.
2. Go to the section on vxdmp NOTICE messages.
3. Look up the NOTICE message for the failure listed in the messages
log. The error message description is as follows:
Description: A path under the control of the DMP driver failed. The
failed device major and minor numbers are supplied in the message.
Action: None

Note – The action recommendation is None because the error was not
caused by the VxVM software. This is a hardware problem, and there are
no VxVM software commands that can fix the error.

4. Find the major and minor numbers, and use the following version of
the ls command to find the cxtxdx number:
# ls -laRL /dev/dsk | grep “118, 48”
brw-r----- 1 root sys 118, 48 Jun 7 23:47 c2t1d0s0
The failed path is c2t1d0s0. Map this path to a specific piece of
hardware using step 3 of the procedure described on page 4-7.

Sun Proprietary: Internal Use Only


4-8 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Logging Errors

The root Mail


When the VxVM software detects volume errors, it sends mail to the root
account on the local system. This example shows mail sent by the VxVM
software after a relocation error.
From root Sat Jun 8 11:56:12 2002
Date: Sat, 8 Jun 2002 11:56:12 -0700 (PDT)
From: Super-User <root>
Message-Id: <200206081856.g58IuCs05735@lowtide.localdomain>
To: root
Subject: Attempting VxVM relocation on host lowtide
Content-Length: 420

Relocation was not successful for subdisks on disk test1dg05 in


volume test1_vol-L02 in disk group test1dg. No replacement was made and the
disk is still unusable.
The following volumes have storage on test1dg05:

test1_vol-L02

These volumes are still usable, but the the redundancy of


those volumes is reduced. Any RAID-5 volumes with storage on
the failed disk may become unusable in the face of further
failures.

Read mail messages using either the mail or mailx utility.

Sun Proprietary: Internal Use Only


Troubleshooting Tools and Utilities 4-9
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Recovering From Errors

Recovering From Errors


The VxVM software is fault resilient and capable of resolving most errors
without work by the system administrator. The vxconfigd daemon can
handle any error as long as it recognizes the error. If vxconfigd cannot
correct an error, then the system administrator must debug and resolve
the problem.

To debug errors, use the messages generated by the VxVM software. A list
of error messages is located in the VERITAS Volume Manager™ 3.2
Troubleshooting Guide in the “Error Messages” section for the version of the
VxVM software installed on the system.

Sun Proprietary: Internal Use Only


4-10 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Using the Debugging Tools and Utilities

Using the Debugging Tools and Utilities


To install many of the VxVM software debugging tools, add the VRTSspt
package. This package is located in the VxVM software support
directory:
bash-2.03# ls
copyright pkgs vr_notes.pdf vxvm_notes.ps
hwnotes.pdf scripts vr_notes.ps win32
hwnotes.ps support vxvm_notes.pdf

The debugging tools are in the /opt/VRTSspt directory. A list of the


contents of this directory is as follows:
bash-2.03# ls -las /opt/VRTSspt
total 14
2 dr-x------ 5 root other 512 Jul 2 08:19 .
2 drwxr-xr-x 12 root sys 512 Jul 2 08:19 ..
2 dr-xr-xr-x 4 root other 512 Jul 2 08:19 FS
4 -r--r--r-- 1 root other 1444 Aug 8 2001 README.VRTSspt
2 dr-xr-xr-x 4 root other 1024 Jul 2 08:19 VRTSexplorer
2 dr-xr-xr-x 3 root other 512 Jul 2 08:19 VVR
/opt/VRTSspt/FS:
total 8
2 dr-xr-xr-x 4 root other 512 Jul 2 08:19 .
2 dr-x------ 5 root other 512 Jul 2 08:19 ..
2 dr-x------ 2 root other 512 Jul 2 08:19 MetaSave
2 dr-x------ 2 root other 512 Jul 2 08:19 VxBench
/opt/VRTSspt/VRTSexplorer:
total 200
2 dr-xr-xr-x 4 root other 1024 Jul 2 08:19 .
2 dr-x------ 5 root other 512 Jul 2 08:19 ..
26 -r--r--r-- 1 root other 12645 Aug 21 2001 README
4 -r-xr--r-- 1 root other 1880 Aug 21 2001 VRTSexplorer
2 dr-xr-xr-x 4 root other 1024 Jul 2 08:19 bin.SunOS
8 -r-xr--r-- 1 root other 3833 Aug 21 2001 dbed
4 -r-xr--r-- 1 root other 1611 Aug 21 2001 dbed1
6 -r-xr--r-- 1 root other 2452 Aug 21 2001 edition.dro
6 -r-xr--r-- 1 root other 2701 Aug 21 2001 edition.sybed
2 -r-xr--r-- 1 root other 955 Aug 21 2001 fw
2 -r-xr--r-- 1 root other 343 Aug 21 2001 gcm
2 -r-xr--r-- 1 root other 764 Aug 21 2001 isis
2 dr-xr-xr-x 2 root other 512 Jul 2 08:19 lib
14 -r-xr--r-- 1 root other 6491 Aug 21 2001 main.SunOS
2 -r-xr--r-- 1 root other 809 Aug 21 2001 ndmp
6 -r-xr--r-- 1 root other 2658 Aug 21 2001 sal
4 -r-xr--r-- 1 root other 1692 Aug 21 2001 salr
8 -r-xr--r-- 1 root other 3192 Aug 21 2001 samba
8 -r-xr--r-- 1 root other 4022 Aug 21 2001 spc
16 -r-xr--r-- 1 root other 7349 Aug 21 2001 spcs
6 -r-xr--r-- 1 root other 2078 Aug 21 2001 spnas
6 -r-xr--r-- 1 root other 2960 Aug 21 2001 sybed1

Sun Proprietary: Internal Use Only


Troubleshooting Tools and Utilities 4-11
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Using the Debugging Tools and Utilities

6 -r-xr--r-- 1 root other 2834 Aug 21 2001 txpt


6 -r-xr--r-- 1 root other 2384 Aug 21 2001 vcs
2 -r-xr--r-- 1 root other 349 Aug 21 2001 vfr
4 -r-xr--r-- 1 root other 1538 Aug 21 2001 visnC
8 -r-xr--r-- 1 root other 3342 Aug 21 2001 visnS
2 -r-xr--r-- 1 root other 229 Aug 21 2001 vmsa
4 -r-xr--r-- 1 root other 1220 Aug 21 2001 vnas
2 -r-xr--r-- 1 root other 663 Aug 21 2001 vras
6 -r-xr--r-- 1 root other 2355 Aug 21 2001 vrtsisp
4 -r-xr--r-- 1 root other 1584 Aug 21 2001 vsap
4 -r-xr--r-- 1 root other 1434 Aug 21 2001 vvr
6 -r-xr--r-- 1 root other 2250 Aug 21 2001 vxfs
2 -r-xr--r-- 1 root other 486 Aug 21 2001 vxld
6 -r-xr--r-- 1 root other 2278 Aug 21 2001 vxvm
/opt/VRTSspt/VVR:
total 26
2 dr-xr-xr-x 3 root other 512 Jul 2 08:19 .
2 dr-x------ 5 root other 512 Jul 2 08:19 ..
4 -r-x------ 1 root other 1592 Jul 3 2001 Client.pl
2 -r-------- 1 root other 388 Jun 29 2001 README.Client_Server
4 -r-x------ 1 root other 1517 Jun 29 2001 Server.pl
2 dr-xr-xr-x 2 root other 512 Jul 2 08:19 VolSig
10 -r-x------ 1 root other 5081 May 21 2001 vvrmemstat

Note – The FS and VVR tools require additional licenses and are not part
of the basic VxVM software. They are not described in this guide.

The VxVM software debugging tools are listed in Table 4-1.

Table 4-1 The VRTSspt Debugging Tools

Tool Use Tool License

metasave Gathers metadata from a VxVM file FS


system
vxbench Generates various I/Os for FS
benchmarking file system
performance
VRTSexplorer* Gathers configuration information VxVM
about the system and any installed software
Sun StorEdge products
vvrmemstat Gathers Sun StorEdge Volume VVR
Replicator (VVR) memory usage
statistics
* Some of the information generated by VRTSexplorer is redundant with the
information generated by the Sun Explorer 3.5.0 data collector.

Sun Proprietary: Internal Use Only


4-12 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Using the Debugging Tools and Utilities

Table 4-1 The VRTSspt Debugging Tools (Continued)

Tool Use Tool License


volsig Determines the data consistency VVR
between the primary and secondary
volumes in a VVR configuration
Client.pl and Simulates traffic between the primary VVR
Server.pl and secondary nodes in a VVR
configuration
* Some of the information generated by VRTSexplorer is redundant with the
information generated by the Sun Explorer 3.5.0 data collector.

There are additional diagnostic and debugging tools installed as part of


the VRTSvxvm package. These tools and utilities are in the
/etc/vx/diag.d directory. A list of the VRTSvxvm tools and utilities
available for use by system administrators is as follows:
bash-2.03# ls /etc/vx/diag.d
config.d vxaslkey vxdevwalk vxdmpinq vxprivutil
macros.d vxautoconfig vxdmpdbprint vxdmptp
scripts vxconfigdump vxdmpdebug vxkprint

bash-2.03# ls -las /etc/vx/diag.d/*


544 -r-xr-xr-x 1 root sys 262172 Apr 4 19:36 /etc/vx/diag.d/vxaslkey
14 -r-xr-xr-x 1 root sys 6788 Aug 15 2001 /etc/vx/diag.d/vxautoconfig
464 -r-xr-xr-x 1 root sys 221192 Apr 4 19:37 /etc/vx/diag.d/vxconfigdump
14 -r-xr-xr-x 1 root sys 6808 Aug 15 2001 /etc/vx/diag.d/vxdevwalk
168 -r-xr-xr-x 1 root sys 85396 Apr 4 19:36 /etc/vx/diag.d/vxdmpdbprint
16 -r-xr-xr-x 1 root sys 7356 Apr 4 19:36 /etc/vx/diag.d/vxdmpdebug
12 -r-xr-xr-x 1 root sys 5932 Apr 4 19:36 /etc/vx/diag.d/vxdmpinq
124 -r-xr-xr-x 1 root sys 62472 Apr 4 19:36 /etc/vx/diag.d/vxdmptp
114 -r-xr-xr-x 1 root sys 58164 Apr 4 19:37 /etc/vx/diag.d/vxkprint
240 -r-xr-xr-x 1 root sys 114640 Aug 15 2001 /etc/vx/diag.d/vxprivutil
/etc/vx/diag.d/config.d:
total 8
2 drwxr-xr-x 4 root sys 512 Jun 8 09:19 .
2 drwxr-xr-x 5 root other 512 Jun 8 09:31 ..
2 drwxr-xr-x 2 root sys 512 Jun 8 09:19 sparcv7
2 drwxr-xr-x 2 root sys 512 Jun 8 09:19 sparcv9

/etc/vx/diag.d/config.d/sparcv7:
total 418
2 drwxr-xr-x 2 root sys 512 Jun 8 09:19 .
2 drwxr-xr-x 4 root sys 512 Jun 8 09:19 ..
114 -r-xr-xr-x 1 root sys 57440 Aug 15 2001 vxautoconfig.SunOS_5.6
114 -r-xr-xr-x 1 root sys 57708 Aug 15 2001 vxautoconfig.SunOS_5.7
114 -r-xr-xr-x 1 root sys 57824 Aug 15 2001 vxautoconfig.SunOS_5.8
24 -r-xr-xr-x 1 root sys 11960 Aug 15 2001 vxdevwalk.SunOS_5.6

Sun Proprietary: Internal Use Only


Troubleshooting Tools and Utilities 4-13
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Using the Debugging Tools and Utilities

24 -r-xr-xr-x 1 root sys 12032 Aug 15 2001 vxdevwalk.SunOS_5.7


24 -r-xr-xr-x 1 root sys 11968 Aug 15 2001 vxdevwalk.SunOS_5.8

/etc/vx/diag.d/config.d/sparcv9:
total 348
2 drwxr-xr-x 2 root sys 512 Jun 8 09:19 .
2 drwxr-xr-x 4 root sys 512 Jun 8 09:19 ..
0 -r-xr-xr-x 1 root sys 0 Aug 15 2001 vxautoconfig.SunOS_5.6
140 -r-xr-xr-x 1 root sys 71408 Aug 15 2001 vxautoconfig.SunOS_5.7
140 -r-xr-xr-x 1 root sys 71560 Aug 15 2001 vxautoconfig.SunOS_5.8
0 -r-xr-xr-x 1 root sys 0 Aug 15 2001 vxdevwalk.SunOS_5.6
32 -r-xr-xr-x 1 root sys 15416 Aug 15 2001 vxdevwalk.SunOS_5.7
32 -r-xr-xr-x 1 root sys 15368 Aug 15 2001 vxdevwalk.SunOS_5.8

/etc/vx/diag.d/macros.d:
total 58
2 drwxr-xr-x 3 root other 1024 Jun 8 09:31 .
2 drwxr-xr-x 5 root other 512 Jun 8 09:31 ..
4 -rwxrwxr-x 1 root sys 1975 Aug 15 2001 dmp
2 -rwxrwxr-x 1 root sys 94 Apr 4 19:36 dmp_cpuiocount
2 -rwxrwxr-x 1 root sys 99 Apr 4 19:36 dmp_cpuiocount_next
0 -rwxrwxr-x 1 root sys 0 Apr 4 19:36 dmp_cpuiocount_zero
2 -rwxrwxr-x 1 root sys 435 Apr 4 19:36 dmp_ctlr
2 -rwxrwxr-x 1 root sys 69 Apr 4 19:36 dmp_ctlr_list_next
2 -rwxrwxr-x 1 root sys 72 Apr 4 19:36 dmp_ctlr_path_next
2 -rwxrwxr-x 1 root sys 285 Apr 4 19:36 dmp_dev_list
2 -rwxrwxr-x 1 root sys 82 Apr 4 19:36 dmp_dev_list_next_dmpnode
2 -rwxrwxr-x 1 root sys 364 Apr 4 19:36 dmp_dmpnode
2 -rwxrwxr-x 1 root sys 448 Apr 4 19:36 dmp_dmpnode_next
2 -rwxrwxr-x 1 root sys 78 Apr 4 19:36 dmp_dmpnode_next_ptr
2 -rwxrwxr-x 1 root sys 104 Apr 4 19:36 dmp_dmpnode_path_next
2 -rwxrwxr-x 1 root sys 52 Aug 15 2001 dmp_dmpopencount
2 -rwxrwxr-x 1 root sys 63 Aug 15 2001 dmp_end_dev_list_ctlrs
2 -rwxrwxr-x 1 root sys 66 Aug 15 2001 dmp_end_dev_list_dmpnodes
2 -rwxrwxr-x 1 root sys 50 Aug 15 2001 dmp_end_dmp_nodes
2 -rwxrwxr-x 1 root sys 150 Aug 15 2001 dmp_errq_buf
2 -rwxrwxr-x 1 root sys 40 Aug 15 2001 dmp_opath_next
2 -rwxrwxr-x 1 root sys 117 Apr 4 19:36 dmp_opaths
2 -rwxrwxr-x 1 root sys 582 Apr 4 19:36 dmp_path
2 -rwxrwxr-x 1 root sys 289 Apr 4 19:36 dmp_print_dev_list_ctlrs
2 -rwxrwxr-x 1 root sys 298 Aug 15 2001 dmp_print_dev_list_dmpnodes
2 -rwxrwxr-x 1 root sys 193 Apr 4 19:36 dmp_print_errq
2 -rwxrwxr-x 1 root sys 121 Apr 4 19:36 dmp_print_errq_next
2 -rwxrwxr-x 1 root sys 25 Apr 4 19:36 dmp_print_errq_null
2 drwxr-xr-x 2 root other 1024 Jun 8 09:31 sparcv9

/etc/vx/diag.d/macros.d/sparcv9:
total 56
2 drwxr-xr-x 2 root other 1024 Jun 8 09:31 .
2 drwxr-xr-x 3 root other 1024 Jun 8 09:31 ..
4 -rwxrwxr-x 1 root sys 1964 Aug 15 2001 dmp
2 -rwxrwxr-x 1 root sys 97 Apr 4 19:36 dmp_cpuiocount
2 -rwxrwxr-x 1 root sys 99 Apr 4 19:36 dmp_cpuiocount_next

Sun Proprietary: Internal Use Only


4-14 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Using the Debugging Tools and Utilities

0 -rwxrwxr-x 1 root sys 0 Apr 4 19:36 dmp_cpuiocount_zero


2 -rwxrwxr-x 1 root sys 436 Apr 4 19:36 dmp_ctlr
2 -rwxrwxr-x 1 root sys 69 Apr 4 19:36 dmp_ctlr_list_next
2 -rwxrwxr-x 1 root sys 72 Apr 4 19:36 dmp_ctlr_path_next
2 -rwxrwxr-x 1 root sys 282 Apr 4 19:36 dmp_dev_list
2 -rwxrwxr-x 1 root sys 84 Apr 4 19:36 dmp_dev_list_next_dmpnode
2 -rwxrwxr-x 1 root sys 388 Apr 4 19:36 dmp_dmpnode
2 -rwxrwxr-x 1 root sys 428 Apr 4 19:36 dmp_dmpnode_next
2 -rwxrwxr-x 1 root sys 81 Apr 4 19:36 dmp_dmpnode_next_ptr
2 -rwxrwxr-x 1 root sys 104 Apr 4 19:36 dmp_dmpnode_path_next
2 -rwxrwxr-x 1 root sys 52 Aug 15 2001 dmp_dmpopencount
2 -rwxrwxr-x 1 root sys 63 Aug 15 2001 dmp_end_dev_list_ctlrs
2 -rwxrwxr-x 1 root sys 66 Aug 15 2001 dmp_end_dev_list_dmpnodes
2 -rwxrwxr-x 1 root sys 50 Aug 15 2001 dmp_end_dmp_nodes
2 -rwxrwxr-x 1 root sys 150 Aug 15 2001 dmp_errq_buf
2 -rwxrwxr-x 1 root sys 40 Aug 15 2001 dmp_opath_next
2 -rwxrwxr-x 1 root sys 117 Apr 4 19:36 dmp_opaths
2 -rwxrwxr-x 1 root sys 590 Apr 4 19:36 dmp_path
2 -rwxrwxr-x 1 root sys 289 Apr 4 19:36 dmp_print_dev_list_ctlrs
2 -rwxrwxr-x 1 root sys 298 Aug 15 2001 dmp_print_dev_list_dmpnodes
2 -rwxrwxr-x 1 root sys 195 Apr 4 19:36 dmp_print_errq
2 -rwxrwxr-x 1 root sys 121 Apr 4 19:36 dmp_print_errq_next
2 -rwxrwxr-x 1 root sys 25 Apr 4 19:36 dmp_print_errq_null

/etc/vx/diag.d/scripts:
total 42
2 drwxr-xr-x 3 root sys 512 Jun 8 09:20 .
2 drwxr-xr-x 5 root other 512 Jun 8 09:31 ..
2 drwxr-xr-x 2 root sys 512 Jun 8 09:20 fix_lib
8 -r-xr-xr-x 1 root sys 3541 Aug 15 2001 fixmountroot
6 -r-xr-xr-x 1 root sys 2382 Aug 15 2001 fixsetup
10 -r-xr-xr-x 1 root sys 4897 Aug 15 2001 fixstartup
12 -r-xr-xr-x 1 root sys 5409 Aug 15 2001 fixunroot

/etc/vx/diag.d/scripts/fix_lib:
total 16
2 drwxr-xr-x 2 root sys 512 Jun 8 09:20 .
2 drwxr-xr-x 3 root sys 512 Jun 8 09:20 ..
8 -r-xr-xr-x 1 root sys 3437 Aug 15 2001 fixdevsetup
4 -r-xr-xr-x 1 root sys 1409 Aug 15 2001 fixgetmajor

Sun Proprietary: Internal Use Only


Troubleshooting Tools and Utilities 4-15
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Using the Debugging Tools and Utilities

Table 4-2 contains a partial list of the VRTSvxvm tools.

Note – For information on how to use utilities in the /etc/vx/diag.d


directory that is not available in this guide or the man pages, on SunSolve,
from the command line prompt or from the VERITAS Software
Corporation support site, call support at Sun Microsystems.

Table 4-2 The VRTSvxvm Debugging Tools

Tool Use

vxprivutil Allows system administrators to list and modify the


private region of disks.
vxdevwalk Traverses the dev_info tree in kernel space. It can be
used to correlate user space entries with those in the
dev_info tree.
vxkprint Prints kernel space information about VxVM objects
such as disk groups, disks, and volumes.
vxdmpdebug Creates a DMP configuration dump suitable for
sending to Sun second-level support. Used in
debugging DMP problems.
vxdmpinq Provides output that displays a disk’s unique serial
number and vendor information. Used in debugging
DMP problems.
vxautoconfig Prints device, driver, and controller information about
a server’s attached storage. This is very low-level
information and might not be useful to system
administrators.
vxdmpdbprint Prints contents of the DMP database. Useful for
troubleshooting DMP path and device errors.

Sun Proprietary: Internal Use Only


4-16 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Using the Debugging Tools and Utilities

Table 4-3 lists scripts found in the /etc/vx/diag.d/scripts directory.


Use these scripts to recover a VxVM software configuration that is
corrupted and prevents a system from booting. These scripts help
reconfigure the VxVM software by defining alternative locations for
binaries which can be located on a CD-ROM, and mounting boot disks
that are VxVM software volumes.

Caution – The /etc/vx/diag.d/scripts scripts are designed to help


recover a corrupted VxVM software configuration that prevents a system
from booting or mounting the VxVM software devices. Module 2,
“Encapsulating Disks,” describes how to recover problems with booting
from encapsulated disks if the VxVM software does not render a system
un-bootable. Use the /etc/vx/diag.d/scripts scripts with caution or
under the direction of Sun support.

Table 4-3 The /etc/vx/diag.d/scripts Script Utilities

Script Use

fixmountroot Mounts a root file system that is encapsulated as a


volume. The file system is mounted as read-only,
using a file system slice that underlies the root file
system volume.
fixsetup Configures the VxVM software to run from an
alternate directory. This allows the system
administrator to make changes to a VxVM software
configuration when the VxVM software binaries are
not accessible.
Run this script prior to running the fixmountroot
script.
fixstartup Starts the VxVM software running from the VxVM
CD-ROM. The script requires some information that
the VxVM software normally gets from the root file
system. Run the fixstartup command if the device
containing the root file system or one of the mirrors of
the root file system volume is known.
The fixstartup command asks for a disk containing
a copy of the root file system, and uses the file system
on that disk to get the necessary startup files.

Sun Proprietary: Internal Use Only


Troubleshooting Tools and Utilities 4-17
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Using the Debugging Tools and Utilities

Table 4-3 The /etc/vx/diag.d/scripts Script Utilities (Continued)

Script Use
fixunroot Converts system files so that the files no longer
require the VxVM software to boot the root file
system. Also disables startup of the VxVM software,
so that future recovery of a mirrored root volume does
not cause corruption.
Caution – After running this script, use caution when
bringing up the VxVM software again. If the VxVM
software configuration retains a mirrored root
volume, starting the VxVM software can cause severe
corruption to the root file system.

The vxexplorer Utility


The vxexplorer utility is an information-gathering tool that provides
detailed information about key system and VxVM software files and
objects. It is one of the most common VxVM software debugging tools
used. This section describes the information collected by vxexplorer.

Note – Not all files and commands described in this section are relevant to
the Solaris OE. Some of the information collected by the vxexplorer
utility reflects the UNIX cross-platform capabilities of the VxVM software.
Some files and command output reference versions of UNIX other than
the Solaris OE.

General System Information

The vxexplorer utility general system information gathering options


include:
● uname -a – Displays host and OS revision information.
● ioscan -f – Scans and lists system I/O hardware.
● model – Prints system model (on systems that do not use SPARC®
technology).
● isainfo -v – Displays supported instruction sets.
● prtdiag – Displays system configuration and diagnostic data
generated by power-on self-test (POST).

Sun Proprietary: Internal Use Only


4-18 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Using the Debugging Tools and Utilities

● prtconf -v – Displays general system-hardware configuration


information.
● showrev -p – Displays system patch list.
● ps -elf – Shows running processes, including the command line
options used to start them.
● pkginfo -l – Displays installed packages.
● modinfo – Displays loadable modules that are active on the system.
● /etc/system, fstab, bootconf vfstab, and
/etc/name_to_major – Captures key system information files from
/etc. (The fstab and bootconf files are not Solaris OE files.)
● /etc/services – Lists Internet services ports and protocols.
● /var/adm/messages – Contains messages showing the VxVM
software and Solaris OE error messages.
● ls -l /kernel/drv/vx*, and /kernel/drv/sparcv9/vx* –
Collects a list of installed VxVM software drivers.
● /kernel/drv/*.conf – Collects a complete list of driver
configuration files.
● vxlicense -p, vxfsserial -p, and vxliccheck -vp – Displays
VxVM software licensing data.
● la -laR /dev/dsk, ls -laR /dev/rdsk, ls -laRL /dev/dsk,
and ls -laRL /dev/rdsk – Displays physical device information,
including major and minor numbers.
● ls -laR /dev/es, ls -laRL /dev/es, luxadm (for each es device),
and luxadm fcal_s_download – Displays information about
installed Sun Enterprise Network Array™ arrays.
● prtvtoc information – Displays the vtoc for each disk.
● diskinfo – A command (not Solaris OE) to show a disk’s
characteristics.
● df -k and mount -v – Lists basic file system information.
● ifconfig -a, netstat -in, netstat -rn, arp -a, and lanscan –
Lists basic network information.
● vgdisplay -v – A command (not Solaris OE) to display information
about logical volume manager (LVM) volume groups.
● ls -l /etc/rc* and cp -pR /etc/rc* – Lists and copies boot and
startup scripts.

Sun Proprietary: Internal Use Only


Troubleshooting Tools and Utilities 4-19
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Using the Debugging Tools and Utilities

● cp -pR /etc/inet, cp -pR /etc/*router, cp -pR /etc/host*,


cp -pR /etc/*nodename*, and cp -pR /etc/nsswitch.conf –
Obtains copies of network configuration files.
● cp -pR /var/spool/cron – Obtains information on cron jobs.
● eeprom command output – Displays OpenBoot PROM information.
● cp -pR /etc/dfs – Obtains information on exported NFS files
systems.

VxVM Software Information

The vxexplorer utility options for gathering VxVM software-specific


information include:
● vxdctl mode – Displays the current operating mode of vxconfigd.
Effectively displays the state of the VxVM software.
● vxprint outputs – Displays the configuration of all VxVM software
objects.
● vxkprint output – Displays the VxVM software’s configuration as
seen by the kernel. This output can occasionally show useful
information.
● vxdisk list output – Shows state and configuration information
for VxVM software disks.
● vxdg list output – Displays disk group ownership and state
information. Shows which disk groups are imported.
● /var/vxvm contents – Lists and displays the contents of the temp_db
files. These files can be useful in troubleshooting odd disk group
problems.
● ls -laR /dev/vx/* – Displays the VxVM software device tree.
● /etc/vx contents – Contains various VxVM software configuration
and library files.
● /VXVM*UPGRADE* contents – Contains information pertaining to the
VxVM software configuration before upgrade. This file is created by
the upgrade_start script.

DMP Information

The vxexplorer utility options for gathering DMP-specific information


are:

Sun Proprietary: Internal Use Only


4-20 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Using the Debugging Tools and Utilities

● vxautoconfig – Captures the disks on a system that are visible to


the VxVM software without DMP. Mimics the VxVM software device
discovery routines.
● vxdevwalk – Captures disk device information by walking the kernel
device tree.
● SCSI inquiry output – DMP requires a unique serial number for each
disk on the system. It uses SCSI inquiries to read the serial number.
The vxexplorer utility uses the vxdmpinq command to capture this
information.
● vxconfigd -x 9 output – Starting vxconfigd with the -x 9 option
enables vxconfigd logging and displays all the steps used to start
vxconfigd. This includes DMP initialization and the display of all
the disks that are or are not visible to DMP. If the Sun StorEdge
Cluster Server (VCS) or EMC PowerPath software is running, a
storage or node failover might occur.
● DMP kernel table variables – Various kernel table information is
captured using the adb utility.

Volume Manager Storage Administrator (VMSA) Information

The contents of /var/opt/vmsa/logs captures system access, server, and


command information.

Other Non-VxVM Software Information

The vxexplorer utility collects output for other non-VxVM software


products, including the following. For more information about these non-
VxVM products, see the documentation for the product:
● SRVM
● VxFS
● VxLD (NFS Accellerator)
● First Watch
● VCS
● Sybase Edition
● Oracle Edition
● Sanbox
● Global Cluster Manager
● VFR

Sun Proprietary: Internal Use Only


Troubleshooting Tools and Utilities 4-21
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Using the Debugging Tools and Utilities

● VRAS
● SANPoint Control
● Information left by VERITAS Software Corporation support

Running VRTSexplorer Utility

To run VRTSexplorer, perform the following:


1. Change to the /opt/VRTSspt/VRTSexplorer directory. Type the
following:
cd /opt/VRTSspt/VRTSexplorer
2. Start VRTSexplorer. Type the following:
./VRTSexplorer
VRTSexplorer: Initializing.
VRTSexplorer: Please enter case number, or just hit enter: 12345 <---- note
VRTSexplorer: Please select a destination directory (default: /tmp):
VRTSexplorer: Using /tmp as destination directory.
VRTSexplorer: Collecting system configuration information for SunOS system.
VRTSexplorer: Collecting VERITAS package version information.
VRTSexplorer: Collecting loadable module information.
VRTSexplorer: Collecting VMSA configuration information.
VRTSexplorer: Determining current VxVM operating mode.
VRTSexplorer: Collecting VxVM configuration information.
VRTSexplorer: Collecting DMP configuration information.

NOTICE: This section will stop and restart the VxVM Configuration Daemon,
vxconfigd. This may cause your VxVA and/or VMSA session to exit.
This may also cause a momentary stoppage of any VxVM configuration
actions. This should not harm any data; however, it may cause some
configuration operations (e.g. moving subdisks, plex
resynchronization) to abort unexpectedly. Any VxVM configuration
changes should be completed before running this section.

If you are using EMC PowerPath devices with VERITAS Volume Manager,
you must run the EMC command(s) ’powervxvm setup’ (or ’safevxvm
setup’) and/or ’powervxvm online’ (or ’safevxvm online’) if this
script terminates abnormally.

Restart VxVM Configuration Daemon? [y,n] (default: n) y <---- note

VRTSexplorer: Script finished.


VRTSexplorer: Please ftp /tmp/VRTSexplorer_12345_lowtide.tar.Z to
ftp.veritas.com:/incoming

When VRTSexplorer executes, the program prompts for an optional case


number and whether to stop and restart vxconfigd to enable vxconfigd
logging. Answer the prompts as appropriate for each system.

Sun Proprietary: Internal Use Only


4-22 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Using the Debugging Tools and Utilities

The VRTSexplorer output file is by default stored in /tmp using the


following file name format:
VRTSexplorer_<case_number>_<VxVM_hostid>.tar.Z

Open this file and use the contents to aid in troubleshooting VxVM
software problems.

Interpreting VRTSexplorer Output

Use the following procedure to open and interpret a vxexplorer output


file:
1. Locate the VRTSexplorer_<case_number>_<VxVM_hostid>.tar.Z
file. The default location is /tmp.
All files in /tmp are removed when the system reboots. Either move
this file to /var/tmp, or instruct VRTSexplorer to store the file in
/var/tmp when you are asked for an output location during the
program execution.
2. Uncompress and untar the file. Type the following:
# uncompress VRTSexplorer _<case_number>_<VxVM_hostid>.tar.Z
# tar xf VRTSexplorer _<case_number>_<VxVM_hostid>.tar
This creates the following directory:
./VRTSexplorer_<case_number>_<VxVM_hostid>
3. Change directories to the new directory:
./VRTSexplorer_<case_number>_<VxVM_hostid>
The following example shows the contents of this directory:
bash-2.03# ls -las
total 1360
16 drwxr-xr-x 9 root other 2294 Jul 3 06:27 .
16 drwxrwxrwt 7 root sys 676 Jul 3 06:27 ..
16 -rw-r--r-- 1 root other 359 Jul 3 06:00 arp_a
16 drwxr-xr-x 4 root sys 309 Jun 7 23:11 cron
16 drwxr-xr-x 3 root other 3308 Jul 3 06:01 dev
16 -rw-r--r-- 1 root other 529 Jul 3 06:00 df_klaV
16 -rw-r--r-- 1 root other 4380 Jul 3 06:00 df_klag
16 -rw-r--r-- 1 root other 1152 Jul 3 06:00 df_klat
16 -rw-r--r-- 1 root other 1503 Jul 3 06:00 eeprom
16 drwxr-xr-x 12 root other 2426 Jul 3 06:01 etc
16 -rw-r--r-- 1 root other 9 Jul 3 05:59 hostid
16 -rw-r--r-- 1 root other 477 Jul 3 06:00 ifconfig_a
16 -rw-r--r-- 1 root other 54 Jul 3 05:59 isainfo_v
16 drwxr-xr-x 3 root other 177 Jul 3 06:00 kernel
16 -rw-r--r-- 1 root other 7132 Jul 3 06:00 modinfo
16 -rw-r--r-- 1 root other 1759 Jul 3 06:00 mount_v
16 -rw-r--r-- 1 root other 542 Jul 3 06:00 netstat_i

Sun Proprietary: Internal Use Only


Troubleshooting Tools and Utilities 4-23
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Using the Debugging Tools and Utilities

16 -rw-r--r-- 1 root other 921 Jul 3 06:00 netstat_r


16 -rw-r--r-- 1 root other 1322 Jul 3 06:00 pkgchk
64 -rw-r--r-- 1 root other 29532 Jul 3 05:59 pkginfo
576 -rw-r--r-- 1 root other 292733 Jul 3 06:00 pkginfo_l
112 -rw-r--r-- 1 root other 51497 Jul 3 05:59 prtconf
16 -rw-r--r-- 1 root other 1708 Jul 3 05:59 prtdiag
16 -rw-r--r-- 1 root other 4832 Jul 3 05:59 ps_elf
16 -rw-r--r-- 1 root other 580 Jul 3 05:59 reboot
96 -rw-r--r-- 1 root other 43673 Jul 3 06:00 showrev
16 -rw-r--r-- 1 root other 70 Jul 3 05:59 uname_a
16 -rw-r--r-- 1 root other 66 Jul 3 05:59 uptime
16 drwxr-xr-x 4 root other 302 Jul 3 06:01 var
16 drwxr-xr-x 3 root other 242 Jul 3 06:00 vmsa
16 -rw-r--r-- 1 root other 233 Jul 3 05:59 vmstat
16 -rw-r--r-- 1 root other 834 Jul 3 05:59 vmstat_s
16 -rw-r--r-- 1 root other 69 Jul 3 05:59 vxexplore_version
16 -rw-r--r-- 1 root other 41 Jul 3 06:00 vxliccheck_vp
16 -rw-r--r-- 1 root other 240 Jul 3 06:00 vxlicense_p
16 drwxr-xr-x 3 root other 1953 Jul 3 06:01 vxvm
4. From information stored in the preceding files, determine the
system’s configuration as follows:
a. Verify the hostid.
# more ./hostid
807d5d60
b. Verify the Solaris OE revision and hardware architecture.
# more uname_a
SunOS lowtide 5.8 Generic_108528-14 sun4u sparc
SUNW,Ultra-Enterprise
c. Determine the version of the VxVM software installed by
viewing the contents of the pkginfo file. If the VxVM software
was upgraded on the system, there may be more than one
pkginfo file. To determine which is the most current listing,
look at the PKGINST and INSTDATE fields, as shown in the
following example.
# more ./pkginfo_l
... skipping
PKGINST: VRTSvxvm
NAME: VERITAS Volume Manager, Binaries
CATEGORY: system
ARCH: sparc
VERSION: 3.2,REV=08.15.2001.23.27
BASEDIR: /
VENDOR: VERITAS Software
DESC: Virtual Disk Subsystem
PSTAMP: VERITAS-3.2t_p2.5:23-May-2002
INSTDATE: Jun 08 2002 09:30
HOTLINE: 800-342-0652
EMAIL: support@veritas.com

Sun Proprietary: Internal Use Only


4-24 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Using the Debugging Tools and Utilities

STATUS: completely installed


FILES: 479 installed pathnames
16 shared pathnames
8 linked files
59 directories
306 executables
1 setuid/setgid executables
116150 blocks used (approx)

Note – The ./pkginfo_l file is a long list of all packages installed on the
system. Search through the file to find the VRTS packages.

d. Verify that all correct modules are loaded for the VRTS
packages installed. If the modules are not loaded, there may
have been problems upgrading the VxVM software, or the
system was not rebooted after the modules were installed. The
following example shows a list of installed modules.
bash-2.03# grep vx ./modinfo
19 101e9005 ffa88 74 1 vxio (VxVM 3.2t_p2.5 I/O driver)
21 102d5440 17428 68 1 vxdmp (VxVM 3.2t_p2.5: DMP Driver)
22 102ea888 83f 75 1 vxspec (VxVM 3.2t_p2.5 control/status
e. Verify the correct licenses are installed by viewing the
vxlicense_p file.
bash-2.03# more ./vxlicense_p

vrts:vxlicense: INFO: Feature name: PHOTON [98]


vrts:vxlicense: INFO: Number of licenses: 99
vrts:vxlicense: INFO: Expiration date: No expiration date
vrts:vxlicense: INFO: Release Level: 20
vrts:vxlicense: INFO: Machine Class: 334147806
f. Check the OpenBoot PROM information by opening the eeprom
file. Some OpenBoot PROM information is shown in the
following example.
# more eeprom

.
.
.

use-nvramrc?=true
nvramrc=devalias test2 /sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203713df01,0:a
devalias test1 /sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w210000203713fc9f,0:a

devalias vx-rootdisk /sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203713fc9f,0:a


devalias vx-rootmirror /sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203713f96d,0:a

Sun Proprietary: Internal Use Only


Troubleshooting Tools and Utilities 4-25
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Using the Debugging Tools and Utilities

g. Viewing the prtdiag displays system POST diagnostic


information.
bash-2.03# more prtdiag
System Configuration: Sun Microsystems sun4u 8-slot Sun Enterprise 4000/5000
System clock frequency: 84 MHz
Memory size: 256Mb

========================= CPUs =========================

Run Ecache CPU CPU


Brd CPU Module MHz MB Impl. Mask
--- --- ------- ----- ------ ------ ----
0 0 0 168 1.0 US-I 4.0
0 1 1 168 1.0 US-I 4.0

========================= Memory =========================

Intrlv. Intrlv.
Brd Bank MB Status Condition Speed Factor With
--- ----- ---- ------- ---------- ----- ------- -------
0 0 256 Active OK 60ns 1-way

========================= IO Cards =========================

Bus Freq
Brd Type MHz Slot Name Model
--- ---- ---- ---------- -------------------------------- -----------------

1 SBus 25 0 SUNW,socal/sf (scsi-3) 501-5266

1 SBus 25 1 SUNW,hme SUNW,501-2919

1 SBus 25 2 DOLPHIN,sci

1 SBus 25 3 SUNW,hme

1 SBus 25 3 SUNW,fas/sd (block)

1 SBus 25 13 SUNW,soc 501-2069

No failures found in System


===========================
h. The ./dev directory contains a copy of the contents of the /dev
directory of the live system. Use this information to verify
which devices that Solaris OE sees, including major and minor
numbers, prtvtoc output and more. An example follows:
bash-2.03# ls -las ./dev
total 880
16 drwxr-xr-x 3 root other 3308 Jul 3 06:01 .
16 drwxr-xr-x 9 root other 2294 Jul 3 06:27 ..
64 -rw-r--r-- 1 root other 31869 Jul 3 06:00 dsk

Sun Proprietary: Internal Use Only


4-26 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Using the Debugging Tools and Utilities

32 -rw-r--r-- 1 root other 14860 Jul 3 06:00 dsk_device


16 -rw-r--r-- 1 root other 663 Jul 3 06:00 es
16 -rw-r--r-- 1 root other 367 Jul 3 06:00 es_device
16 -rw-r--r-- 1 root other 1779 Jul 3 06:00 es_luxadm_display_ses0
16 -rw-r--r-- 1 root other 1779 Jul 3 06:00 es_luxadm_display_ses1
16 -rw-r--r-- 1 root other 1779 Jul 3 06:00 es_luxadm_display_ses2
16 -rw-r--r-- 1 root other 1779 Jul 3 06:00 es_luxadm_display_ses3
16 -rw-r--r-- 1 root other 143 Jul 3 06:00 es_luxadm_fcal_s_download
16 -rw-r--r-- 1 root other 41 Jul 3 06:00 prtvtoc_c0t6d0s2
16 -rw-r--r-- 1 root other 930 Jul 3 06:00 prtvtoc_c1t0d0s2
16 -rw-r--r-- 1 root other 648 Jul 3 06:00 prtvtoc_c1t16d0s2
16 -rw-r--r-- 1 root other 648 Jul 3 06:00 prtvtoc_c1t17d0s2
16 -rw-r--r-- 1 root other 648 Jul 3 06:00 prtvtoc_c1t18d0s2
16 -rw-r--r-- 1 root other 648 Jul 3 06:00 prtvtoc_c1t19d0s2
16 -rw-r--r-- 1 root other 929 Jul 3 06:00 prtvtoc_c1t1d0s2
16 -rw-r--r-- 1 root other 481 Jul 3 06:00 prtvtoc_c1t20d0s2
16 -rw-r--r-- 1 root other 481 Jul 3 06:00 prtvtoc_c1t21d0s2

bash-2.03# more ./dev/dsk


/dev/dsk:
total 482
drwxr-xr-x 2 root sys 5120 Jun 7 23:47 .
drwxr-xr-x 14 root sys 3584 Jul 3 05:29 ..
lrwxrwxrwx 1 root root 50 Jun 7 23:47 c0t6d0s0 ->
../../devices/sbus@3,0/SUNW,fas@3,8800000/sd@6,0:a
lrwxrwxrwx 1 root root 50 Jun 7 23:47 c0t6d0s1 ->
../../devices/sbus@3,0/SUNW,fas@3,8800000/sd@6,0:b
lrwxrwxrwx 1 root root 50 Jun 7 23:47 c0t6d0s2 ->
../../devices/sbus@3,0/SUNW,fas@3,8800000/sd@6,0:c
lrwxrwxrwx 1 root root 50 Jun 7 23:47 c0t6d0s3 ->
../../devices/sbus@3,0/SUNW,fas@3,8800000/sd@6,0:d
lrwxrwxrwx 1 root root 50 Jun 7 23:47 c0t6d0s4 ->
../../devices/sbus@3,0/SUNW,fas@3,8800000/sd@6,0:e
lrwxrwxrwx 1 root root 50 Jun 7 23:47 c0t6d0s5 ->
../../devices/sbus@3,0/SUNW,fas@3,8800000/sd@6,0:f

bash-2.03# more prtvtoc_c1t16d0s2


* /dev/rdsk/c1t16d0s2 partition map
*
* Dimensions:
* 512 bytes/sector
* 133 sectors/track
* 27 tracks/cylinder
* 3591 sectors/cylinder
* 4926 cylinders
* 4924 accessible cylinders
*
* Flags:
* 1: unmountable
* 10: read-only
*
* Unallocated space:
* First Sector Last
* Sector Count Sector
* 3591 3591 7181

Sun Proprietary: Internal Use Only


Troubleshooting Tools and Utilities 4-27
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Using the Debugging Tools and Utilities

*
* First Sector Last
* Partition Tag Flags Sector Count Sector Mount Directory
2 5 00 0 17682084 17682083
3 15 01 0 3591 3590
4 14 01 7182 17674902 17682083
i. Check the ./etc directory, which contains a copy of the
contents of the /etc directory of the live system. An example is
as follows:
bash-2.03# ls -las ./etc
total 640
16 drwxr-xr-x 12 root other 2426 Jul 3 06:01 .
16 drwxr-xr-x 9 root other 2294 Jul 3 06:27 ..
16 -rw-r--r-- 1 root other 12 Jun 8 05:43 defaultrouter
16 drwxr-xr-x 2 root sys 308 Jun 7 23:12 dfs
16 -rw-r--r-- 1 root other 12 Jul 3 06:01 err.out
16 -rw-r--r-- 1 root root 8 Jun 7 23:47 hostname.hme0
16 -rw-r--r-- 1 root root 1 Jun 7 23:47 hostname6.hme0
16 -r--r--r-- 1 root sys 97 Jun 8 05:43 hosts
16 drwxr-xr-x 2 root sys 1223 Jun 8 04:52 inet
32 -rw-r--r-- 1 root other 12200 Jul 3 06:00 ls_l_rc
16 -rw-r--r-- 1 root sys 1731 Jun 8 09:20 name_to_major
16 -rw-r--r-- 1 root root 8 Jun 7 23:47 nodename
16 -rw-r--r-- 1 root sys 780 Jun 8 07:34 nsswitch.conf
16 -r--r--r-- 1 root root 5036 Jun 8 09:28 path_to_inst
16 -rwxr--r-- 1 root sys 2792 Jan 5 2000 rc0
16 drwxr-xr-x 2 root sys 2558 Jun 8 09:32 rc0.d
16 -rwxr--r-- 1 root sys 3177 Jan 5 2000 rc1
16 drwxr-xr-x 2 root sys 2347 Jun 8 09:23 rc1.d
16 -rwxr--r-- 1 root sys 2885 Jan 5 2000 rc2
16 drwxr-xr-x 2 root sys 3456 Jun 8 09:23 rc2.d
16 -rwxr--r-- 1 root sys 2341 Jan 5 2000 rc3
16 drwxrwxr-x 2 root sys 650 Jun 8 08:11 rc3.d
16 -rwxr--r-- 1 root sys 2792 Jan 5 2000 rc5
16 -rwxr--r-- 1 root sys 2792 Jan 5 2000 rc6
32 -rwxr--r-- 1 root sys 9973 Jan 5 2000 rcS
16 drwxr-xr-x 2 root sys 3275 Jun 8 09:32 rcS.d
16 drwxr-xr-x 3 root sys 181 Jun 7 23:11 rcm
16 -r--r--r-- 1 root sys 184 Dec 18 2001 release
16 -r--r--r-- 1 root sys 3701 Jun 8 06:50 services
16 -rw-r--r-- 1 root sys 1001 Jun 7 23:12 syslog.conf
16 drwxr-xr-x 2 root other 182 Jul 3 06:00 syslog.d
16 -rw-r--r-- 1 root root 2161 Jun 8 09:49 system
16 -rw-r--r-- 1 root root 2161 Jun 8 09:49 system.GOOD
16 -rw-r--r-- 1 root other 2161 Jun 21 17:15 system.sav
16 -rw-r--r-- 1 root other 2161 Jun 24 14:39 system_06242002
16 -rw-r--r-- 1 root root 728 Jun 8 11:23 vfstab
16 -rw-r--r-- 1 root other 415 Jun 8 09:41 vfstab.prevm
16 drwxr-xr-x 9 root other 1483 Jul 3 06:01 vx

bash-2.03# ls -las ./etc/vx


total 320

Sun Proprietary: Internal Use Only


4-28 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Using the Debugging Tools and Utilities

16 drwxr-xr-x 9 root other 1483 Jul 3 06:01 .


16 drwxr-xr-x 12 root other 2426 Jul 3 06:01 ..
0 -rw-r--r-- 1 root root 0 Jun 8 09:48 .dumpadm
16 -rw-rw-rw- 1 root other 30 Jul 3 05:29 array.info
16 drwxr-xr-x 2 root other 1319 Jun 8 09:31 aslkey.d
16 lrwxrwxrwx 1 root other 17 Jul 3 06:27 bin -> /usr/lib/vxvm/bin
16 -rw-r--r-- 1 root other 1261 Jun 8 09:33 ddl.support
16 lrwxrwxrwx 1 root other 20 Jul 3 06:27 diag.d ->
/usr/lib/vxvm/diag.d
16 drwxr-xr-x 2 root sys 180 Jul 3 06:00 elm
16 -rwxrwxrwx 1 root root 16 Jun 21 16:56 guid.state
16 -rw-r--r-- 1 root other 22 Jun 8 09:33 jbod.info
16 drwxr-xr-x 3 root other 185 Jun 8 09:19 lib
16 drwxr-xr-x 6 root sys 498 Jun 8 09:49 reconfig.d
16 drwxr-xr-x 2 root root 201 Jun 8 11:56 saveconfig.d
16 drwxr-xr-x 2 root other 1101 Jun 8 09:33 slib
16 -rw-r--r-- 1 root other 5 Jun 8 09:22 sr_port
16 drwxr-xr-x 2 root sys 117 Jun 8 09:19 type
16 lrwxrwxrwx 1 root other 22 Jul 3 06:27 voladm.d ->
/usr/lib/vxvm/voladm.d
16 -rw-r--r-- 1 root root 512 Jun 8 09:48 volboot
0 prw------- 1 root other 0 Jun 8 09:20 vold_diag
0 prw------- 1 root other 0 Jun 8 09:20 vold_request
16 -rw-r--r-- 1 root other 59 Jun 8 09:41 vxdmp.exclude
16 -rw-r--r-- 1 root other 59 Jun 8 09:41 vxvm.exclude
j. The ./var directory contains the captured messages files. Use
the information in this directory to detect failure messages
logged by the VxVM software. An example is as follows:
bash-2.03# ls -las ./var
total 64
16 drwxr-xr-x 4 root other 302 Jul 3 06:01 .
16 drwxr-xr-x 9 root other 2294 Jul 3 06:27 ..
16 drwxr-xr-x 2 root other 308 Jul 3 06:00 adm
0 -rw-r--r-- 1 root other 0 Jul 3 06:01 err.out
16 drwxr-xr-x 3 root sys 180 Jul 3 05:29 vxvm

bash-2.03# more ./var/adm/messages

.
.
.

Jun 8 09:56:55 lowtide pseudo: [ID 129642 kern.info] pseudo-device: devinfo0


Jun 8 09:56:55 lowtide genunix: [ID 936769 kern.info] devinfo0 is /pseudo/devinfo@0
Jun 8 11:55:40 lowtide vxdmp: [ID 619769 kern.notice] NOTICE: vxdmp: Path failure on
118/0x84
Jun 8 11:55:40 lowtide vxdmp: [ID 997040 kern.notice] NOTICE: vxvm:vxdmp: disabled path
118/0x80 belonging to the
dmpnode 68/0x10

Sun Proprietary: Internal Use Only


Troubleshooting Tools and Utilities 4-29
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Using the Debugging Tools and Utilities

The temp_db files are also captured under the ./var/vxvm


directory. An example is as follows:
bash-2.03# ls -las ./var/vxvm/tempdb
total 48
16 drwxr-xr-x 2 root root 180 Jul 3 05:29 .
16 drwxr-xr-x 3 root sys 180 Jul 3 05:29 ..
16 -rw-r--r-- 1 root root 5120 Jul 3 05:29 rootdg
k. The ./vxvm directory contains information about the VxVM
software’s disks and disk groups. There is also a ./vxvm/dmp
directory that contains DMP debugging information. The
./dmp/dmp.out file contains the output from execution of the
autoconfig, vxdevwalk, and vxdmpinq commands, among
others.
An example of the ./vxvm directory contents is as follows:
bash-2.03# ls -las
total 464
16 drwxr-xr-x 3 root other 1953 Jul 3 06:01 .
16 drwxr-xr-x 9 root other 2294 Jul 3 06:27 ..
16 drwxr-xr-x 2 root other 181 Jul 3 06:01 dmp
16 -rw-r--r-- 1 root other 14 Jul 3 06:00 vxdctl_mode
16 -rw-r--r-- 1 root other 78 Jul 3 06:01 vxdg_list
16 -rw-r--r-- 1 root other 491 Jul 3 06:01 vxdg_list_rootdg
16 -rw-r--r-- 1 root other 841 Jul 3 06:00 vxdisk_list
16 -rw-r--r-- 1 root other 925 Jul 3 06:00 vxdisk_list_c1t0d0s2
16 -rw-r--r-- 1 root other 891 Jul 3 06:00 vxdisk_list_c1t16d0s2
16 -rw-r--r-- 1 root other 889 Jul 3 06:00 vxdisk_list_c1t17d0s2
16 -rw-r--r-- 1 root other 889 Jul 3 06:01 vxdisk_list_c1t18d0s2
16 -rw-r--r-- 1 root other 890 Jul 3 06:01 vxdisk_list_c1t19d0s2
16 -rw-r--r-- 1 root other 927 Jul 3 06:00 vxdisk_list_c1t1d0s2
16 -rw-r--r-- 1 root other 219 Jul 3 06:01 vxdisk_list_c1t20d0s2
16 -rw-r--r-- 1 root other 222 Jul 3 06:01 vxdisk_list_c1t21d0s2
16 -rw-r--r-- 1 root other 222 Jul 3 06:01 vxdisk_list_c1t22d0s2
16 -rw-r--r-- 1 root other 929 Jul 3 06:00 vxdisk_list_c1t2d0s2
16 -rw-r--r-- 1 root other 218 Jul 3 06:00 vxdisk_list_c1t3d0s2
16 -rw-r--r-- 1 root other 883 Jul 3 06:00 vxdisk_list_c1t4d0s2
16 -rw-r--r-- 1 root other 883 Jul 3 06:00 vxdisk_list_c1t5d0s2
16 -rw-r--r-- 1 root other 215 Jul 3 06:00 vxdisk_list_c1t6d0s2
16 -rw-r--r-- 1 root other 2107 Jul 3 06:00 vxdisk_s_list
32 -rw-r--r-- 1 root other 13555 Jul 3 06:00 vxkprint
16 -rw-r--r-- 1 root other 1985 Jul 3 06:00 vxprint
16 -rw-r--r-- 1 root other 2585 Jul 3 06:00 vxprint_ht
32 -rw-r--r-- 1 root other 15366 Jul 3 06:01 vxprint_mpvshr_rootdg
16 -rw-r--r-- 1 root other 432 Jul 3 06:01 vxstat_g_rootdg

bash-2.03# ls -las ./dmp


total 848
16 drwxr-xr-x 2 root other 181 Jul 3 06:01 .
16 drwxr-xr-x 3 root other 1953 Jul 3 06:01 ..
816 -rw-r--r-- 1 root other 414252 Jul 3 06:02 dmp.out

Sun Proprietary: Internal Use Only


4-30 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Using the Debugging Tools and Utilities

bash-2.03# more vxdg_list


NAME STATE ID
rootdg enabled 1023554924.1025.lowtide

bash-2.03# more vxdisk_list


DEVICE TYPE DISK GROUP STATUS
c1t0d0s2 sliced rootdisk rootdg online
c1t1d0s2 sliced rootmirror rootdg online
c1t2d0s2 sliced disk01 rootdg online spare
c1t3d0s2 sliced - - error
c1t4d0s2 sliced - - online
c1t5d0s2 sliced - - online
c1t6d0s2 sliced - - error
c1t16d0s2 sliced - - online
c1t17d0s2 sliced - - online
c1t18d0s2 sliced - - online
c1t19d0s2 sliced - - online
c1t20d0s2 sliced - - error
c1t21d0s2 sliced - - error
c1t22d0s2 sliced - - error

bash-2.03# more vxprint_ht


Disk group: rootdg

DG NAME NCONFIG NLOG MINORS GROUP-ID


DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
V NAME RVG KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO

dg rootdg default default 0 1023554924.1025.lowtide

dm disk01 c1t2d0s2 sliced 3590 17674902 SPARE


dm rootdisk c1t0d0s2 sliced 3590 17678493 -
dm rootmirror c1t1d0s2 sliced 3590 17678493 -

v rootvol - ENABLED ACTIVE 4197879 ROUND - root


pl rootvol-01 rootvol ENABLED ACTIVE 4197879 CONCAT - RW
sd rootdisk-B0 rootvol-01 rootdisk 17678492 1 0 c1t0d0 ENA
sd rootdisk-02 rootvol-01 rootdisk 0 4197878 1 c1t0d0 ENA
pl rootvol-02 rootvol ENABLED ACTIVE 4197879 CONCAT - RW
sd rootmirror-01 rootvol-02 rootmirror 0 4197879 0 c1t1d0 ENA

v swapvol - ENABLED ACTIVE 1052163 ROUND - swap


pl swapvol-01 swapvol ENABLED ACTIVE 1052163 CONCAT - RW
sd rootdisk-01 swapvol-01 rootdisk 4197878 1052163 0 c1t0d0 ENA
pl swapvol-02 swapvol ENABLED ACTIVE 1052163 CONCAT - RW
sd rootmirror-02 swapvol-02 rootmirror 4197879 1052163 0 c1t1d0 ENA

Sun Proprietary: Internal Use Only


Troubleshooting Tools and Utilities 4-31
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Using the Debugging Tools and Utilities

v usr - ENABLED ACTIVE 4197879 ROUND - fsgen


pl usr-01 usr ENABLED ACTIVE 4197879 CONCAT - RW
sd rootdisk-04 usr-01 rootdisk 5250041 4197879 0 c1t0d0 ENA
pl usr-02 usr ENABLED ACTIVE 4197879 CONCAT - RW
sd rootmirror-03 usr-02 rootmirror 5250042 4197879 0 c1t1d0 ENA

v var - ENABLED ACTIVE 4197879 ROUND - fsgen


pl var-01 var ENABLED ACTIVE 4197879 CONCAT - RW
sd rootdisk-03 var-01 rootdisk 9447920 4197879 0 c1t0d0 ENA
pl var-02 var ENABLED ACTIVE 4197879 CONCAT - RW
sd rootmirror-04 var-02 rootmirror 9447921 4197879 0 c1t1d0 ENA

bash-2.03# more ./dmp/dmp.out


DMP Debugging information

testautoconfig output

binding_name: ssd
node_name: ssd
node name: ssd
node addr: w220000203713f582,0
parent_name: sf
instance: 14
minor: a, ddi_block:wwn
binding_name: ssd
node_name: ssd
node name: ssd
node addr: w220000203713f643,0
parent_name: sf
instance: 15
minor: a, ddi_block:wwn
binding_name: ssd
node_name: ssd
node name: ssd
node addr: w220000203713e0b9,0
.
.
.

vxdevwalk output

SUNW,Ultra-Enterprise :: id=-268264420
driver properties:
pm-hardware-state value=6e6f2d73 75737065 6e642d72 6573756d 6500
ascii=no-suspend-resume.
system properties:
relative-addressing value=00000001
MMU_PAGEOFFSET value=00001fff
MMU_PAGESIZE value=00002000
PAGESIZE value=00002000
Driver packages :: id=-268251420
Driver packages/terminal-emulator :: id=-268212908

Sun Proprietary: Internal Use Only


4-32 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Using the Debugging Tools and Utilities

Driver packages/deblocker :: id=-268201280


Driver packages/obp-tftp :: id=-268199524

scsi inquiry command outputs <--- vxdmpinq exexution

/dev/rdsk/c1t0d0s2
Vendor id : SEAGATE
Product id : ST19171FCSUN9.0G
Revision : 177E
Serial Number : 9831X06256

/dev/rdsk/c1t16d0s2
Vendor id : SEAGATE
Product id : ST19171FCSUN9.0G
Revision : 177E
Serial Number : 9831X07536
.
.
.

The dmp.out file is large and contains the output of all DMP debugging
and information gathering l commands executed by vxexplorer. Use
shell commands to parse this file for the information needed to verify the
existence of a DMP problem.

The vxprivutil Utility


System administrators use the vxprivutil command to list and modify
the private region of disks. Use this command with care, because it is
capable of damaging the VxVM software configuration database beyond
repair.

Use the following syntax for this utility:


# /etc/vx/bin/vxprivutil
vxvm:vxprivutil: TO FIX: Usage:
vxprivutil [-o privoffset] scan device-pathname
vxprivutil [-o privoffset] list device-pathname
vxprivutil [-o privoffset] set device-pathname attribute ...
vxprivutil [-o poffs] [-l] dumplog device-pathname [copy-number]
vxprivutil [-o poffs] [-cf] [-C path-to-vxconfigdump] dumpconfig
device-pathname [copy-number]

Caution – The set option can damage the configuration database (private
region) beyond repair if not used properly. Some of the information
presented in this section is for reference only.

Sun Proprietary: Internal Use Only


Troubleshooting Tools and Utilities 4-33
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Using the Debugging Tools and Utilities

The dumpconfig Option

The dumpconfig option dumps any enabled configuration copy stored in


the specified private region (slice 3). The most common usage of this
option is to pipe the dumpconfig into a vxprint command as shown in
the following example:
bash-2.03# vxprivutil dumpconfig /dev/rdsk/c1t2d0s3 | vxprint -D - -ht
Disk group: rootdg

DG NAME NCONFIG NLOG MINORS GROUP-ID


DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
V NAME RVG KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO

dg rootdg default default 0 1023554924.1025.lowtide

dm disk01 - - - - SPARE
dm rootdisk - - - - -
dm rootmirror - - - - -

v rootvol - DISABLED ACTIVE 4197879 ROUND - root


pl rootvol-01 rootvol DISABLED ACTIVE 4197879 CONCAT - RW
sd rootdisk-B0 rootvol-01 rootdisk 17678492 1 0 - DIS
sd rootdisk-02 rootvol-01 rootdisk 0 4197878 1 - DIS
pl rootvol-02 rootvol DISABLED ACTIVE 4197879 CONCAT - RW
sd rootmirror-01 rootvol-02 rootmirror 0 4197879 0 - DIS

v swapvol - DISABLED ACTIVE 1052163 ROUND - swap


pl swapvol-01 swapvol DISABLED ACTIVE 1052163 CONCAT - RW
sd rootdisk-01 swapvol-01 rootdisk 4197878 1052163 0 - DIS
pl swapvol-02 swapvol DISABLED ACTIVE 1052163 CONCAT - RW
sd rootmirror-02 swapvol-02 rootmirror 4197879 1052163 0 - DIS

v usr - DISABLED ACTIVE 4197879 ROUND - fsgen


pl usr-01 usr DISABLED ACTIVE 4197879 CONCAT - RW
sd rootdisk-04 usr-01 rootdisk 5250041 4197879 0 - DIS
pl usr-02 usr DISABLED ACTIVE 4197879 CONCAT - RW
sd rootmirror-03 usr-02 rootmirror 5250042 4197879 0 - DIS

v var - DISABLED ACTIVE 4197879 ROUND - fsgen


pl var-01 var DISABLED ACTIVE 4197879 CONCAT - RW
sd rootdisk-03 var-01 rootdisk 9447920 4197879 0 - DIS
pl var-02 var DISABLED ACTIVE 4197879 CONCAT - RW
sd rootmirror-04 var-02 rootmirror 9447921 4197879 0 - DIS

Sun Proprietary: Internal Use Only


4-34 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Using the Debugging Tools and Utilities

The dumplog Option

The dumplog option lists the group log copy information stored in the
private region. An example of the dumplog option is shown here.
# ./vxprivutil dumplog /dev/rdsk/c0t3d0s3
LOG #01
BLOCK 0: KLOG 0 : COMMIT tid=0.1352
BLOCK 0: KLOG 1 : DIRTY rid=0.1154

The list Option

The list option provides output similar in format and content to the
vxdisk list command. This is a useful option to use when vxconfigd is
not running. An example is as follows:
# ./vxprivutil list /dev/rdsk/c0t3d0s3
diskid: 921260771.1030.stoli.veritas.com
group: name=rootdg id=921260769.1025.stoli.veritas.com
flags: private autoimport
hostid: stoli.veritas.com
version: 2.1
iosize: 512
public: slice=4 offset=0 len=673536
private: slice=3 offset=1 len=1151
update: time: 921263222 seqno: 0.17
headers: 0 248
configs: count=1 len=825
logs: count=1 len=125
tocblks: 0
tocs: 1/1150
Defined regions:
config priv 000017-000247[000231]: copy=01 offset=000000 enabled
config priv 000249-000842[000594]: copy=01 offset=000231 enabled
log priv 000843-000967[000125]: copy=01 offset=000000 enabled

The scan Option

The scan option produces an abbreviated disk information listing.


# ./vxprivutil scan /dev/rdsk/c0t3d0s3
diskid: 921260771.1030.stoli.veritas.com
group: name=rootdg id=921260769.1025.stoli.veritas.com
flags: private autoimport
hostid: stoli.veritas.com
version: 2.1
iosize: 512
public: slice=4 offset=0 len=673536
private: slice=3 offset=1 len=1151
update: time: 921263222 seqno: 0.17
headers: 0 248

Sun Proprietary: Internal Use Only


Troubleshooting Tools and Utilities 4-35
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Using the Debugging Tools and Utilities

configs: count=1 len=825


logs: count=1 len=125

The set Option

Use the set option to change the attributes of a disk or disk group
without using vxconfigd.

Caution – Use the set option carefully. This option can cause irreversible
damage to the configuration database in a disk’s private region and
damage it beyond repair.

Use the set option to modify the following attributes:


{ "diskid", optsetstr, DTAG(dka_diskid) },
{ "hostid", optsetstr, DTAG(dka_hostid) },
{ "dgid", optsetstr, DTAG(dka_dgid) },
{ "dg_name", optsetstr, DTAG(dka_dg_name) },
{ "dgname", optsetstr, DTAG(dka_dg_name) },
{ "dg", optsetstr, DTAG(dka_dg_name) },
{ "shared", optsetbool, OFFS(struct disk_attrs, dka_flags), DKA_FLAG_SHARED},
{ "autoimport", optsetbool, OFFS(struct disk_attrs,dka_flags),
DKA_FLAG_AUTOIMPORT },

Example Use of the set Option

See the following for how to use the set option. In this example, a new
disk group initialization failed because disk c1t15d0 is owned by rootdg.
# vxdg init newdg disk05=c1t15d0s2
vxvm:vxdg: ERROR: Device c1t15d0s2 appears to be owned by disk group rootdg.

# vxdisk list c1t15d0


Device: c1t15d0s2
devicetag: c1t15d0
type: sliced
hostid: plstr04.veritas.com
disk: name= id=968346728.1264.plstr04.veritas.com
group: name=rootdg id=968426567.1289.plstr04.veritas.com
flags: online ready private autoconfig noautoimport
pubpaths: block=/dev/vx/dmp/c1t15d0s4 char=/dev/vx/rdmp/c1t15d0s4
privpaths: block=/dev/vx/dmp/c1t15d0s3 char=/dev/vx/rdmp/c1t15d0s3
version: 2.1
iosize: min=512 (bytes) max=2048 (blocks)
public: slice=4 offset=0 len=17877575
private: slice=3 offset=1 len=3049
update: time=968446320 seqno=0.18
headers: 0 248

Sun Proprietary: Internal Use Only


4-36 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Using the Debugging Tools and Utilities

configs: count=1 len=2226


logs: count=1 len=337
Defined regions:
config priv 000017-000247[000231]: copy=01 offset=000000 enabled
config priv 000249-002243[001995]: copy=01 offset=000231 enabled
log priv 002244-002580[000337]: copy=01 offset=000000 enabled
Multipathing information:
numpaths: 1
c1t15d0s2 state=enabled

An attempt to import a disk group locks clearing import (-C), as shown in


the following output:
# vxdg -C import 968426567.1289.plstr04.veritas.com
vxvm:vxdg: ERROR: Disk group 968426567.1289.plstr04.veritas.com: import failed:
Record already exists in disk group

Use the vxprivutil command to clear the dgname, as follows:


# ./vxprivutil set /dev/rdsk/c1t15d0s3 dgname=""
# vxdisk list c1t15d0
Device: c1t15d0s2
devicetag: c1t15d0
type: sliced
hostid: plstr04.veritas.com
disk: name= id=968346728.1264.plstr04.veritas.com
group: name= id=968426567.1289.plstr04.veritas.com
flags: online ready private autoconfig noautoimport
pubpaths: block=/dev/vx/dmp/c1t15d0s4 char=/dev/vx/rdmp/c1t15d0s4
privpaths: block=/dev/vx/dmp/c1t15d0s3 char=/dev/vx/rdmp/c1t15d0s3
version: 2.1
iosize: min=512 (bytes) max=2048 (blocks)
public: slice=4 offset=0 len=17877575
private: slice=3 offset=1 len=3049
update: time=968446579 seqno=0.21
headers: 0 248
configs: count=1 len=2226
logs: count=1 len=337
Defined regions:
config priv 000017-000247[000231]: copy=01 offset=000000 enabled
config priv 000249-002243[001995]: copy=01 offset=000231 enabled
log priv 002244-002580[000337]: copy=01 offset=000000 enabled
Multipathing information:
numpaths: 1
c1t15d0s2 state=enabled

Next, import the disk group again:


# vxdg import 968426567.1289.plstr04.veritas.com
# vxdisk list
DEVICE TYPE DISK GROUP STATUS
c0t0d0s2 sliced rootdisk rootdg online
c1t8d0s2 sliced - - online

Sun Proprietary: Internal Use Only


Troubleshooting Tools and Utilities 4-37
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Using the Debugging Tools and Utilities

c1t9d0s2 sliced - - online


c1t10d0s2 sliced - - online
c1t11d0s2 sliced - - online
c1t12d0s2 sliced disk01 m9dg online
c1t13d0s2 sliced disk02 m9dg online
c1t14d0s2 sliced disk03 m9dg online
c1t15d0s2 sliced disk04 m9dg online

The vxdevwalk Utility


The vxdevwalk utility scans the kernel dev_info tree and prints detailed
information about disk devices. This information includes physical device
paths, instance information, driver properties, and major and minor
numbers for each possible slice on the disk. For example:
Driver sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w210000203713f579,0 :: id=18 instance=11
driver properties:
inquiry-revision-id value=31373745 00
ascii=177E.
inquiry-product-id value=53543139 31373146 4353554e 392e3047 00
ascii=ST19171FCSUN9.0G.
inquiry-vendor-id value=53454147 41544500
ascii=SEAGATE.
pm-hardware-state value=6e656564 732d7375 7370656e 642d7265 73756d65
00
ascii=needs-suspend-resume.
ddi-kernel-ioctl
device nodes:
a :: dev=118,88:block nodetype=ddi_block:wwn
b :: dev=118,89:block nodetype=ddi_block:wwn
c :: dev=118,90:block nodetype=ddi_block:wwn
d :: dev=118,91:block nodetype=ddi_block:wwn
e :: dev=118,92:block nodetype=ddi_block:wwn
f :: dev=118,93:block nodetype=ddi_block:wwn
g :: dev=118,94:block nodetype=ddi_block:wwn
h :: dev=118,95:block nodetype=ddi_block:wwn
a,raw :: dev=118,88:char nodetype=ddi_block:wwn
b,raw :: dev=118,89:char nodetype=ddi_block:wwn
c,raw :: dev=118,90:char nodetype=ddi_block:wwn
d,raw :: dev=118,91:char nodetype=ddi_block:wwn
e,raw :: dev=118,92:char nodetype=ddi_block:wwn
f,raw :: dev=118,93:char nodetype=ddi_block:wwn
g,raw :: dev=118,94:char nodetype=ddi_block:wwn
h,raw :: dev=118,95:char nodetype=ddi_block:wwn

Sun Proprietary: Internal Use Only


4-38 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Using the Debugging Tools and Utilities

Use this information to correlate the kernel space information with user
space entries by performing a long list on the /dev/dsk directory and
grep on the wwn or major and minor numbers of the listed device. For
example:
bash-2.03# ls -las /dev/dsk | grep 3f579
2 lrwxrwxrwx 1 root root 70 Jun 7 23:47 c1t2d0s0 ->
../../devices/sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203713f579,0:a
2 lrwxrwxrwx 1 root root 70 Jun 7 23:47 c1t2d0s1 ->
../../devices/sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203713f579,0:b
2 lrwxrwxrwx 1 root root 70 Jun 7 23:47 c1t2d0s2 ->
../../devices/sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203713f579,0:c
2 lrwxrwxrwx 1 root root 70 Jun 7 23:47 c1t2d0s3 ->
../../devices/sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203713f579,0:d
2 lrwxrwxrwx 1 root root 70 Jun 7 23:47 c1t2d0s4 ->
../../devices/sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203713f579,0:e
2 lrwxrwxrwx 1 root root 70 Jun 7 23:47 c1t2d0s5 ->
../../devices/sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203713f579,0:f
2 lrwxrwxrwx 1 root root 70 Jun 7 23:47 c1t2d0s6 ->
../../devices/sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203713f579,0:g
2 lrwxrwxrwx 1 root root 70 Jun 7 23:47 c1t2d0s7 ->
../../devices/sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203713f579,0:h
2 lrwxrwxrwx 1 root root 70 Jun 7 23:47 c2t2d0s0 ->
../../devices/sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w210000203713f579,0:a
2 lrwxrwxrwx 1 root root 70 Jun 7 23:47 c2t2d0s1 ->
../../devices/sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w210000203713f579,0:b
2 lrwxrwxrwx 1 root root 70 Jun 7 23:47 c2t2d0s2 ->
../../devices/sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w210000203713f579,0:c
2 lrwxrwxrwx 1 root root 70 Jun 7 23:47 c2t2d0s3 ->
../../devices/sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w210000203713f579,0:d
2 lrwxrwxrwx 1 root root 70 Jun 7 23:47 c2t2d0s4 ->
../../devices/sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w210000203713f579,0:e
2 lrwxrwxrwx 1 root root 70 Jun 7 23:47 c2t2d0s5 ->
../../devices/sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w210000203713f579,0:f
2 lrwxrwxrwx 1 root root 70 Jun 7 23:47 c2t2d0s6 ->
../../devices/sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w210000203713f579,0:g
2 lrwxrwxrwx 1 root root 70 Jun 7 23:47 c2t2d0s7 ->
../../devices/sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w210000203713f579,0:h

If the device is multi-pathed, this output shows both paths. In the


previous example, controllers c1 and c2 are multiple paths for a single
device.

Sun Proprietary: Internal Use Only


Troubleshooting Tools and Utilities 4-39
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Using the Debugging Tools and Utilities

The vxkprint Utility


The vxkprint utility prints kernel space information about VxVM
software objects, such as disk groups, disks, and volumes. Output from
this utility can be useful if other VxVM software commands are not
providing useful information about a VxVM software object.

Note – The output of this command can be very large. Consider re-
directing the command output to a file and using a text editor to search
the file contents.

The following example shows truncated output from a vxkprint


command:
.
.
.
# Groups: (cnt: 2)
#
Group NULLDG: iid=0.0 id=
version: 0
configtid=0.0 logsize=0
kflag=()
vflag=(created|notempdb|noautoreimport)
diskdetpolicy= 0
# Group-Objects: (cnt: 0)
# End-group: NULLDG

Group rootdg: iid=0.1 id=1023554924.1025.lowtide


version: 90
configtid=0.1351 logsize=398
kflag=(activated_ew)
vflag=(created|tempdb|autoreimport)
diskdetpolicy= 1
# Group-Objects: (cnt: 21)
.
.
.

Disk group information presented by this command shows that disk


group rootdg is imported by system lowtide. The disk group revision
level is 90.

Sun Proprietary: Internal Use Only


4-40 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Using System-Level Debugging Utilities

Using System-Level Debugging Utilities


The Solaris OE has system level debugging utilities that help debug
problems which cause the system to panic. When the system panics, a
panic dump file is usually written. Some system-level utilities include:
● adb
● kadb
● crash

System administrators use these utilities to analyze panic dump files. The
VxVM software could at some time cause a system panic. It is helpful to
be able to diagnose a panic dump file to determine whether the VxVM
software was responsible for the crash and what component of the VxVM
software failed.

Note – Analyzing panic dump files is beyond the scope of this class. Sun
Education provides classes that teach the use of system-level debugging
tools to analyze crash dump files.

Sun Proprietary: Internal Use Only


Troubleshooting Tools and Utilities 4-41
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Using the Error Logging and Debugging Utilities

Exercise: Using the Error Logging and Debugging Utilities


In this exercise, you will complete the following tasks:
● Enable vxconfigd debugging
● Generate simple errors
● View error messages in the messages log and the vxconfigd log
● Use the VERITAS Volume Manager™ 3.2 Troubleshooting Guide for an
explanation of error message
● Run VRTSexplorer and view data
● Use vxprivutil to display the contents of a disk’s private region
● Use vxdevwalk to display dev_info tree data

Preparation
To prepare for this exercise, make sure that the VxVM software is installed
and operational.

Task 1 – Enabling vxconfigd Debug Logging


Complete the following steps:
1. Enable maximum vxconfigd logging from the command line. What
command did you use?
_____________________________________________________________
2. What is the location of the log?
_____________________________________________________________
3. View the contents of the log. Is there data present?
_____________________________________________________________
_____________________________________________________________
_____________________________________________________________
4. Disable vxconfigd logging. What command did you use?
_____________________________________________________________
5. Remove the vxconfigd log.

Sun Proprietary: Internal Use Only


4-42 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Using the Error Logging and Debugging Utilities

Task 2 – Viewing the VxVM Software Messages


Complete the following steps:
1. Enable maximum vxconfigd logging. What command did you use?
_____________________________________________________________
2. Open a new window, and view the contents of the vxconfigd log in
real time. What command did you use?
_____________________________________________________________
3. Open a new window, and view the current messages file in real time.
What command did you use?
_____________________________________________________________
4. Pull a fiber cable.
5. Perform an I/O operation to the attached disks by executing the
following command:
# find / -name “foo.txt”
6. View the messages and vxconfigd log as the DMP notices that a
path is missing.
7. After the logging completes, find the error message logged by DMP
in the messages file, and locate the appropriate message in the
VERITAS Volume Manager™ 3.2 Troubleshooting Guide.
Answer the following questions:
● What error messages from the messages file were you able find
in the VERITAS Volume Manager™ 3.2 Troubleshooting Guide?
________________________________________________________
________________________________________________________
________________________________________________________
________________________________________________________
________________________________________________________
● What is the cxtxdx address of the failed path?
________________________________________________________
________________________________________________________
● What is the physical path of the failed path?
________________________________________________________
________________________________________________________

Sun Proprietary: Internal Use Only


Troubleshooting Tools and Utilities 4-43
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Using the Error Logging and Debugging Utilities

Power user – The following task is for students who have the skills to
map a device tree entry to a physical hardware slot.
If you have access to SunSolve Online, pull the hardware mapping
for the system used by your lab group and map the physical device
address of the failing path to the cable that was disconnected.
What is the physical Peripheral Component Interconnect (PCI) or
SBus slot of the pulled cable?
_____________________________________________________________
8. Replace the pulled cable.
9. View any messages logged by the VxVM software as the path is
brought online.
_____________________________________________________________
_____________________________________________________________
10. Disable vxconfigd logging.
11. Close the real-time viewing window for the vxconfigd log.
12. Remove the vxconfigd log.

Task 3 – Using VxVM Software Debug Utilities


Complete the following steps:
1. Install the VRTSspt package, if it is not already installed.
2. Run the VRTSexplorer utility using the procedure described in ‘‘The
vxexplorer Utility’’ on page 4-18.
3. Browse the output, and view the following files:
● hostid
● uname_a
● pkginfo_l – Search for the VxVM software packages and
verify the package levels and installation dates.
● vxlicense_p
● prtdiag
● eeprom

Sun Proprietary: Internal Use Only


4-44 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Using the Error Logging and Debugging Utilities

● Contents of various files under the ./dev subdirectory


● Contents of the various files under the ./vxvm subdirectory.
4. Run the vxprivutil command with the dumpconfig option on a
selected VxVM software disk. Pipe the output to the vxprint
command to display a vxprint -ht type of printout.
What command did you use?
_____________________________________________________________
5. Run the vxprivutil command with the list option on a selected
VxVM software disk, and view the output.
6. Run the vxdevwalk command, and view the output.
7. Run the vxkprint command, and view the output.

Sun Proprietary: Internal Use Only


Troubleshooting Tools and Utilities 4-45
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise Summary

Exercise Summary

Discussion – Take a few minutes to discuss what experiences, issues, or


discoveries you had during the lab exercise.
!
?
● Experiences
● Interpretations
● Conclusions
● Applications

Sun Proprietary: Internal Use Only


4-46 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Using the Error Logging and Debugging Utilities

Exercise: Using the Error Logging and Debugging Utilities


The solutions to the lab exercises are provided in this section.

Task 1 Solutions
Complete the following steps:
1. Enable maximum vxconfigd logging from the command line. What
command did you use?

# pkill -9 vxconfigd; vxconfigd -x 9 -x log -x syslog&


2. What is the location of the log?

In /var/vxvm/vxconfigd.log.
3. View the contents of the log. Is there data present?
_____________________________________________________________
_____________________________________________________________
4. Disable vxconfigd logging. What command did you use?

# pkill -9 vxconfigd; vxconfigd -m enable&


5. Remove the vxconfigd log.

Task 2 Solutions
Complete the following steps:
1. Enable maximum vxconfigd logging. What command did you use?

# pkill -9 vxconfigd; vxconfigd -x 9 -x log -x syslog&


2. Open a new window, and view the contents of the vxconfigd log in
real time. What command did you use?

# tail -f /var/vxvm/vxconfigd.log
3. Open a new window and view the current messages file in real time.
What command did you use?

# tail -f /var/adm/messages

Sun Proprietary: Internal Use Only


Troubleshooting Tools and Utilities 4-47
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Using the Error Logging and Debugging Utilities

4. Pull a fiber cable.


5. Perform an I/O operation to the attached disks by executing the
following command:
# find / -name “foo.txt”
6. View the messages and vxconfigd log as the DMP notices that a
path is missing.
7. After the logging completes, find the error message logged by DMP
in the messages file and locate the appropriate message in the
VERITAS Volume Manager™ 3.2 Troubleshooting Guide.
Answer the following questions:
● What error messages from the messages file were you able find
in the VERITAS Volume Manager™ 3.2 Troubleshooting Guide?
________________________________________________________
________________________________________________________
________________________________________________________
________________________________________________________
________________________________________________________
● What is the cxtxdx address of the failed path?
________________________________________________________
________________________________________________________
● What is the physical path of the failed path?
________________________________________________________
________________________________________________________

Power user – The following task is for students who have the skills to
map a device tree entry to a physical hardware slot.
If you have access to SunSolve Online, pull the hardware mapping
for the system used by your lab group and map the physical device
address of the failing path to the cable that was disconnected.
What is the physical Peripheral Component Interconnect (PCI) or
SBus slot of the pulled cable?
_____________________________________________________________
8. Replace the pulled cable.

Sun Proprietary: Internal Use Only


4-48 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Using the Error Logging and Debugging Utilities

9. View any messages logged by the VxVM software as the path is


brought online.
_____________________________________________________________
_____________________________________________________________
10. Disable vxconfigd logging.
11. Close the real-time viewing window for the vxconfigd log.
12. Remove the vxconfigd log.

Task 3 Solutions
Complete the following steps:
1. Install the VRTSspt package, if it is not already installed.
2. Run the VRTSexplorer utility using the procedure described in ‘‘The
vxexplorer Utility’’ on page 4-18.
3. Browse the output, and view the following files:
● hostid
● uname_a
● pkginfo_l – Search for the VxVM software packages and
verify the package levels and installation dates.
● vxlicense_p
● prtdiag
● eeprom
● Contents of various files under the ./dev subdirectory
● Contents of the various files under the ./vxvm subdirectory.
4. Run the vxprivutil command with the dumpconfig option on a
selected VxVM software disk. Pipe the output to the vxprint
command to display a vxprint -ht type of printout.
What command did you use?

# vxprivutil dumpconfig /var/rdsk/cxtxdxs3 | vxprint -D - -


ht

Sun Proprietary: Internal Use Only


Troubleshooting Tools and Utilities 4-49
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Using the Error Logging and Debugging Utilities

5. Run the vxprivutil command with the list option on a selected


VxVM software disk, and view the output.
6. Run the vxdevwalk command, and view the output.
7. Run the vxkprint command, and view the output.

Sun Proprietary: Internal Use Only


4-50 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Module 5

Recovering Boot and System Processes

Objectives
Upon completion of this module, you should be able to:
● Describe the VxVM software system recovery processes
● Describe how the VxVM software is initialized during system boot
● Successfully troubleshoot boot problems that prevent the VxVM
software from starting
● Identify errors that prevent the VxVM software from functioning
● Use the correct recovery procedures to resolve initialization and
operational problems
● Correctly determine when to reinstall the VxVM software
● Identify the VxVM software errors, match these errors to a list of
known errors and successfully repair the problems

Sun Proprietary: Internal Use Only


5-1
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Relevance

Relevance

Discussion – The following questions are relevant to understanding what


the VxVM software boot and recovery processes are:
!
?
● What files are accessed during system boot to initialize the VxVM
software?
● What entries in the /etc/system file are necessary for the VxVM
software initialization?
● What recovery procedure is used on a system that has a problem
preventing the VxVM software from starting?
● What procedures can be used to boot a system without starting the
the VxVM software?
● What recovery procedure is used on a system that has corrupted
VxVM software binaries?
● What VxVM software configuration allows multiple rootdgs to co-
exist on a single system?

Sun Proprietary: Internal Use Only


5-2 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Additional Resources

Additional Resources

Additional resources – The following references provide additional


information on the topics described in this module:
● VERITAS Volume Manager™ 3.2 Administrator’s Guide. Mountain
View, California: VERITAS Software Corporation, August 2001,
number 30-000392-011, TechPDF ID 240253.
● VERITAS Volume Manager™ 3.2 Installation Guide. Mountain View,
California: VERITAS Software Corporation, August 2001,
number 30-000395-011, TechPDF ID 240256.
● VERITAS Volume Manager™ 3.2 Troubleshooting Guide. Mountain
View, California: VERITAS Software Corporation, August 2001,
number 30-000394-011, TechPDF ID 240255.
● VERITAS Volume Manager™ Storage Administrator 3.2 Administrator’s
Guide. Mountain View, California: VERITAS Software Corporation,
July 2001, number 30-000393-011, TechPDF ID 240257.
● SunSolveSM Online SRDB 24657,
[http://sunsolve.Sun.COM/pub-cgi/search.pl?mode=advanced].
● http://storage.east, http://storage.central, and
http://storage.west.

Sun Proprietary: Internal Use Only


Recovering Boot and System Processes 5-3
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Surveying VxVM Software System Recovery Processes

Surveying VxVM Software System Recovery Processes


The VxVM software utilities, commands and devices are only available
when the VxVM software is functional. Recovery of the VxVM software
and the ability to improve the fault resiliency of storage devices under
VxVM software management is critical for maintaining servers in
production environments. The VxVM software recovery processes and
procedures presented in this guide are organized as follows:
● Boot process failures that prevent the VxVM software from starting
● Failures of the VxVM software which prevent it from functioning
once the system is booted
● Storage device errors

This module addresses errors that occur in the boot process and the
VxVM software functionality. Storage device errors are addressed in
Module 6, “Recovering Disk, Disk Group, and Volume Failures.”

Sun Proprietary: Internal Use Only


5-4 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining the VxVM Software Boot Process

Examining the VxVM Software Boot Process


The VxVM software initializes as part of the Solaris OE boot processing.
This section lists the processes initiated by the startup scripts used to
initialize the VxVM software. Startup script processing is divided into
actions initiated by scripts executing in single-user mode and those
executing in multi-user mode. Figure 5-1 illustrates when these scripts
execute.

/etc/rcS.d

Single-User Start-Script
Execution

Boot System
Ready
/etc/rc2.d

Multi-User Start-Script
Execution

Figure 5-1 Boot Scripts

Scripts which execute in single-user mode that affect the VxVM software
initialization are:
● /etc/rcS.d/S25vxvm-sysboot
● /etc/rcS.d/S35vxvm-startup1
● /etc/rcs.d/S40standardmounts
● /etc/rcS.d/S50drvconfig
● /etc/rcS.d/S60devlinks
● /etc/rcS.d/S85vxvm-startup2
● /etc/rcS.d/S86vxvm-reconfig

Sun Proprietary: Internal Use Only


Recovering Boot and System Processes 5-5
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining the VxVM Software Boot Process

Scripts which execute in multi-user mode that affect the VxVM software
initialization are:
● /etc/rc2.d/S20sysetup
● /etc/rc2.d/S94vxnm-host_infod
● /etc/rc2.d/S94vxnm-vxnetd
● /etc/rc2.d/S95vxvm-recover
● /etc/rc2.d/S96vmsa-server

Note – Appendix D, “The Boot Process,” contains an example of the


boot -v process, with annotations which indicate when each of these
scripts execute. Refer to this appendix for more information on processes
that execute during boot processing.

Single-User Boot Processing


This section describes the startup scripts that execute during the single-
user mode of boot processing.

The S25vxvm-sysboot Script

Figure 5-2 illustrates the actions started during the execution of the
/etc/rcS.d/S25vxvm-sysboot script.

Disk access records are created.

The rootdg ownership is


determined.
/etc/rcS.d/S25vxvm-
sysboot
VxVM software is started in boot
mode.

If the boot disk is on a stale plex, a


new mirror is suggested.

Figure 5-2 The /etc/rcS.dvxvm-sysboot Script

Sun Proprietary: Internal Use Only


5-6 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining the VxVM Software Boot Process

Exclusion of certain DMP devices was implemented in the VxVM


software version 3.1. Information about excluded drivers is downloaded
to DMP by invoking vxdmpadm with the doioctl option prior to the start
of vxconfigd by the S25vxvm-sysboot script. The VxVM software
version 3.2 does not use the vxdmpadm doioctl command; this
functionality is now incorporated into the vxconfigd process.

Disk Access Record Creation

Disk access records are created for all devices, as follows:


● The dev_info tree is scanned for new devices.
If the Solaris OE does not see a disk using format, the VxVM
software does not see the disk.
● Kernel and user-space entries are matched.
These are the /dev/dsk links. If a user-space entry does not exist, the
VxVM software guesses what it should be.
● New names are given to unmatched entries.

The dev_info tree is a kernel structure that is built from device tree
information in the boot PROM. To view the dev_info tree, use the
/etc/vx/diag.d/vxdevwalk command.

An example shows an excerpt from the execution of vxdevwalk


command. The following example shows the dev_info tree data for device
ssd@w220000203713fc9f.
Driver sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203713fc9f,0 :: id=25 instance=18
driver properties:
inquiry-revision-id value=31373745 00
ascii=177E.
inquiry-product-id value=53543139 31373146 4353554e 392e3047 00
ascii=ST19171FCSUN9.0G.
inquiry-vendor-id value=53454147 41544500
ascii=SEAGATE.
pm-hardware-state value=6e656564 732d7375 7370656e 642d7265 73756d65
00
ascii=needs-suspend-resume.
ddi-kernel-ioctl
device nodes:
a :: dev=118,144:block nodetype=ddi_block:wwn
b :: dev=118,145:block nodetype=ddi_block:wwn
c :: dev=118,146:block nodetype=ddi_block:wwn
d :: dev=118,147:block nodetype=ddi_block:wwn
e :: dev=118,148:block nodetype=ddi_block:wwn
f :: dev=118,149:block nodetype=ddi_block:wwn
g :: dev=118,150:block nodetype=ddi_block:wwn

Sun Proprietary: Internal Use Only


Recovering Boot and System Processes 5-7
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining the VxVM Software Boot Process

h :: dev=118,151:block nodetype=ddi_block:wwn
a,raw :: dev=118,144:char nodetype=ddi_block:wwn
b,raw :: dev=118,145:char nodetype=ddi_block:wwn
c,raw :: dev=118,146:char nodetype=ddi_block:wwn
d,raw :: dev=118,147:char nodetype=ddi_block:wwn
e,raw :: dev=118,148:char nodetype=ddi_block:wwn
f,raw :: dev=118,149:char nodetype=ddi_block:wwn
g,raw :: dev=118,150:char nodetype=ddi_block:wwn
h,raw :: dev=118,151:char nodetype=ddi_block:wwn

Matching Driver String Entries

User-space entries are located in the /dev/dsk directory. For example:


bash-2.03# ls -las /dev/dsk/c1t0d0s2
2 lrwxrwxrwx 1 root root 70 Jun 7 23:47 /dev/dsk/c1t0d0s2 ->
../../devices/sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203713fc9f,0:c

The entries are matched by the dev_info tree entry, shown in the
following example.
Driver sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203713fc9f,0 :: id=25 instance=18

Unmatched Entries

Unmatched entries are given new names, as follows:


● Entries are identified in the dev_info tree.
● If a match is not located in user space, an arbitrary name is created.

Disk Access Records

Use the vxdisk list command to display all user-space disk access
records. The following example shows typical output from the vxdisk
list command.
bash-2.03# vxdisk list
DEVICE TYPE DISK GROUP STATUS
c1t0d0s2 sliced rootdisk rootdg online
c1t1d0s2 sliced rootmirror rootdg online
c1t2d0s2 sliced disk01 rootdg online spare
c1t3d0s2 sliced - - error
c1t4d0s2 sliced - - online
c1t5d0s2 sliced - - online
c1t6d0s2 sliced - - error
c1t16d0s2 sliced - - online
c1t17d0s2 sliced - - online
c1t18d0s2 sliced - - online
c1t19d0s2 sliced - - online

Sun Proprietary: Internal Use Only


5-8 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining the VxVM Software Boot Process

c1t20d0s2 sliced - - error


c1t21d0s2 sliced - - error
c1t22d0s2 sliced - - error

Note – Only disks that are members of imported disk groups have entries
in the DISK and GROUP columns.

Disk Ownership

The /etc/vx/volboot file is used to delineate disk ownership. The


vxdctl process uses this file to manage the state of vxconfigd and to
bootstrap rootdg during the VxVM software initialization. The following
example illustrates a basic volboot file.
volboot 3.1 0.2 30
hostid lowtide
end
###############################################################
###############################################################
###############################################################
###############################################################
###############################################################
###############################################################
###############################################################
#########################

If simple disks are part of a server’s VxVM software configuration, these


disks must be listed in the volboot file, or the VxVM software does not
recognize them. The following example illustrates a /etc/vx/volboot
file that includes information on simple disks.
volboot 3.1 0.2 30
hostid lowtide
disk c1t6d0s4 simple privoffset=1
disk c2t4d0s3 simple privoffset=1
disk c1t2d0s5 simple privoffset=1
end
###############################################################
###############################################################
###############################################################
###############################################################
###############################################################
######################################################

This file must be 512 bytes in length, including padding characters. Do not
edit this file using vi or other text editor. If this file is corrupted, the
VxVM software initialization fails.

Sun Proprietary: Internal Use Only


Recovering Boot and System Processes 5-9
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining the VxVM Software Boot Process

To update the /etc/vx/volboot file, use the following commands:


● vxdctl hostid newname
● vxdctl init newname

Running the vxdctl init newname command recreates the


/etc/vx/volboot file.

Caution – If simple disks are part of a server’s VxVM disk configuration,


these entries are deleted if a vxdctl init newname command is run.
Record this information prior to running this command to allow
restoration of the simple disk configuration using the vxdctl add disk
c#t#d#s# command.

Boot Mode

During this part of the VxVM software initialization, the software is


started in boot mode using the vxconfigd -m boot command. The
following initialization operations occur at this time:
● The rootdg disk group is imported.
● If the boot disk is under the VxVM software control, rootvol and
user volumes are started.

Note – If the /etc/rcS.d/S25vxvm-sysboot file is modified for


vxconfigd debugging using the vxconfigd -x 1-9 command, it starts in
debug mode at this time. The 1-9 option increases the level of debugging
information which vxconfigd displays during initialization. Other debug
sub-options are available for use with the -x 1-9 option and are listed in
the vxconfigd man page.

Boot Disk Stale Plex

During /etc/rcS.d/S25vxvm-sysboot processing, booting from a stale


plex is checked. If a stale boot disk plex is detected, the following actions
occur:
1. The boot process stops.
2. An alternate mirror is suggested, if one exists.

Sun Proprietary: Internal Use Only


5-10 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining the VxVM Software Boot Process

An original equipment manufacturer (OEM)-supported script can be


created to automatically suggest the alternate mirror. The
/etc/vx/sbin/vxstale file is no longer packaged with the VxVM
software distribution.

Flag Files Are Checked

The following flag files are checked. The files can affect the way in which
the VxVM software is initialized:
● /etc/vx/reconfig.d/state.d/install-db –
● This file is created during installation of the VxVM software
packages.
This file is also created if an installation of the VxVM software
is incomplete. In that case, if the boot disk is under the VxVM
software control, the system does not boot past single-user
mode.
● This file prevents vxconfigd from starting when the system
boots. The VxVM software starts if the boot disk is under the
VxVM software control, but it is crippled.

Note – Although vxconfigd may not be started, the vxdmp and vxspec
modules are started.

● /VXVM#.#.#-UPGRADE/.start_runed –
● The value #.#.# is the software level to which the system is
being upgraded, such as 3.1.1.
● This is a hidden file, created by the upgrade_start script. It is
removed when the upgrade is finished.
● This file prevents vxconfigd from starting even if the boot disk
is under the VxVM software control.

Sun Proprietary: Internal Use Only


Recovering Boot and System Processes 5-11
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining the VxVM Software Boot Process

The S35vxvm-startup1 Script

The processes started by execution of the S35vxvm-startup1 script are


illustrated in Figure 5-3.

Start special volumes.

/etc/rcS.d/
S35vxvm-startup1

Setup dump devices.

Figure 5-3 The /etc/rcS,d/S35vxvm-startup1 Script

This script executes after the / and /usr volumes are available and makes
other volumes available that are needed early in the Solaris OE boot
sequence.

Special Volumes

The following special volumes, if configured, are started:


● swap
● /var

Note – These volumes must be in rootdg directory.

Recovery operations are not performed, including:


● Mirror re-synchronization
● Parity re-synchronization

Sun Proprietary: Internal Use Only


5-12 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining the VxVM Software Boot Process

Dump Device

The dump device is used to store core information when the system
panics. Dump device configuration is as follows:
● A swap device must be listed in the /etc/vfstab file.
● The swap device must be in rootdg.
● A physical partition must be available underneath the swap volume.
Swap files cannot be used as a dump device.
● The dump device is registered by adding and then removing the swap
device. The VxVM software does not have hooks for dumping, so the
swap device must be created prior to the creation of the dump
device.
The dump device must be the first swap device listed in the
/etc/vfstab file.
● Core file creation and recovery are performed outside of the VxVM
software operations.

The S40standardmounts Script

The S40standardmounts script sets the swap device. This script is not
part of the VxVM software Swap devices have the following
characteristics:
● The VxVM software treats all file systems with file type swap in the
/etc/vfstab file as swap volumes.
● The primary swap volume must be:
● Usage type swap
● A real partition
● In rootdg
● Swap device size is limited to 2000 megabytes in the early releases of
the Solaris OE.

Sun Proprietary: Internal Use Only


Recovering Boot and System Processes 5-13
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining the VxVM Software Boot Process

The S50devfsadm Script

Figure 5-4 illustrates the processes started by the S50devfsadm script. This
script is not part of the VxVM software but is used to scan the device tree
and build new devices.

Scan dev_info tree for new


devices.

New entries are created in


/devices.
/etc/rcS.d/S50devfsadm

Create new /dev/dsk


and
/dev/rdsk entries.

Figure 5-4 The /etc/rcS.d/S50devfsadm Script

The /etc/rcS.d/S50devfsadm script has the following characteristics:


● The script executes during all boots.
● When the script executes as part of a reconfiguration reboot, new
devices are discovered and configured. To perform a reconfiguration
reboot, do the following:
1. Use the boot -r command.
2. Set the /reconfigure flag.
● Solaris OE releases prior to release 8 used the
/etc/rcS.d/S50drvconfig and S60devlinks scripts.

Sun Proprietary: Internal Use Only


5-14 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining the VxVM Software Boot Process

Discovery of new devices can be performed without a system reboot


using the following commands:
● In the Solaris 8 OE:
● devfsadm
● vxdctl enable
● In Solaris 7 OE and earlier:
● drvconfig
● disks
● devlinks
● vxdctl enable

The S85vxvm-startup2 Script

The processes started by the S85vxvm-startup2 script are illustrated in


Figure 5-5.

The vxconfigd is switched from Boot


mode to Enable mode.

The dev_info
tree is checked for new devices.

All auto-import disk groups are


imported.
/etc/rcS.d/S85vxvm-startup2

All volumes are started.

Failed disk media records are attached


to Disk Access records.

Figure 5-5 The /etc/rcS.d/S85vxvm-startup2 Script

Sun Proprietary: Internal Use Only


Recovering Boot and System Processes 5-15
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining the VxVM Software Boot Process

The /etc/rcS.d/S85vxvm-startup2 script must be run after the /, /usr,


and /var volumes are mounted. This script:
● Starts some I/O daemons
● Rebuilds the /dev/vx/dsk and /dev/vx/rdsk directories
● Imports all disk groups
● Starts all volumes that were not started earlier in the boot sequence

Other activities which affect the /etc/rcS.d/S85vxvm-startup2 script


are:
● Flag files may affect this process.
● Volume recovery is not performed.
● New device entries are added, and invalid entries are kept.

The S86vxvm-reconfig Script

The processes started by the S86vxvm-reconfig script are shown in


Figure 5-6.

Flag files are queried to determine


action.

/etc/rcS.d/S86vxvm-reconfig New disks are added.

Encapsulations are performed.

Figure 5-6 The /etc/rcS.d/S86vxvm-reconfig Script

Sun Proprietary: Internal Use Only


5-16 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining the VxVM Software Boot Process

Flag Files

Flag files are queried for reconfiguration and reboot procedures. The
/etc/vx/reconfig.d/state.d file contains flag files set by prior
operations, as follows:
● Pre-recovery events that might have occurred
● Encapsulation procedures – Encapsulation requires a reboot and
creates flag files to delineate actions that need to be taken. If
encapsulation fails or is incomplete, the flag files must be removed
manually.

Note – The root_done flag tells the VxVM software that the boot disk is
under VxVM software control and that this startup script can exit without
any action.

Multi-User Startup Files


This section describes the system startup files executed during multi-user
boot processing.

The S20syssetup Script

The S20syssetup script is not a VxVM software script. This script


discovers and processes any crash dumps. When the script processes
finish, it moves the dumps to /var/crash.

The S94vxnm-host_infod Daemon

The S94vxnm-host_infod daemon starts the host_infod daemon

This script requires a VVR license. The license is located in the


/etc/vx/elm file.

Note – The host_infod daemon is new. It is used by VVR to support


remote operations between nodes without the use of rshell protocols.

Sun Proprietary: Internal Use Only


Recovering Boot and System Processes 5-17
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining the VxVM Software Boot Process

The S94vxnm-vxnetd Daemon

The S94vxnm-vxnetd daemon starts vxnetd, but only if the


/etc/vx/reconfig.d/state.d/install-db flag is not present.

This daemon is required for replication on the secondary replication


volume group (RVG). Also, a valid VVR license must be present to enable
the S94vxnm-vxnetd daemon

The S95vxvm-recover Daemon

The S95vxvm-recover daemon performs the following:


● Volume recovery and re-synchronization is started on all volumes.
● Relocation daemons are started on all volumes.
If a disk failure is detected, subdisks are relocated to another disk.
During relocation operations, disks marked as spare are used to
relocate data.

The S96-vmsa-server Daemon

The S96-vmsa-server daemon starts the Volume Manager Storage


Administrator (VMSA) software.

Other Boot Process Failures


If the system does not boot and all boot-related VxVM software recovery
options fail, it may be necessary to reinstall the VxVM software. This
procedure can be performed without the loss of system or application
data.

To recover the VxVM software by reinstallation


1. Install the VxVM software.
2. Recover the VxVM software configuration.

Sun Proprietary: Internal Use Only


5-18 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Examining the VxVM Software Boot Process

3. Clean up the system configuration, following these steps:


a. Clean up rootability.
b. Clean up volumes.
c. Clean up disk configuration.
d. Reconfigure rootability.
e. Perform the final volume reconfiguration.
4. Start the hot-relocation process.

Note – A detailed and extensive reinstallation recovery procedure is


found in the VERITAS Volume Manager™ 3.2 Troubleshooting Guide under
the heading “Recovery from Boot Disk Failure.”

Sun Proprietary: Internal Use Only


Recovering Boot and System Processes 5-19
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Troubleshooting Boot Process Failures

Troubleshooting Boot Process Failures


During system boot processing, the VxVM software daemons and devices
must be started or the VxVM software is unavailable to the system.
Successful startup of the VxVM software requires the following:
● Bootable boot disk
● Valid /etc/system file
● Valid /etc/vfstab file
● Valid rootdg disk group
● Startable volumes, including no stale plexes
● Non-corrupted driver and daemon binaries
● Appropriate VxVM software binaries loaded on the system for the
specific release of the Solaris OE installed
● Access to library files needed by the VxVM software for initialization
of devices and system volumes
● Valid /etc/vx/volboot file
● Existing /var/vxvm/tempdb directory and supporting tempdb files:
This includes the temporary storage area for the VxVM software
configuration copies. The vxconfigd process needs these files to
transition to the enabled state.
● Execution of the appropriate VxVM software startup scripts

Bootable Boot Disk


A bootable boot disk must be selected from the OpenBoot PROM as the
system boot device. This boot disk should have an appropriate device
alias configured by the system administrator or the full path inserted in
the boot-device NVRAM environment variable by the Solaris OE
installation program. If the boot disk is under VxVM software control and
under the VxVM software management, the following device aliases are
built by the VxVM software:
vx-rootmirror – /sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w220000203713f579,0:a
vx-rootdisk – /sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w220000203713f643,0:a

Sun Proprietary: Internal Use Only


5-20 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Troubleshooting Boot Process Failures

Note – The boot disk device address depends on the system and on which
storage device is configured as the boot disk.

If the boot disk is under VxVM software control, check that the
boot-device is set to the proper VxVM software-generated device alias,
as follows:
boot-file: data not available
boot-device=vx-rootdisk
local-mac-address?=false

Power user – If the primary boot device fails and the system must be
rebooted, use the vx-rootmirror device alias to boot the system.

If the primary boot disk fails and a spare was configured in rootdg, after
the spare disk replaces the failed boot disk and the failed volumes are
being hot-relocated, the VxVM software builds an additional device alias
to enable booting from the spare disk, as follows:
vx-rdgspare01 /sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w220000203713f96d,0:a

Use this device alias if the system must be booted from the spare disk.

Note – The spare disk device address depends on the system and on
which storage device is configured as the spare disk. Also, the spare disk
name is arbitrary and is reflected in the device alias name.

Valid /etc/system File


The /etc/system file must exist and have the following line inserted at
the end of the file:
* vxvm_START (do not remove)
forceload: drv/vxdmp
forceload: drv/vxio
forceload: drv/vxspec
forceload: drv/ssd
forceload: drv/sf
forceload: drv/SUNW,socal
forceload: drv/sbus
forceload: drv/sd
rootdev:/pseudo/vxio@0:0
set vxio:vol_rootdev_is_volume=1
* vxvm_END (do not remove)

Sun Proprietary: Internal Use Only


Recovering Boot and System Processes 5-21
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Troubleshooting Boot Process Failures

The last two lines of the /etc/system file are only present if the boot disk
is under VxVM software control. If these lines are not present and the
boot disk is under VxVM software control, the system will not boot.

Also, the forceload statements for drv/sd and drv/ssd must be present
in the /etc/system file if the boot device is one of these class of disks. If
these forceload statements are not present or corrupted, the system
rebootsrecursively.

Power user – Recovery of a corrupted /etc/system file requires booting


with the -a option and selecting a backup /etc/system file that has the
previously mentioned entries. If a backup /etc/system file is unavailable
or does not contain these statements, perform the following:
1. Boot from CD-ROM. Type the following:
cdrom -s
2. Mount the boot disk to /a.
3. Set the terminal type to vt100.
4. Use a text editor such as vi to edit /etc/system and fix the
problem, or copy the default from CD-ROM and modify it to match
the system configuration.
5. Save the new /etc/system file.
6. Unmount /a.
7. Reboot the system.

Note – This process is presented in greater detail in the VERITAS Volume


Manager™ Troubleshooting Guide, in the section “Recovery from Boot Disk
Failure.”

Valid /etc/vfstab File


The /etc/vfstab file must have valid entries for all system file systems
(/, swap, /usr, and /var), or the system drops into the single-user mode
and does not proceed further. If the boot disk is encapsulated, the
following entries must be present:
/dev/vx/dsk/swapvol - - swap - no -
/dev/vx/dsk/rootvol /dev/vx/rdsk/rootvol / ufs 1 no logging

Sun Proprietary: Internal Use Only


5-22 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Troubleshooting Boot Process Failures

/dev/vx/dsk/usr /dev/vx/rdsk/usr /usr ufs 1 no logging


/dev/vx/dsk/var /dev/vx/rdsk/var /var ufs 1 no logging

Recovery of this file requires editing the file to correct problems and, in
severe cases, can require booting from the CD-ROM.

Note – Additional information on how to recover from problems in the


/etc/vfstab file are found in the VERITAS Volume Manager™
Troubleshooting Guide in the section “Recovery from Boot Disk Failure.”

Valid rootdg Disk Group


The VxVM software requires that a valid rootdg exist, or it does not
initialize. If the boot disk is not encapsulated and rootdg does not exist,
the recovery process creates a new rootdg and reboots or restarts the
VxVM software. If vxinstall is run twice and two rootdg disk groups
exist, then remove the new rootdg and import the original rootdg. When
the original rootdg is the only one on the system, reboot or restart the
VxVM software manually.

If the boot disk is encapsulated, the procedure is more complex. Consult


the SunSolve SRDB 24657, which addresses the problem, to properly
correct failures with importing rootdg.

Startable Volumes
If the boot disk is under the VxVM software control, all system volumes
(/, swap, /usr, and /var) must start. One reason a volume might not start
is the presence of a stale or unusable plex.

A stale plex is defined as a plex that has data which is inconsistent with
other mirrors of that volume. During the boot process, only the plexes on
the boot disk are accessed until the VxVM software is fully initialized and
a complete configuration for the volumes on the boot disk can be
obtained. If the data on the boot disk is stale, then the system must be
rebooted from an alternate boot disk which does not contain stale plexes.

Sun Proprietary: Internal Use Only


Recovering Boot and System Processes 5-23
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Troubleshooting Boot Process Failures

The vxconfigd command displays a bootable disk that is suitable for


booting and which does not contain stale or unusable plexes. If stale
plexes are present, vxconfigd displays the following message:
vxvm:vxconfigd: Warning Ples rootvol-01 for root volume is stale or unusable.
vxvm:vxconfigd: Error: System boot disk does not have a valid root plex
Please boot from one of the following disks:
Disk rootmirror Device: c1t21d0s2
vxvm:vxconfigd: Error: System startup failed
The system is down

Note – Additional information on how to recover from stale or unusable


plexes is found in the VERITAS Volume Manager™ Troubleshooting Guide
under the heading “Recovery from Boot Disk Failure.”

If the boot-device nvram parameter is set, and aliases are set for both the
rootdisk and rootmirror, the system reboots use the rootmirror disk.

Non-Corrupted and Appropriate Binaries and Libraries


The VxVM software initialization requires that the appropriate daemon
and driver binaries and system libraries are available during boot
processing.

Binary Files

Each version of the Solaris OE requires different VxVM software daemon


and driver binaries. These binaries are found in /kernel/drv and
/kernel/drv/sparcv9 (for the 64-bit Solaris OE). The binaries are:
● 32-Bit Solaris OE:
bash-2.03# ls -las /kernel/drv/vx*
640 -rw-r--r-- 1 root sys 314156 Nov 21 23:03 ./vxdmp
608 -rw-r--r-- 1 root sys 297920 Aug 15 2001 ./vxdmp.SunOS_5.6
608 -rw-r--r-- 1 root sys 300980 Aug 15 2001 ./vxdmp.SunOS_5.7
640 -rw-r--r-- 1 root sys 314156 Nov 21 23:03 ./vxdmp.SunOS_5.8
4 -rw-r--r-- 1 root sys 1026 Aug 15 2001 ./vxdmp.conf
3776 -rw-r--r-- 1 root sys 1920316 Nov 21 23:03 ./vxio
3664 -rw-r--r-- 1 root sys 1860500 Aug 15 2001 ./vxio.SunOS_5.6
3696 -rw-r--r-- 1 root sys 1878432 Aug 15 2001 ./vxio.SunOS_5.7
3776 -rw-r--r-- 1 root sys 1920316 Nov 21 23:03 ./vxio.SunOS_5.8
2 -rw-r--r-- 1 root sys 991 Aug 15 2001 ./vxio.conf
30 -rw-r--r-- 1 root other 14968 May 5 08:32 ./vxspec
28 -rw-r--r-- 1 root sys 14060 Aug 15 2001 ./vxspec.SunOS_5.6
30 -rw-r--r-- 1 root sys 14388 Aug 15 2001 ./vxspec.SunOS_5.7

Sun Proprietary: Internal Use Only


5-24 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Troubleshooting Boot Process Failures

30 -rw-r--r-- 1 root sys 14968 Aug 15 2001 ./vxspec.SunOS_5.8


4 -rw-r--r-- 1 root sys 1315 Aug 15 2001 ./vxspec.conf
● 64-Bit Solaris OE:
bash-2.03# ls -las /kernel/drv/sparcv9/vx*
800 -rw-r--r-- 1 root sys 393968 Nov 21 23:03 ./vxdmp
768 -rw-r--r-- 1 root sys 380840 Aug 15 2001 ./vxdmp.SunOS_5.7
800 -rw-r--r-- 1 root sys 393968 Nov 21 23:03 ./vxdmp.SunOS_5.8
5328 -rw-r--r-- 1 root sys 2714688 Nov 21 23:03 ./vxio
5232 -rw-r--r-- 1 root sys 2666256 Aug 15 2001 ./vxio.SunOS_5.7
5328 -rw-r--r-- 1 root sys 2714688 Nov 21 23:03 ./vxio.SunOS_5.8
38 -rw-r--r-- 1 root other 18504 May 5 08:32 ./vxspec
36 -rw-r--r-- 1 root sys 17928 Aug 15 2001 ./vxspec.SunOS_5.7
38 -rw-r--r-- 1 root sys 18504 Aug 15 2001 ./vxspec.SunOS_5.8

Power user – If a system message lists an incorrect version of the VxVM


software installed for the Solaris OE, the most probable cause is that the
incorrect version of the Solaris OE was selected during the VRTSvxvm
package addition. The solution to this is to copy the correct binary file
(vxio.xxxxx, vxdmp.xxxx, or vxspec.xxx) to either the vxio, vxspec or
xvdmp files. Additionally, if these files become corrupt, copy over these
files using the correct binary file for the Solaris OE installed.

If the boot disk is under the VxVM software control and a driver file is
corrupt, the system boot fails. The system administrator must perform a
basic or functional unencapsulation (refer to ‘‘Performing a Basic or
Functional Unencapsulation’’ on page 2-68 for details).

Library Files

The following library files must be in the /etc/vx/slib directory:


bash-2.03# ls -las /etc/vx/slib
total 5346
2 drwxr-xr-x 2 root other 512 May 11 13:18 .
2 drwxr-xr-x 9 root other 512 May 20 07:20 ..
186 -rwxr-xr-x 1 root other 95052 May 5 08:33 liba5k.so.2
10 -rwxr-xr-x 1 root other 4392 May 5 08:33 liba5k_stub.so.2
2256 -rwxr-xr-x 1 root other 1146284 May 5 08:32 libc.so.1
10 -rwxr-xr-x 1 root other 4848 May 5 08:34 libc_psr.so.1
46 -rwxr-xr-x 1 root other 23348 May 5 08:33 libdevice.so.1
26 -rwxr-xr-x 1 root other 12432 May 5 09:56 libdevid.so
26 -rwxr-xr-x 1 root other 12432 May 5 09:56 libdevid.so.1
172 -rwxr-xr-x 1 root other 87108 May 5 08:33 libdevinfo.so.1
256 -rwxr-xr-x 1 root other 119244 May 5 08:33 libg_fc.so.2
12 -rwxr-xr-x 1 root other 5456 May 5 08:33 libg_fc_stub.so.2
50 -rwxr-xr-x 1 root other 24968 May 5 08:33 libmp.so
1776 -rwxr-xr-x 1 root other 898600 May 5 08:33 libnsl.so.1
48 -rwxr-xr-x 1 root other 24360 May 5 08:33 libnvpair.so.1
140 -rwxr-xr-x 1 root other 70864 May 5 08:32 libsocket.so.1

Sun Proprietary: Internal Use Only


Recovering Boot and System Processes 5-25
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Troubleshooting Boot Process Failures

288 -rwxr-xr-x 1 root other 139224 May 5 09:55 libthread.so.1


40 -rwxrwxr-x 1 root sys 20096 Nov 21 17:18 libvxdiscovery.so

If any of these library files are not accessible and the boot disk is
encapsulated, a message similar to the following is displayed:
Starting VxVM restore daemon...
VxVM starting in boot mode...
ld.so.1: vxconfigd: fatal: <missing library file name is displayed>: open failed: No
such file or directory
Killed

Errors were encountered in starting the root disk group, as a result


VxVM is unable to configure the root and/or /usr volumes. If you have
mirrored the root disk, you can try booting from that disk. Please
refer to Appendix C of the Installation Guide for more details.

If you cannot boot from the root disk, you can try to repair the problem
using a network-mounted root file system or some other alternate root
file system. Again, see the Installation Guide for more details.

Would you like a shell prompt right now? [no]

If this problem exists, perform a basic or functional unencapsulation and


copy the missing libraries from /usr/lib to /etc/vx/slib. Refer to
‘‘Performing a Basic or Functional Unencapsulation’’ on page 2-68 for
details.

Sun Proprietary: Internal Use Only


5-26 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Troubleshooting Boot Process Failures

Valid /etc/vx/volboot File


The /etc/vx/volboot file is the VxVM software bootstrap file. This file
must be valid or the VxVM software does not initialize. The contents of
this file includes the following:
● The /etc/vx/volboot file is an ASCII file that adheres to a strict
format and should not be edited. It has the following characteristics:
● 512 bytes in length, including padding
● Updated using the vxdctl command
● The /etc/vx/volboot file holds the VxVM software host identifier
hostid. This is usually the Solaris OE nodename, not the hardware
hostid. The VxVM software hostid does not have to match the
server’s nodename, which can be very confusing.
This file has the following characteristics –
● It establishes disk and diskgroup ownership.
● If two or more servers access the same disks using the same
bus, the VxVM software hostid ensures that the two hosts do
not interfere with each other when they are accessing VxVM
software disks.

If this file is corrupted or deleted, recover the file from backups of the
affected system. If backups are not available, a reinstall may be necessary.

VxVM Software Startup Scripts


There are five startup scripts located in /etc/rcX.d which are executed
during system boot processing:
● /etc/rcS.d/S25vxvm-sysboot – Checks system and vfstab files,
including install-db, to determine whether the VxVM software
should start. (For VxVM software version 2.3, this script also checks
for the /VXVM2.3 UPGRADE/.start_runed file.) If the software
should start, the script prints the message VxVM starting in boot
mode... and executes a vxconfigd -m boot command.
● /etc/rcS.d/S35vxvm-startup1 – If needed, special volumes such as
/ and swap are started.

Sun Proprietary: Internal Use Only


Recovering Boot and System Processes 5-27
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Troubleshooting Boot Process Failures

● /etc/rcS.d/S85vxvm-startup2 – Prints the message VxVM


general startup... , and starts the vxiod daemons. This script
also enables vxconfigd or starts it if it is not already started. This
script is where the /dev/vx directory is made or updated, if needed,
and a vxrecover command starts all volumes.
● /etc/rcS.d/S86vxvm-reconfig – Checks for the existence of the
/etc/vx/reconfig.d/state.d/reconfig file. If there are
reconfigurations needed (for example, for encapsulated disks), the
work is done here. The system can be rebooted from this script if
needed.
● /etc/rc2.d/S95vxvm-recover – Runs vxrecover, which fixes any
remaining problems or starts any unstarted volumes. Also starts
vxrelocd, which starts another vxrelocd and a vxnotify deamon.
● /etc/rc2.d/S96vmsa-server – [VxVM software version 3.x only]
Starts the vmsa GUI server, which is needed for the vmsa GUI to
work.

Power user – The recovery from errors resulting from the execution of
these scripts requires reading each individual script and determining
where the failure is within the script. This requires shell programming
expertise and a detailed understanding of the VxVM software commands
and support files. All of the VxVM software startup scripts are hard
linked from the /etc/init.d file.

Sun Proprietary: Internal Use Only


5-28 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Troubleshooting VxVM Software Failures

Troubleshooting VxVM Software Failures


Failures of the VxVM software other than boot-process failures include,
but are not limited to, the following:
● Corrupt or missing commands and utilities – The VxVM software
commands and utilities are found in the following directories:
● /etc/vx/bin
● /usr/sbin
● /sbin
● Corrupt or missing configuration files – Most VxVM software
configuration files are in the /etc/vx directory. Driver configuration
files are in the /kernel/drv directory.

To correct these problems, restore a valid backup of the missing or


corrupted information, or copy non-system-specific files from another
system on the network.

Caution – Be careful when copying files from other systems on the


network. If a system-specific file, such as /etc/vx/volboot, is copied
from another server, the system to which it is copied can be placed into a
non-bootable state. Use backups from the system missing the needed file
whenever possible.

Sun Proprietary: Internal Use Only


Recovering Boot and System Processes 5-29
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Determining the VxVM Software Problem

Exercise: Determining the VxVM Software Problem


In this exercise, you will troubleshoot the VxVM software functional and
initialization problems injected into your lab system, using a supplied
break-and-fix script.

Preparation
To prepare for this exercise:
● The VxVM software must be installed and operational.
● The boot disk must be encapsulated and mirrored.
● There must be one additional disk group other than rootdg with at
least one configured, started and mounted volume.
● The instructor must give you the location and name of the break-
and-fix script.

The break-and-fix script is a menu-driven script used to break a system


based on bugs submitted by Sun Support. This script performs the
following tasks:
● Provides a list of bugs for use by students to inject real world bugs
into the lab systems.
● Keeps track of bugs completed and those skipped or not completed
by lab teams.
● Provides pointers to SunSolve SRDB records and INFODOCs.
● Provides hints to help lab teams troubleshoot bugs.
● Fixes the injected bug if the lab team desires to proceed to another
bug without resolving the current bug.

To run the break-and-fix script:


1. Enter the following:
# breakfix
The following is displayed:
breakfix Main Menu

1. Problem 1 11. Problem 11


2. Problem 2 12. Problem 12
3. Problem 3 13. Problem 13
4. Problem 4 14. Problem 14

Sun Proprietary: Internal Use Only


5-30 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Determining the VxVM Software Problem

5. Problem 5 15. Problem 15


6. Problem 6 16. Problem 16
7. Problem 7 17. Problem 17
8. Problem 8 18. Problem 18
9. Problem 9 19. Problem 19
10. Problem 10 20. Problem 20

x. Exit

Enter option [x]:


2. Select a bug by entering the bug number from the menu.
For example, if you select Bug 1, the following is displayed:
Problem #1, Invalid disk.exclude file

b) Break it
s) Solution
h) Hint
m) Return to Main Menu
x) Exit

Enter option [x]:


3. Enter b to break, f to fix and m to return to the main menu.
If you select the b or break option, a system reboot is necessary to
activate the bug. After the symptom is observed by your lab team,
access SunSolve Online, and search for the proper SRDB to help
diagnose the problem.
If you need or want a hint, re-execute the break-and-fix script and
select the h option for the bug currently being worked on.
The following is displayed:
This is SRDB ID: 14736.

The /etc/vx/disks.exclude file requires the following format: c#t#d#


for each disk you wish to exclude in /etc/vx/disks.exclude.

Press ’enter’ to return to the menu.


Notice that the referring SRDB is listed along with hints to help
resolve the problem.

Sun Proprietary: Internal Use Only


Recovering Boot and System Processes 5-31
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Determining the VxVM Software Problem

Tasks
Execute the break-and-fix script, and select bugs 1 through 10. List
problem resolution steps for each bug in the space listed below.

Problem 1:

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

Problem 2:

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

Problem 3:

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

Sun Proprietary: Internal Use Only


5-32 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Determining the VxVM Software Problem

Problem 4:

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

Problem 5:

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

Problem 6:

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

Problem 7:

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

Sun Proprietary: Internal Use Only


Recovering Boot and System Processes 5-33
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Determining the VxVM Software Problem

__________________________________________________________________

__________________________________________________________________

Problem 8:

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

Problem 9:

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

Problem 10:

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

The lab is complete when all 10 bugs are fixed, your lab system with the
VxVM software is operational, and all volumes started and accessible.

Sun Proprietary: Internal Use Only


5-34 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise Summary

Exercise Summary

Discussion – Take a few minutes to discuss what experiences, issues, or


discoveries you had during the lab exercise.
!
?
● Experiences
● Interpretations
● Conclusions
● Applications

Sun Proprietary: Internal Use Only


Recovering Boot and System Processes 5-35
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Determining the VxVM Software Problem

Exercise: Determining the VxVM Software Problem


Following is information about the solutions to this lab exercise.

Task Solutions
The solutions for the tasks in the lab exercise are found in the break-and-
fix script and on SunSolve in the appropriate SRDB and INFODOC files.

Sun Proprietary: Internal Use Only


5-36 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Module 6

Recovering Disk, Disk Group, and


Volume Failures

Objectives
Upon completion of this module, you should be able to:
● Describe the VxVM software utilities, commands, and virtual devices
to help with recovery processes
● Identify and repair failed disks
● Perform disk recovery processes and procedures
● Identify and repair volume errors
● Identify and repair disk group failures

Sun Proprietary: Internal Use Only


6-1
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Relevance

Relevance

Discussion – The following questions are relevant to understanding the


VxVM software disk recovery processes:
!
?
● What tools, utilities, commands, or files are available to help identify
failed VxVM software disks and other objects used to access
managed storage devices?
● What are the indications that a disk failed?
● How are damaged disk groups repaired?
● How are new disks made visible to the VxVM software?
● What are the procedures to replace a failed VM disk?
● How are duplicate record entries recognized and repaired?

Sun Proprietary: Internal Use Only


6-2 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Additional Resources

Additional Resources

Additional resources – The following references provide additional


information on the topics described in this module:
● VERITAS Volume Manager™ 3.2 Administrator’s Guide. Mountain
View, California: VERITAS Software Corporation, August 2001,
number 30-000392-011, TechPDF ID 240253.
● VERITAS Volume Manager™ 3.2 Installation Guide. Mountain View,
California: VERITAS Software Corporation, August 2001,
number 30-000395-011, TechPDF ID 240256.
● VERITAS Volume Manager™ 3.2 Troubleshooting Guide. Mountain
View, California: VERITAS Software Corporation, August 2001,
number 30-000394-011, TechPDF ID 240255.
● VERITAS Volume Manager™ Storage Administrator 3.2 Administrator’s
Guide. Mountain View, California: VERITAS Software Corporation,
July 2001, number 30-000393-011, TechPDF ID 240257.
● http://storage.east, http://storage.central, and
http://storage.west.
● SunSolveSM Online SRDBs and INFODOCs 13364, 14820, 21626 and
26367,
[http://sunsolve.Sun.COM/pub-cgi/search.pl?mode=advanced].
● Man pages for the following commands:
● vxmend
● vxvol
● vxplex
● vxreattach
● vxstat
● vxrecover

Sun Proprietary: Internal Use Only


Recovering Disk, Disk Group, and Volume Failures 6-3
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Introducing Recovery Processes

Introducing Recovery Processes


The VxVM software provides utilities, commands, and virtual devices to
improve the fault resiliency of storage subsystems brought under VxVM
software management. When a disk experiences a failure, the VxVM
software runs processes that ensure continued availability of the data
hosted on the failed disk. This is only true if the data on that disk was
made fault resilient using RAID 1 or RAID 5 data resiliency protocols.

Recovery of failed disks requires the following:


● The disk must be a VxVM software disk.
● The disk group of which the disk is a member must be imported.
● The disk must have subdisks that belong to an active volume that is
in use by the system.

Depending on the architecture of the volume, the loss of a single disk


should not effect the availability of the volume.

Disk groups’ configuration problems can prevent the import or creation of


the disk group. The result of this problem is that the resources managed
by the affected disk group are not available to the system. Disk group
import problems are generally due to ownership issues, such as the
appearance that a disk group is imported when it is not.

This module describes how to identify and recover from disk, disk group,
and volume failures using the VxVM software.

Sun Proprietary: Internal Use Only


6-4 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Identifying Disk Errors

Identifying Disk Errors


When disk failures occur, the system and VxVM software display error
information which describes the failure. If properly configured, error
information is logged in the following locations:
● /etc/vx/vxconfigd.log
● /var/adm/messages
● root mail

Use the following commands and utilities to display the operational state
of managed disks:
● vxprint
● vxdisk
● vxdiskadm
● vxstat
● The vmsa GUI administrative console

Note – The vmsa GUI administrative console is not discussed in this


module because it is unavailable when you are diagnosing problems
remotely. Configuration of vxconfigd logging, interpreting error
messages and using root mail is discussed in Module 4, “Troubleshooting
Tools and Utilities.”

This section discusses these commands and their use in recovering from
full or partial disk failures.

The vxprint Command


The vxprint command displays the status of the VxVM software objects.
It is a useful tool to determine the operational status of those objects.

This example from a vxprint command was collected after the


rootmirror disk failed.
bash-2.03# vxprint -htg rootdg
DG NAME NCONFIG NLOG MINORS GROUP-ID
DM NAME DEVICE TYPE PRIVLE
UBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL

Sun Proprietary: Internal Use Only


Recovering Disk, Disk Group, and Volume Failures 6-5
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Identifying Disk Errors

RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK


V NAME RVG KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO

dg rootdg default default 0 1022020746.1025.lowtide

dm rootdisk c1t0d0s2 sliced 3590 17678493 -


dm rootmirror - - - - NODEVICE

v rootvol - ENABLED ACTIVE 15581349 ROUND - root


pl rootvol-01 rootvol ENABLED ACTIVE 15581349 CONCAT - RW
sd rootdisk-02 rootvol-01 rootdisk 7181 15581349 0 c1t0d0 ENA
pl rootvol-02 rootvol DISABLED NODEVICE 15581349 CONCAT - RW
sd rootmirror-01 rootvol-02 rootmirror 0 15581349 0 - RLOC

v swapvol - ENABLED ACTIVE 1052163 ROUND - swap


pl swapvol-01 swapvol ENABLED ACTIVE 1052163 CONCAT - RW
sd rootdisk-01 swapvol-01 rootdisk 15588530 1052163 0 c1t0d0 ENA
pl swapvol-02 swapvol DISABLED NODEVICE 1052163 CONCAT - RW
sd rootmirror-02 swapvol-02 rootmirror 15581349 1052163 0 - NDEV

Note in this example that the failed plex for volume rootvol is
rootvol-0. This volume has a failed subdisk named rootmirror-01. The
volume is unaffected because it is enabled and active.

The vxdisk Command


Use the vxdisk command to display the status of the VxVM software
disks. An example of this command shows the rootmirror from ‘‘The
vxprint Command’’ on page 6-5 in a failed state.
bash-2.03# vxdisk list
DEVICE TYPE DISK GROUP STATUS
c1t0d0s2 sliced rootdisk rootdg online
c1t3d0s2 sliced - - error
c1t4d0s2 sliced - - online
c1t5d0s2 sliced - - online
c1t6d0s2 sliced - - error
c1t16d0s2 sliced - - error
c1t18d0s2 sliced - - error
c1t19d0s2 sliced - - error
c1t22d0s2 sliced - - online
- - rootmirror rootdg failed was:c1t22d0s2

Sun Proprietary: Internal Use Only


6-6 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Identifying Disk Errors

The STATUS column displays the status of disks visible to the VxVM
software. A status of error does not always indicate a disk error; it
usually indicates that the disk is not under the VxVM software control.

The vxdiskadm list Option


The list option from the vxdiskadm utility is another source of disk
operational status information. The following example shows the list
option output for the rootmirror disk.
List disk information
Menu: VolumeManager/Disk/ListDisk

Use this menu operation to display a list of disks. You can


also choose to list detailed information about the disk at
a specific disk device address.

Enter disk device or "all" [<address>,all,q,?] (default: all) c1t22d0

Device: c1t22d0s2
devicetag: c1t22d0
type: sliced
flags: online error private autoconfig
errno: Device path not valid
Multipathing information:
numpaths: 2
c1t22d0s2 state=disabled
c2t22d0s2 state=disabled

The vxstat Command


Use the vxstat command to determine the underlying subdisk of a failed
plex. This is useful when a partial disk failure occurs that only affects a
sub-set of the subdisks on a failed or failing disk. An instance of the
command when used for this purpose is shown in the following example.
The example assumes that there are two failing plexes, opt-01 and var-
01. The -s option displays information about individual subdisks, and the
-ff option displays failed read and write operations.
bash-2.03# vxstat -s ff opt-01 var-01
FAILED
TYP NAME READS WRITES
sd rootmirror-05 0 0
sd rootmirror-06 0 0
sd rootdisk-04 1 0
sd rootdisk-05 1 0

Sun Proprietary: Internal Use Only


Recovering Disk, Disk Group, and Volume Failures 6-7
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Identifying Disk Errors

This clearly shows that the rootdisk disk experienced a read failure
when accessing subdisks rootdisk-04 and -05.

Disk Failure Categories and root Mail


If hot-relocation is enabled, mail is sent to root to notify administrators of
either a disk or plex failure. There are two types of disk errors:
● Full disk failure
● Partial disk failure

Full Disk Failure


Full disk failures are caused by a total loss of a disk, often due to a cabling
problem or a catastrophic failure of the disk’s physical or logical
components. Mail is sent to root describing the failed disk and the
affected plexes.

This example of a root mail message is typical of a complete disk failure.


To: root
Subject: Volume Manager failure on lowtide

Failures have been detected by the VERITAS Volume Manager:

failed disks:
rootdisk

failed plexes:
opt-01
var-01

failing disks:
rootdisk

This root mail message shows that disk03 failed. Plexes home-02,
fin-02, and hr-01 also failed; if a spare disk with sufficient space is
available for hot-relocation, these disk are relocated.

Note – For a problem of this type, check cabling or disk seating, or replace
the disk.

Sun Proprietary: Internal Use Only


6-8 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Identifying Disk Errors

Partial Disk Failure

Partial disk failures are those errors that cause a region of a disk to fail,
affecting some but not all of the subdisks stored on the failed disk. Errors
of this type are usually caused by media errors, but do not ignore the
possibility of a cabling problem. Mail is sent to root describing the failed
plexes but not the failed disks or subdisks.

This example root mail message is typical of a partial disk failure.


To: root
Subject: Volume Manager failures on host lowtide
Failures have been detected by the VERITAS Volume Manager

failed plexes:
opt-01
var-01

Use the vxstat command to determine the failed disk. This procedure is
described in the ‘‘The vxstat Command’’ on page 6-7.

Sun Proprietary: Internal Use Only


Recovering Disk, Disk Group, and Volume Failures 6-9
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Replacing Disks

Replacing Disks
If a disk fails, it must be replaced. Use one of the following two methods
to replace a failed disk:
● The vxdiskadm utility
● The command line

Replacing Disks Using the vxdiskadm Utility


To replace a disk in the VxVM software, use the menu-driven vxdiskadm
utility option 5, Replace a failed or removed disk. This procedure
is described in SunSolveSM INFODOCs 14820 and 21626. Refer to these
documents if replacing a disk due to failure.

The vxdiskadm utility uses the vxreattach command to perform the


reattachment task. Prior to running the vxdiskadm utility, issue a
vxreattach -c command to show if reattachment is possible and to
determine whether to attempt the task. For more information on the
vxreattach command, consult the man page.

Replacing Disks Using the Command Line


Discrete command line utilities, such as vxdg and vxrecover, are also
available for use in replacing disks. Use the following commands to
replace a failed disk at the command line:
# vxdg -g <disk_group> rmdisk <failed_disk_VxVM_name>
# vxdg -g <disk_group> adddisk <failed_disk_VxVM_name>=cxtxdxs2
# vxrecover -bs.

Caution – The vxrecover -bs command attempts to perform recovery


operations on all volumes with failed plexes, including starting any non-
starting volumes, prior to performing the plex recovery operations. This
might not be the correct operation to perform, depending on the current
volume configuration of the system being recovered. See the vxrecover
man page for more information.

Sun Proprietary: Internal Use Only


6-10 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Replacing Disks

Recovering a Disk Which VxVM Software Cannot See


When you are replacing a disk due to a failure, the disk must be visible to
the VxVM software or the replacement procedure fails. Figure 6-1 is a
flow diagram for troubleshooting disk visibility problems.

Start

See disk in The VxVM Software


Yes
vxdisk list? can see the disk.

No

First time The VxVM Software cannot


No
through flow? see the disk. Call support.

Yes

Can the Solaris OE


Run devfsadm or
see the disk? No
drvconfig; disks
(format, prtvtoc)

Yes

Run vxdctl initdmp Can the Solaris OE


Yes see the disk?
and vxdctl enable
(format, prtvtoc)

No

Hardware or
Solaris OE Problem

Figure 6-1 Recovery Process When VxVM Software Cannot See a Disk

Sun Proprietary: Internal Use Only


Recovering Disk, Disk Group, and Volume Failures 6-11
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Replacing Disks

Perform the following steps to troubleshoot a disk visibility problem:


1. Does the VxVM software see the disk? If not, check the Solaris OE. If
the Solaris OE cannot see a disk device, then the VxVM software
cannot see it. Use the format or prtvtoc command to verify that the
Solaris OE is able to detect the disk.
2. If the Solaris OE cannot see the disk, run the devfsadm or
drvconfig;disks command to make the Solaris OE re-scan the
devices. If the Solaris OE can see the disk, then the visibility problem
is with the VxVM software.
3. If the Solaris OE still cannot see the disk, check again to see if the
Solaris OE can see the disk.
If the Solaris OE can see the disk, proceed to the next step in the flow
diagram and execute the necessary commands that allow the VxVM
software to see the disk.
If the Solaris OE still cannot see the disk, there is a hardware,
software or driver problem that must be corrected before this disk
can be used by the VxVM software.
4. It might be necessary to recreate the DMP user-level nodes for all
nodes in the system. In addition, it might be necessary to build (or
rebuild) the VxVM software user-space devices. To force the VxVM
software to re-scan the Solaris OE device tree and build the necessary
VxVM device nodes, run the vxdctl initdmp or the
vxdctl enable command.

When these procedures are complete the disk device should be available
for use, unless there is a hardware or functional VxVM software problem.

Sun Proprietary: Internal Use Only


6-12 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Troubleshooting Volume Errors

Troubleshooting Volume Errors


Volumes are the high-order VxVM software objects that are mounted by
the Solaris OE. If a volume has a configuration error or other condition
that prevents it from starting, it is in an unstartable state. If the volume
has an error that was caused by a disk failure and the data managed by
the volume is corrupted, the volume is in a disabled state. Volumes that
are in an unstartable or disabled state cannot be mounted and used by the
system until the error is resolved.

This section describes how to identify and work with volume-related


errors.

Listing Unstartable Volumes


To display unstartable volumes, use the following syntax:
# vxinfo -g disk_group_name volume

This example shows the results of the vxinfo -g command.


bash-2.03# vxinfo -g rootdg
usr fsgen Started
var fsgen Started
rootvol root Started
swapvol swap Started

bash-2.03# vxinfo -g rootdg rootvol


rootvol root Started

An unstartable volume is listed as Unstartable in the third column of


the command output.

After the problem with the volume is corrected, use the vxvol command
to restart the volume.

Sun Proprietary: Internal Use Only


Recovering Disk, Disk Group, and Volume Failures 6-13
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Troubleshooting Volume Errors

Restarting a Disabled Volume


Disabled volumes are caused by disk failures that breech the fault
resiliency of a mirrored or RAID 5 volume. Restore the backup data for a
disabled volume after the failed disk or disks are repaired. Prior to
restoring the volumes data from tape, restart the volume. Use the
following version of the vxvol command to restart the volume:
# vxvol -o bg -f start volume_name

The -f option forcibly restarts the volume, while the -o bg option


resynchronizes the volume’s plexes in the background.

Recovering a Mirrored Volume


Some errors, such as a system crash or I/O error, can leave a mirrored
volume with multiple plexes in neither a clean or active state. This requires
administrator intervention to tell the VxVM software which plex has the
most current copy of the data, and then to use that plex to resynchronize
the remaining plexes of the volume.

Use the following procedure to recover a mirrored volume:


1. Place the desired plex (using an administrative best guess) in the
clean state.
# vxmend fix clean plex_name
2. Mark all other plexes as stale if they are not already in the stale state.
# vxmend fix stale plex_name
3. Recover the other plexes from the plex just placed in the clean state.
# vxvol start volume_name

Note – For more information on this procedure, refer to the VERITAS


Volume Manager™ Troubleshooting Guide for the version of the VxVM
software installed on the system, and to the man pages for the vxmend and
vxvol commands.

Sun Proprietary: Internal Use Only


6-14 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Troubleshooting Volume Errors

Recovering a RAID 5 Volume


RAID 5 volume failures can be more complex and therefore more difficult
and time consuming to recover than mirrored volumes. This is due to the
parity calculations and a RAID 5 volume’s ability to reconstruct data from
a failed subdisk, while leaving the volume’s data available to the
Solaris OE.

RAID 5 failures are of two varieties:


● System failures – The system stopped operating abruptly due to a
power outage or kernel panic. Errors of this type can leave the
RAID 5 volume with stale parity.
Stale parity must be reconstructed by reading all the data on the
volume and rebuilding the parity blocks. This can take considerable
time and impacts system performance. To mitigate errors of this
type, attach a RAID 5 log.
● Disk failures – Data on some number of disks can become
unavailable due to controller, disk or media failure. Errors of this
type have a subdisk detached and can leave the RAID 5 volume in
degraded mode.
Once the failure that forced the RAID 5 volume into degraded mode
is repaired, the data resident on the failed subdisk is reconstructed
from data on other stripe units in the stripe. This reconstruction of
the missing data, while the data managed by the volume remains
available to the Solaris OE, is called a reconstructing-read. This process
results in degraded performance and, if another subdisk fails during
the reconstruction, the volume experiences total failure and moves
into the disabled state.

Recovery of RAID 5 volumes is categorized into three groups, and is


accomplished as follows:
● Parity resynchronization uses the vxvol command. Type the
following:
# vxvol resync RAID_5_volume_name
● Log plex recovery uses the vxplex command. Type the following:
# vxplex att RAID_5_volume_name Log_Plex_name

Sun Proprietary: Internal Use Only


Recovering Disk, Disk Group, and Volume Failures 6-15
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Troubleshooting Volume Errors

● Stale subdisk recovery uses two versions of the vxvol command.


To recover one stale subdisk, type the following:
# vxvol recover RAID_5_volume_name stale_plex_name
To recover multiple stale subdisks, type the following:
# vxvol recover RAID_5_volume_name

Note – RAID 5 recovery is complex. For more information, reference the


VERITAS Volume Manager™ Troubleshooting Guide for the version of the
VxVM software installed on the system, and to the man pages for vxplex
and vxvol commands. Additional information is found in SunSolve
INFODOCs 13364 and 26367.

Forcibly Starting RAID 5 Volumes


Sometimes it is necessary to forcibly start a RAID 5 volume. This could be
due to maintenance activities which mark a plex as stale even though the
data is good.

Use the vxvol command with the -f option to forcibly start a RAID 5
volume. Use the following syntax:
# vxvol -f start RAID_5_volume _name

Caution – Do not forcibly start a RAID 5 volume that is known to have


stale subdisks with out-of-date or corrupted data. This can cause
corruption of volume data and affect the operation of applications or the
Solaris OE.

Sun Proprietary: Internal Use Only


6-16 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Troubleshooting Disk Group Errors

Troubleshooting Disk Group Errors


Disk groups can move into an inconsistent state and become
unimportable. The usually occurs if a disk group move was in process at
the time of a system crash. After the system is restarted, the VxVM
software attempts to reverse or complete the operation. If it is unable to
do so, administrative intervention is required to recover the disk group.

Use the vxdg command to repair a disk group, as follows:


1. To determine if a disk group is stuck in a move state, use the
vxprint command to print the disk group and look for move status
in the TUTIL0 field for the disk group.

Note – The TUTIL filed temporarily locks objects during configuration


changes and recoveries. This field should be cleared after a system reboot.
The TUTIL and PUTIL fields are discussed in “Changing Disk Group
Configurations” in Appendix E, “Configuring the VxVM Software.”

2. Try to use the vxdg command to complete the move. Type the
following:
# vxdg recover disk_group_name
3. If the previous step does not work, try to reset the move flag as
follows:
# vxdg -o clean recover disk_group_name
4. If the previous step does not work, try to remove the move flag. Type
the following:
# vxdg -o remove recover disk_group_name

Power user – If these commands cannot be run on the system performing


the move and the disk group is still accessible from a second system, run
these commands on the second system. To make sure that any disk group
name conflicts are resolved, use a different name for the disk group being
repaired.

Sun Proprietary: Internal Use Only


Recovering Disk, Disk Group, and Volume Failures 6-17
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Determining the VxVM Software Disk Problem

Exercise: Determining the VxVM Software Disk Problem


In this exercise, you troubleshoot the VxVM software disk problems
injected into your lab system using a supplied break-and-fix script.

Preparation
To prepare for this exercise:
● Make sure that the VxVM software is installed and operational.
● The boot disk must be encapsulated and mirrored.
● There must be one additional disk group other than rootdg with at
least one configured, started, and mounted volume.
● You must have access to another system’s VxVM software-managed
storage devices, either through a storage area network (SAN) or by
reconfiguring shared storage attached to the two systems.
● The instructor must give you the location and name of the break-
and-fix script.

The break-and-fix script is a menu driven script that is used to break a


system based on bugs submitted by Sun Support. This script performs the
following tasks:
● Provides a list of bugs for use by students to inject real world bugs
into lab systems.
● Keeps track of bugs completed and those skipped or not completed
by lab teams.
● Provides pointers to SunSolve SRDB records and INFODOCs.
● Provide hints to help lab teams troubleshoot bugs.
● Fixes the injected bug if the lab team desires to proceed to another
bug without resolving the current bug.

Sun Proprietary: Internal Use Only


6-18 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Determining the VxVM Software Disk Problem

To run the break-and-fix script:


1. Enter the following:
# breakfix
The following is displayed:
breakfix Main Menu

1. Problem 1 11. Problem 11


2. Problem 2 12. Problem 12
3. Problem 3 13. Problem 13
4. Problem 4 14. Problem 14
5. Problem 5 15. Problem 15
6. Problem 6 16. Problem 16
7. Problem 7 17. Problem 17
8. Problem 8 18. Problem 18
9. Problem 9 19. Problem 19
10. Problem 10 20. Problem 20

x. Exit

Enter option [x]:


2. Select a bug by entering the bug number from the menu.
For example, if you select Bug 1, the following is displayed:
Problem #1, Invalid disk.exclude file

b) Break it
f) Solution
h) Hint
m) Return to Main Menu
x) Exit

Enter option [x]:

Sun Proprietary: Internal Use Only


Recovering Disk, Disk Group, and Volume Failures 6-19
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Determining the VxVM Software Disk Problem

3. Enter b to break, f to fix and m to return to the main menu.


If you select the b or break option, a system reboot is necessary to
activate the bug. After the symptom is observed by your lab team,
access SunSolve Online and search for the proper SRDB to help
diagnose the problem.
If you need or want a hint, re-execute the break-and-fix script and
select the h option for the bug currently being worked on.
The following is displayed:
This is SRDB ID: 14736.

The /etc/vx/disks.exclude file requires the following format: c#t#d#


for each disk you wish to exclude in /etc/vx/disks.exclude.

Press ’enter’ to return to the menu.


Notice that the referring SRDB is listed along with hints to help
resolve the problem.

Tasks
Execute the break-and-fix script, and select bugs 11 through 20. List
problem resolution steps for each bug in the space provided.

Problem 11:

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

Problem 12:

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

Sun Proprietary: Internal Use Only


6-20 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Determining the VxVM Software Disk Problem

Problem 13:

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

Problem 14:

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

Problem 15:

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

Problem 16:

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

Sun Proprietary: Internal Use Only


Recovering Disk, Disk Group, and Volume Failures 6-21
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Determining the VxVM Software Disk Problem

Problem 17:

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

Problem 18:

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

Problem 19:

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

Problem 20:

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

The lab is complete when all 10 bugs are fixed and your lab system is
operational, with the VxVM software and all volumes started and
accessible.

Sun Proprietary: Internal Use Only


6-22 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise Summary

Exercise Summary

Discussion – Take a few minutes to discuss what experiences, issues, or


discoveries you had during the lab exercise.
!
?
● Experiences
● Interpretations
● Conclusions
● Applications

Sun Proprietary: Internal Use Only


Recovering Disk, Disk Group, and Volume Failures 6-23
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Determining the VxVM Software Disk Problem

Exercise: Determining the VxVM Software Disk Problem


Following is information about the solutions to this lab exercise.

Task Solutions
The solutions for the tasks in the lab exercise are found in the break-and-
fix script and on SunSolve in the appropriate SRDB and INFODOC files.

Sun Proprietary: Internal Use Only


6-24 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Module 7

Upgrading the VxVM Software

Objectives
Upon completion of this module, you should be able to:
● Describe the processes used to upgrade the VxVM software from
release 3.1 to 3.2 and from 3.2 to 3.5
● Perform an upgrade of the VxVM software to release 3.5
● Read, review, and interpret release notes
● Identify the top three installation problems
● Identify bugs, find patches, and apply patches for the release of the
VxVM software installed
● Identify and resolve licensing issues
● Upgrade the Solaris OE with VxVM is installed on the system

Sun Proprietary: Internal Use Only


7-1
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Relevance

Relevance

Discussion – The following questions are relevant to understanding the


VxVM software upgrade processes:
!
?
● What is the upgrade process for the VxVM software when the boot
disk is encapsulated?
● What is the upgrade process for the VxVM software when the boot
disk is not encapsulated?
● What is the resolution process for licensing problems when
upgrading the VxVM software?
● How is the Solaris OE upgraded when the VxVM software is
installed?
● Do release notes provide useful information when upgrading the
VxVM software?

Sun Proprietary: Internal Use Only


7-2 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Additional Resources

Additional Resources

Additional resources – The following references provide additional


information on the topics described in this module:
● VERITAS Volume Manager™ 3.2 Administrator’s Guide. Mountain
View, California: VERITAS Software Corporation, August 2001,
number 30-000392-011, TechPDF ID 240253.
● VERITAS Volume Manager™ 3.2 Installation Guide. Mountain View,
California: VERITAS Software Corporation, August 2001,
number 30-000395-011, TechPDF ID 240256.
● VERITAS Volume Manager™ 3.2 Troubleshooting Guide. Mountain
View, California: VERITAS Software Corporation, August 2001,
number 30-000394-011, TechPDF ID 240255.
● VERITAS Volume Manager™ Storage Administrator 3.2 Administrator’s
Guide. Mountain View, California: VERITAS Software Corporation,
July 2001, number 30-000393-011, TechPDF ID 240257.
● http://storage.east, http://storage.central, and
http://storage.west.
● SunSolveSM Online INFODOCS 14189, 17714, and 19492,
[http://sunsolve.Sun.COM/pub-cgi/search.pl?mode=advanced].
● VERITAS Software Corporation support knowledge base
TechNote IDs 230184, 240004, and 240006,
[http://seer.support.veritas.com/nav_bar/index.asp?]
[content_sURL=%2Fsearch%5Fforms%2Ftechsearch%2Easp].

Sun Proprietary: Internal Use Only


Upgrading the VxVM Software 7-3
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Surveying the Upgrade Processes and Procedures

Surveying the Upgrade Processes and Procedures


Performing the VxVM software and Solaris OE upgrades requires
thoughtful planning and exact execution of the upgrade procedures to be
successful. Upgrades to the VxVM software are performed using either
scripted or manual procedures. Additionally, with the release of version
3.5 of the VxVM software, upgrade procedures may be performed using
the Solaris OE pkgadd utility.

The VxVM software upgrade procedures are affected by the presence of


the following:
● Encapsulated boot disk
● Non-encapsulated boot disk
● The VxVM software licensing
● Patches

The scripted, manual, and pkgadd upgrade processes make provisions for
the encapsulated state of the boot disk. These procedures, as applied to
the boot disk, must be executed precisely, or the upgrade fails.

Upgrades to the Solaris OE are also affected if the VxVM software is


installed. Some of the issues faced with this upgrade are:
● Encapsulated boot disk
● Volume configuration information preservation
● Manual upgrade as opposed to a JumpStart™ process or flash
upgrade
● Concurrent upgrades of both the Solaris OE and the VxVM software
● The VxVM software licensing
● Patches

Upgrading With a Script (VxVM 3.1/3.2/3.5)


The VxVM software provides upgrade scripts which can be used to
perform the VxVM software upgrades. These scripts are found in:
● /CD_Path/scripts
● /Package_Distribution_Path/scripts

Sun Proprietary: Internal Use Only


7-4 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Surveying the Upgrade Processes and Procedures

The upgrade scripts include:


● upgrade_start – Performs first-phase checks and preliminary
operations for the VxVM software upgrade, including the following:
● Check for encapsulated boot disk
● Check for supported level of alternate pathing (AP) installed
● Check for required patches
● Check for problems that prevent VxVM software upgrades
Operational activities include the following:
● Handles encapsulated boot disk issues
● Performs necessary reboots
● Saves system configuration files such as the /etc/vfstab and
/etc/system files
● Saves pre-upgrade VxVM software configuration files such as
the /kernel/drv/dmp.conf file
● Unmounts volumes
● Upgrades the VxVM software eeprom entries
● upgrade_finish – Completes the VxVM software upgrade by
restoring saved files and restarting the VxVM software.

There are extensive sections of code in each script that is dedicated to


managing the upgrade process if AP is installed.

Although this is called a scripted upgrade, there are manual steps needed
to finish the upgrade process. A general description of the manual steps
needed is as follows:
1. Obtain and install any new license keys needed for the new release
of the VxVM software.
2. Make sure that any system-level file systems that are under VxVM
software control have at least one plex where they begin on a
cylinder boundary.
3. If installing any documentation or man page packages, /opt must
exist, be writable and not be symbolically linked.
4. Boot the system to single-user mode.
5. Load and mount the VxVM software upgrade CD-ROM.
6. Execute the upgrade_start script.
7. Reboot to single-user mode.

Sun Proprietary: Internal Use Only


Upgrading the VxVM Software 7-5
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Surveying the Upgrade Processes and Procedures

8. Remove the old VxVM software packages.


9. Reboot the system to single-user mode.
10. Load and mount the VxVM software upgrade CD-ROM.
11. Install the VRTSlic package.
12. Install the remaining VxVM software upgrade packages.
13. Run the upgrade_finish script.
14. Perform a reconfiguration reboot.

A complete description of the upgrade process is found in VERITAS


Software Corporation TechNote ID 230184 and the VERITAS Volume
Manager™ 3.2 Installation Guide.

Caution – Be sure to backup the boot disk prior to performing any


upgrade of the VxVM software or the Solaris OE.

Upgrading Manually (VxVM 3.1/3.2/3.5)


The manual upgrade process is performed according to the operations
handled by the upgrade scripts, as discussed in the previous section. In
the manual upgrade process, however, the scripts are not used, and all
operations are performed by administrators.

There are two types of manual upgrades.


● Upgrading when the boot disk is encapsulated – Detailed
instructions for this procedure are found in VERITAS Software
Corporation TechNote ID 240006.
● Upgrading when the boot disk is unencapsulated – Detailed
instructions for this procedure are found in VERITAS Software
Corporation TechNote ID 240004.

Caution – Be sure to back up the boot disk prior to performing any


upgrade of the VxVM software or the Solaris OE.

Sun Proprietary: Internal Use Only


7-6 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Surveying the Upgrade Processes and Procedures

Manually Upgrading With an Encapsulated Boot Disk

Detailed instructions on manually upgrading the VxVM software with an


encapsulated boot disk are found in VERITAS Software Corporation
TechNote ID 240006. Refer to the procedures in this document when
upgrading the VxVM software.

The general steps to manually upgrade the VxVM software with an


encapsulated boot disk are:
1. Back up the system partitions.
2. If the boot disk is under VxVM control and it is mirrored, use the
vxplex command to break off the mirrors.
3. Unencapsulate the boot disk using the vxunroot script.
4. Completely remove the VMSA software package. Make sure that the
/opt/VRTSvmsa directory is removed.
5. Remove the remainder of the VxVM software packages.
6. Install the required Solaris OE patches.
7. Install the new VxVM and VMSA software packages.
8. Apply the latest VxVM and VMSA software patches for the release
installed
9. Run the vxinstall command
10. Re-encapsulate the boot disk using the recommended Sun best
practices.
11. Re-mirror the boot disk.
12. Import the original disk groups. Make sure that they are upgraded, if
necessary, to use any new features provided by the new release of
the VxVM software.

Sun Proprietary: Internal Use Only


Upgrading the VxVM Software 7-7
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Surveying the Upgrade Processes and Procedures

Manually Upgrading With a Non-Encapsulated Boot Disk

Detailed instructions on manually upgrading the VxVM software with a


non-encapsulated boot disk are found in VERITAS Software Corporation
TechNote ID 240004. Refer to the procedures in this document when
upgrading the VxVM software.

The general steps to upgrade the VxVM software manually with a non-
encapsulated boot disk are:
1. Back up the system partitions.
2. Completely remove the VMSA software package. Make sure that the
/opt/VRTSvmsa directory is removed.
3. Install required Solaris OE patches.
4. Install the new VxVM and VMSA software packages.
5. Apply the latest VxVM and VMSA software patches for the release
installed.
6. Run the vxinstall command.
7. Import the original disk groups. Make sure that they are upgraded, if
necessary, to use any new features provided by the new release of
the VxVM software.

Sun Proprietary: Internal Use Only


7-8 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Surveying the Upgrade Processes and Procedures

Upgrade Using pkgadd (VxVM 3.5)


Release 3.5 of the Volume Manager software provides a full upgrade from
previous releases using scripts included in the packages supplied with the
upgrade. These scripts perform all tasks needed to address boot disk
encapsulation and any other operations necessary to perform the
upgrade. The full upgrade procedure using pkgadd is found in the
VERITAS Volume Manager™ 3.5 Installation Guide.

Upgrading to release 3.5 using pkgadd has its advantages over using the
supplied upgrade scripts, and there are some disadvantages. Table 7-1
compares the two processes.

Table 7-1 VxVM 3.5 Upgrade Comparison

Using pkgadd Using Upgrade Scripts

Advantage Usually requires one 1. There will only be


reboot. one instance of the
VRTSvxvm package
displayed by the
pkginfo command.
2. VxVM configuration
data is backed up.
3. The root disk is
unencapsulated, so
there is no risk of
not being able to
boot the system.
4. The VRTSlic
package can be
deinstalled if there
are no other
packages using it.

Sun Proprietary: Internal Use Only


Upgrading the VxVM Software 7-9
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Surveying the Upgrade Processes and Procedures

Table 7-1 VxVM 3.5 Upgrade Comparison (Continued)

Using pkgadd Using Upgrade Scripts


Disadvantage 1. If a system failure Usually requires three
occurs during the reboots.
pkgadd process, the
boot disk may
become unbootable.
2. The new package is
displayed as
VRTSvxvm2 by
pkginfo unless
instance=
overwrite is
inserted in the
default packge
admin file.

Note – The procedure for


modifying the default
packge admin file to
support the upgrade
without having multiple
instances of VxVM
packages is listed in
VERITAS TechNoteID:
248394

3. The VRTSlic
package might not
be able to be
removed if a second
VRTSvxvm package
is using it.

Following is a high level overview of the VxVM 3.5 upgrade process


using pkgadd.
1. Make a copy of the default packge admin file.
2. Modify the copy so “instance=overwrite.”
3. Upgrade to VxVM 3.5 using the following pkgadd command syntax:
# pkgadd -a <location_of_the_modified_package_admin_file> -d . VRTSvxvm

Sun Proprietary: Internal Use Only


7-10 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Surveying the Upgrade Processes and Procedures

Caution – It is important that the correct procedure is used to upgrade to


release 3.5. If the procedures listed in the VERITAS Volume Manager™ 3.5
Installation Guide are not followed precisely, the system may become
unusable.

Upgrading a Disk Group


When a new version of the VxVM software is installed, the versions of
configured disk groups might need to be upgraded to enable new
functionality provided by the upgrade.

To upgrade a disk group, use the vxdg command with the following
syntax:
# vxdg upgrade disk_group_name

Sun Proprietary: Internal Use Only


Upgrading the VxVM Software 7-11
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Surveying the Upgrade Processes and Procedures

Release Notes
All upgrades and patches include release notes. Be sure to read the release
notes prior to performing any upgrade or patch. Release notes contain
valuable installation and operation information specific to the patch or
release of the VxVM software being installed.

Release notes are normally found in the base directory of the patch or
upgrade distribution and contain either release or notes in the file
name.

Licensing
When upgrading the VxVM software, licensing must be taken into
consideration. If upgrading from the VxVM software version 2.x to 3.x, a
version 3.x license key must be installed prior to the upgrade, or the
VxVM software does not start.

Licensing issues are discusses in the following SunSolve INFODOCs:


● 17714 – “VxVM and SEVM Licensing Explained”
● 19492 – “How to Move a VxVM License between Systems”

In addition, the upgrade procedures referred to in this module have a


section on pre-upgrade licensing.

Sun Proprietary: Internal Use Only


7-12 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Upgrading the Solaris OE

Upgrading the Solaris OE


Upgrading the Solaris OE with the VxVM software installed requires
execution of additional steps to account for encapsulation of the boot disk.
Additionally, the VxVM software can be upgraded along with the
Solaris OE. This is a complex and risky upgrade.

The following documents provide procedures to upgrade the Solaris OE


and the VxVM software on a system with the VxVM software installed.
The instructions provided in these documents must be followed exactly to
ensure success:
● VERITAS Volume Manager 3.2 Installation Guide
● SunSolve INFODOC 14189
● VERITAS Software Corporation TechNote ID 230184

Caution – Be sure to back up the boot disk prior to performing any


upgrade of the VxVM software or the Solaris OE.

A Solaris OE upgrade using the JumpStart process or flash install requires


extensive scripting to successfully complete the upgrade. It is beyond the
scope of this course to provide the knowledge necessary to build these
scripts.

Sun Proprietary: Internal Use Only


Upgrading the VxVM Software 7-13
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Upgrading the VxVM Software

Exercise: Upgrading the VxVM Software


In this exercise, you complete the following tasks:
● Upgrade the VxVM software to a later release using either scripted
or manual methods
● Upgrade disk groups

Preparation
To prepare for this exercise:
● All lab systems must have Solaris 8 OE flash or JumpStart installed,
with all supporting packages and patches.
● At a minimum, the VxVM software version 3.2 must be installed and
configured.
● The boot disk must be encapsulated.
● The VxVM software packages and patches for the new version must
be available either through network file system (NFS) mounts, ftp,
or local access.
● You must have access either to all SunSolve and VERITAS Software
Corporation documents referenced in this module.

Tasks
Complete the following steps:
1. Upgrade the VxVM software to a newer release using one of the
following methods:
● Upgrade using the upgrade scripts located in the
/Package_Distribution_Path/scritps directory.
Reference the VERITAS Volume Manager™ 3.x Installation Guide
section “Upgrading VxVM on an Encapsulated Root Disk.” This
section provides a detailed, step-by-step procedure for using
the upgrade scripts.
● Upgrade using manual procedures.
Reference VERITAS Software Corporation TechNote ID 240006
for the procedures to perform the upgrade without using the
upgrade scripts.

Sun Proprietary: Internal Use Only


7-14 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise: Upgrading the VxVM Software

The method used by your lab group is not as important as


completing the procedure successfully. Both procedures work. Pick
the procedure that best fits the experience level of your lab group
Do not perform the backup of the boot disk as recommended by the
procedures. Tape drives are not available in the lab.
● Upgrade using pkgadd procedures.
Reference VERITAS Software Corporation TechNote ID 248394
for the procedures to perform the upgrade without generating
additional instances of the VxVM software packages.
2. Re-encapsulate the boot disk.
3. Upgrade all configured disk groups if not specified by the upgrade
procedures.
4. Mount any volumes not mounted as part of the upgrade procedures.
5. Verify full functionality and fix any problems encountered.

Sun Proprietary: Internal Use Only


Upgrading the VxVM Software 7-15
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Exercise Summary

Exercise Summary

Discussion – Take a few minutes to discuss what experiences, issues, or


discoveries you had during the lab exercise.
!
?
● Experiences
● Interpretations
● Conclusions
● Applications

Sun Proprietary: Internal Use Only


7-16 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Appendix A

SunSolve INFODOCs

This appendix contains the following SunSolveSM Online Free Info Docs
online sources, available at:
http://sunsolve.Sun.COM/pub-cgi/search.pl?mode=advanced
● INFODOC 16051 – “How to ‘Encapsulate’ Disks With No Free Space
Using Volume Manager.” 22 March 2002.
● INFODOC 24663 – “Full and Basic/Functional Unencapsulation of a
Volume Manager Encapsulated Root Disk While Booted CDROM.”
22 March 2002.

Sun Proprietary: Internal Use Only


A-1
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
INFODOC 16051

INFODOC 16051
INFODOC ID: 16051
SYNOPSIS: How to 'Encapsulate' Disks With No Free Space Using Volume Manager
DETAIL DESCRIPTION:
To encapsulate a disk into Veritas Volume Manager or Sun Enterprise Volume Manager[TM],
you must have some free space on the disk in order for Volume Manager to write a private
region to the disk. The private region is generally smaller than 2mb.
However, if you absolutely do not have any free space on the disk, and you can't free up
any, and you really want to get this data under Volume Manager control, you can work
around this by TEMPORARILY "encapsulating" one or more slices from a disk into Volume
Manager so that the data may be mirrored to another disk. Once the data is mirrored to
a "real" Volume Manager disk with a private and public region, you can then break the
mirror, leaving the data on the "real" Volume Manager disk.
SOLUTION SUMMARY:
Here is how to do it:
For each slice on the disk (excluding slice 2), run the following command. In this
example, only slices 5 and 6 have data on them.
vxdisk define c#t#d#s5 type=nopriv
vxdisk define c#t#d#s6 type=nopriv
Then add each of these "slices" as a disk in a disk group and give them a name. This
example names them NPdisk05 and NPdisk06.
vxdg -g <diskgroup> adddisk NPdisk05=c#t#d#s5
vxdg -g <diskgroup> adddisk NPdisk06=c#t#d#s6
Next we create a simple volume (not a file system, just a volume) on each of these new
"disks" that spans the entire "disk". To do this we first check to see what the max size
is for the volumes we are about to create. We're looking for the len value to then use
with the vxassist command to create the volumes.
vxdisk list NPdisk05 | grep public
public: slice=0 offset=0 len=8196096
vxdisk list NPdisk06 | grep public
public: slice=0 offset=0 len=9400320
With this info we create the volumes, naming them NPdisk05vol and NPdisk06vol:
vxassist -g <diskgroup> make NPdisk05vol 8196096 layout=nostripe alloc="NPdisk05"
vxassist -g <diskgroup> make NPdisk06vol 9400320 layout=nostripe alloc="NPdisk06"
Next step is to mirror the volumes, assuming that we are mirroring the volumes to a disk
named disk01 that has enough space to mirror both volumes to it:
vxassist -g <diskgroup> mirror NPdisk05vol layout=nostripe alloc="disk01"
vxassist -g <diskgroup> mirror NPdisk06vol layout=nostripe alloc="disk01"
Once that is complete we then remove the original side of the mirror.
vxplex -g <diskgroup> -o rm dis NPdisk05vol-01
vxplex -g <diskgroup> -o rm dis NPdisk06vol-01
The final step is to remove the old disks from the disk group and return them to
their original state.
vxdg -g <diskgroup> rmdisk NPdisk05
vxdg -g <diskgroup> rmdisk NPdisk06
vxdisk rm c0t5d10s5
vxdisk rm c0t5d10s6
This leaves us with two concat volumes named NPdisk05vol and NPdisk06vol.
These volumes will contain the data that was orignally located on
c#t#d#s5 and c#t#d#s6.

Sun Proprietary: Internal Use Only


A-2 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
INFODOC 16051

Keywords: SEVM, VxVM, Volume Manager, recover, recovery, configure, configured,


configuration
APPLIES TO: Storage/Veritas, Storage/Volume Manager, AFO Vertical Team Docs/Storage
ATTACHMENTS:

Sun Proprietary: Internal Use Only


SunSolve INFODOCs A-3
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
INFODOC 24663

INFODOC 24663
INFODOC ID: 24663
SYNOPSIS: Full and Basic/Functional Unencapsulation of a Volume Manager Encapsulated
Root Disk While Booted CDROM
DETAIL DESCRIPTION:
Overview:
This document explains the steps necessary to unencapsulate the root disk from Volume
Manager control. This document applies to both Sun Enterprise Volume Manager[TM] (SEVM)
2.x and Veritas Volume Manager (VxVM) 3.x.
This document is divided into two distinct sections. The first section describes full
unencapsulation while booted from a Solaris CDROM. This procedure should be used any
time it is necessary to completely remove the root disk from Volume Manager control and
bring the disk back to a pre-encapsulation state including all partitions such as
/export, and /opt.
The second section explains the steps to perform a Basic/Functional (BF) unencapsulation
while booted from a Solaris CDROM. Basic/Functional unencapsulation temporarily
unencapsulates the root disk so that troubleshooting of booting issues or other issues
can be done. BF unencapsulation gives you access to an unencapsulated /, swap, /usr, and
/var but no access to non "big-4" partitions.
SOLUTION SUMMARY:
Notes for Full Unencapsulation:
Under normal circumstances, if the system can be booted to at least single user mode, it
is recommended that the vxunroot command be used to unencapsulate root. A full
unencapsulation should be performed if the vxunroot command is not working for some
reason, or if the system cannot be booted and we want to completely remove Volume
Manager from having any control over the root disk.
You cannot perform a full unencapsulation and still maintain Volume Manager
functionality if the root disk is the ONLY disk in the rootdg diskgroup. If the root
disk is the only disk in rootdg, you can still unencapsulate, but Volume Manager will
not work until another disk is initialized into rootdg using vxinstall after the system
has been fully unencapsulated. Normally if root it is encapsulated, it is also mirrored
which gives us another disk in rootdg, however it should always be verified that there
is at least one other disk in rootdg before following this procedure so that we know
what to expect once root is unencapsulated.
Also note that Volume Manager will allow you to create volumes using free space on the
root disk, after the root disk has been encapsulated. Volumes created post-encapsulation
like this do not have underlying hard partitions and therefore are not recoverable with
this procedure. If at all possible, make backups of any volumes created on the rootdisk
post-encapsulation before following this procedure. Once you are unencapsulated, if you
have free space and a free partition, you could newfs that partition and restore to it
from your backup.
Steps for Full Unencapsulation:
Bring the system to the OK prompt and insert a Solaris CD into the CDROM drive.
Then issue:
boot cdrom -s
Once booted to the cdrom, set your terminal type so that vi will work correctly.
If TERM=sun doesn't work, often times TERM=vt100 will.
TERM=sun;export TERM
Fsck your root filesystem:
fsck -y /dev/rdsk/c#t#d#s0

Sun Proprietary: Internal Use Only


A-4 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
INFODOC 24663

If fsck comes back cleanly, mount slice 0 to /a. If fsck cannot repair the root file
system, there are obviously a number of possibilities. This procedure does not attempt
to explain file system corruption or how to repair it beyond fsck. Fsck must come back
cleanly to continue and mount root.
mount /dev/dsk/c#t#d#s0 /a
Make a backup of /a/etc/system and then edit it:
cp /a/etc/system /a/etc/system.orig
vi /a/etc/system
Completely remove the following lines from the system file. If you re-encapsulate in the
future, these lines will be added back correctly so there is nothing to be lost by
removing them.
rootdev:/pseudo/vxio@0:0
set vxio:vol_rootdev_is_volume=1
Make a backup of /a/etc/vfstab and then edit it:
cp /a/etc/vfstab /a/etc/vfstab.orig
vi /a/etc/vfstab
Edit the vfstab file back to it's original state, pointing /, swap, /usr, and /var to
hard partitions on the disk like /dev/dsk and /dev/rdsk rather than /dev/vx/ entries.
Temporarily comment out all other /dev/vx volumes from the /a/etc/vfstab file using the
# character. This includes filesystems like /opt and /export, if they exist.
The original /etc/vfstab will look something like this, assuming root is c0t0d0:
Note: Columns have been aligned and spaces added for clarity.
---------------------------------------------------------------------------
/dev/vx/dsk/swapvol - - swap - no -
/dev/vx/dsk/rootvol /dev/vx/rdsk/rootvol / ufs 1 no -
/dev/vx/dsk/usr /dev/vx/rdsk/usr /usr ufs 1 no -
/dev/vx/dsk/var /dev/vx/rdsk/var /var ufs 1 no -
/dev/vx/dsk/export /dev/vx/rdsk/export /export ufs 2 yes -
swap - /tmp tmpfs - yes -

/dev/vx/dsk/datadg/somevol /dev/vx/rdsk/datadg/somevol /somevol ufs 2 yes -

#NOTE: volume rootvol (/) encapsulated partition c0t0d0s0


#NOTE: volume swapvol (swap) encapsulated partition c0t0d0s1
#NOTE: volume usr (/usr) encapsulated partition c0t0d0s5
#NOTE: volume var (/var) encapsulated partition c0t0d0s6
#NOTE: volume export (/export) encapsulated partition c0t0d0s7
---------------------------------------------------------------------------
Once edited, the vfstab should look something like this:
---------------------------------------------------------------------------
/dev/dsk/c1t0d0s1 - - swap - no -
/dev/dsk/c1t0d0s0 /dev/rdsk/c0t0d0s0 / ufs 1 no -
/dev/dsk/c1t0d0s5 /dev/rdsk/c0t0d0s5 /usr ufs 1 no -
/dev/dsk/c1t0d0s6 /dev/rdsk/c0t0d0s6 /var ufs 1 no -
#/dev/dsk/c1t0d0s7 /dev/rdsk/c0t0d0s7 /export ufs 2 yes -
swap - /tmp tmpfs - yes -

#/dev/vx/dsk/datadg/somevol /dev/vx/rdsk/datadg/somevol /somevol ufs 2 yes -

#NOTE: volume rootvol (/) encapsulated partition c0t0d0s0


#NOTE: volume swapvol (swap) encapsulated partition c0t0d0s1
#NOTE: volume usr (/usr) encapsulated partition c0t0d0s5
#NOTE: volume var (/var) encapsulated partition c0t0d0s6
#NOTE: volume export (/export) encapsulated partition c0t0d0s7

Sun Proprietary: Internal Use Only


SunSolve INFODOCs A-5
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
INFODOC 24663

---------------------------------------------------------------------------
Now make sure Volume Manager does not start on the next boot:
touch /a/etc/vx/reconfig.d/state.d/install-db
This is important because IF the root disk contains mirrors, and the system boots up,
the mirrors will get resynced, corrupting the changes we just made.
Remove the flag that tells Volume Manager that the root disk is encapsulated:
rm /a/etc/vx/reconfig.d/state.d/root-done
Reboot the system for changes to take effect:
reboot
When we reboot we will come up in an partially unencapsulated state with /, /usr, /var,
and swap mounted. Volume Manager will not start but we can start it manually once we
are booted.
To start Volume Manager, run the following commands:
rm /etc/vx/reconfig.d/state.d/install-db
vxiod set 10
vxconfigd -m disable
vxdctl enable
Now we can remove the volumes that existed on the encapsulated boot disk. They will
generally be rootvol, swapvol, usr, and var. They might also include home, opt, or
other non-standard root partitions. Use the command 'vxprint -htg rootdg' to list the
volumes in rootdg before removing them. Then, for each volume, run the command:
/usr/sbin/vxedit -rf rm <volume name>
Remove the rootdisk from rootdg now that it has no volumes, plexes or subdisks: The disk
name is usually 'rootdisk'
/usr/sbin/vxdg rmdisk <disk name>
The final step is to re-write the vtoc of the disk so that hard partitions are again
defined for the root file systems. There are several ways to put the hard partitions
back, including using fmthard on a modified /etc/vx/reconfig.d/disk.d/c#t#d#/vtoc file,
using format to manually repartition the disk, or using the vxmksdpart command. The
simplest method however is to use the vxedvtoc command as explained below.
When Volume Manager encapsulates a disk, it makes a record of the old vtoc of the disk.
This file is stored for each disk in /etc/vx/reconfig.d/disk.d/c#t#d#. This file is
stored in a Volume Manager specific format, so it can't be used as an argument to
fmthard unless it is modified. The 'vxedvtoc' command is similar to fmthard but knows
how to read this vtoc file and write that vtoc to a disk. The command takes the form:
vxedvtoc -f <filename> <devicename>
Assuming that the boot disk is c0t0d0 we would now run the command
/etc/vx/bin/vxedvtoc -f /etc/vx/reconfig.d/disk.d/c0t0d0/vtoc /dev/rdsk/c0t0d0s2
# THE ORIGINAL PARTITIONING IS AS FOLLOWS :
#SLICE TAG FLAGS START SIZE
0 0x0 0x200 0 0
1 0x0 0x200 0 0
2 0x5 0x201 0 8794112
3 0x0 0x200 0 0
4 0x0 0x200 0 0
5 0x0 0x200 0 0
6 0xe 0x201 0 8794112
7 0xf 0x201 8790016 4096
# THE NEW PARTITIONING WILL BE AS FOLLOWS :
#SLICE TAG FLAGS START SIZE
0 0x0 0x200 0 2048000
1 0x0 0x200 2048000 2048000
2 0x5 0x201 0 8794112
3 0x0 0x201 4096000 2048000

Sun Proprietary: Internal Use Only


A-6 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
INFODOC 24663

4 0x0 0x201 6144000 2048000


5 0x0 0x200 0 0
6 0x0 0x200 0 0
7 0x0 0x200 0 0
DO YOU WANT TO WRITE THIS TO THE DISK ? [Y/N] :y
WRITING THE NEW VTOC TO THE DISK
This will partition the disk back to a pre-encapsulation state. Now we can uncomment
the entries for any of the non Big-4 root partitions from /etc/vfstab, as well as any
data volumes. In this example we removed comments from /export and the data volume
/somevol.
vi /etc/vfstab

/dev/dsk/c0t0d0s1 - - swap - no -
/dev/dsk/c0t0d0s0 /dev/rdsk/c0t0d0s0 / ufs 1 no -
/dev/dsk/c0t0d0s5 /dev/rdsk/c0t0d0s5 /usr ufs 1 no -
/dev/dsk/c0t0d0s6 /dev/rdsk/c0t0d0s6 /var ufs 1 no -
/dev/dsk/c0t0d0s7 /dev/rdsk/c0t0d0s7 /export ufs 2 yes -
swap - /tmp tmpfs - yes -

/dev/vx/dsk/datadg/somevol /dev/vx/rdsk/datadg/somevol /somevol ufs 2 yes -


Just to make sure, start all volumes
/usr/sbin/vxvol startall
Now issue a mountall to mount the now uncommented volumes
mountall
At this point the root disk is completely free of Volume Manager control, Volume Manager
daemons are started, and all file systems / volumes should be mounted.
Notes for Basic/Functional Unencapsulation:
This section explains the steps necessary to temporarily unencapsulate the root disk
from Volume Manager control. BF unencapsulation allows the system to be booted from the
raw Solaris partitions while still leaving the root disk under Volume Manager control.
This is a good method for troubleshooting boot issues that appear to be due to the disk
being encapsulated because it can be undone by reversing the steps. This procedure can
be used even if the root disk is the only disk in the rootdg diskgroup because,
throughout the procedure, root will keep it's private and public region. This procedure
will only allow you to mount /, /usr, /var, and swap. Any non "Big-4" partitions will
not be mounted. If you must have non "Big-4" partitions available, you should perform a
full unencapsulation as outlined above.
Steps for Basic/Functional Unencapsulation:
Bring the system to the OK prompt and insert a Solaris CD into the cdrom drive. Then
issue:
boot cdrom -s
Once booted to the cdrom, set your terminal type so that vi will work correctly. If
TERM=sun doesn't work, often times TERM=vt100 will.
TERM=sun;export TERM
Fsck your root filesystem:
fsck -y /dev/rdsk/c#t#d#s0
If fsck comes back cleanly, mount slice 0 to /a. If fsck cannot repair the root file
system, there are obviously a number of possibilities. This procedure does not attempt
to explain file system corruption or how to repair it. fsck must come back cleanly to
continue and mount root.
mount /dev/dsk/c#t#d#s0 /a
Make a backup of /a/etc/system and then edit it:
cp /a/etc/system /a/etc/system.orig
vi /a/etc/system

Sun Proprietary: Internal Use Only


SunSolve INFODOCs A-7
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
INFODOC 24663

Comment out the following lines using double asterisks **.

**rootdev:/pseudo/vxio@0:0
**set vxio:vol_rootdev_is_volume=1
Make a backup of /a/etc/vfstab and then edit it:
cp /a/etc/vfstab /a/etc/vfstab.orig
vi /a/etc/vfstab
Edit the vfstab file back to it's original state, pointing /, swap, /usr, and /var to
hard partitions on the disk like /dev/dsk and /dev/rdsk rather than /dev/vx/ entries.
Temporarily comment out all other /dev/vx volumes from the /a/etc/vfstab file using the
# character. This includes filesystems like /opt and /export, if they exist.
The original /etc/vfstab will look something like this, assuming root is c0t0d0: Note:
Columns have been aligned and spaces added for clarity.
---------------------------------------------------------------------------
/dev/vx/dsk/swapvol - - swap - no -
/dev/vx/dsk/rootvol /dev/vx/rdsk/rootvol / ufs 1 no -
/dev/vx/dsk/usr /dev/vx/rdsk/usr /usr ufs 1 no -
/dev/vx/dsk/var /dev/vx/rdsk/var /var ufs 1 no -
/dev/vx/dsk/export /dev/vx/rdsk/export /export ufs 2 yes -
swap - /tmp tmpfs - yes -

/dev/vx/dsk/datadg/somevol /dev/vx/rdsk/datadg/somevol /somevol ufs 2 yes -

#NOTE: volume rootvol (/) encapsulated partition c0t0d0s0


#NOTE: volume swapvol (swap) encapsulated partition c0t0d0s1
#NOTE: volume usr (/usr) encapsulated partition c0t0d0s5
#NOTE: volume var (/var) encapsulated partition c0t0d0s6
#NOTE: volume export (/export) encapsulated partition c0t0d0s7
---------------------------------------------------------------------------
Once edited, it should look something like this:
---------------------------------------------------------------------------
/dev/dsk/c0t0d0s1 - - swap - no -
/dev/dsk/c0t0d0s0 /dev/rdsk/c0t0d0s0 / ufs 1 no -
/dev/dsk/c0t0d0s5 /dev/rdsk/c0t0d0s5 /usr ufs 1 no -
/dev/dsk/c0t0d0s6 /dev/rdsk/c0t0d0s6 /var ufs 1 no -
#/dev/dsk/c0t0d0s7 /dev/rdsk/c0t0d0s7 /export ufs 2 yes -
swap - /tmp tmpfs - yes -

#/dev/vx/dsk/datadg/somevol /dev/vx/rdsk/datadg/somevol /somevol ufs 2 yes -


#NOTE: volume rootvol (/) encapsulated partition c0t0d0s0
#NOTE: volume swapvol (swap) encapsulated partition c0t0d0s1
#NOTE: volume usr (/usr) encapsulated partition c0t0d0s5
#NOTE: volume var (/var) encapsulated partition c0t0d0s6
#NOTE: volume export (/export) encapsulated partition c0t0d0s7
---------------------------------------------------------------------------
Now make sure Volume Manager does not start on the next boot:
touch /a/etc/vx/reconfig.d/state.d/install-db
The presence of install-db tells Volume Manager not to start at boot. This is important
because IF the root disk contains mirrors, and the system boots up, the mirrors will get
resynced, corrupting the changes we just made.
Remove the flag that tells Volume Manager that the root disk is encapsulated:
rm /a/etc/vx/reconfig.d/state.d/root-done
Reboot the system for changes to take effect:
reboot

Sun Proprietary: Internal Use Only


A-8 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
INFODOC 24663

When we reboot we will come up in an unencapsulated state with /, /usr, /var, and swap
mounted.
At this point we have performed a Basic/Functional unencapsulation. This is a not a
state that they system should be left in permanently. It is a state that is useful for
troubleshooting and system maintenance.
If problems with the system are resolved and you are ready to re-encapsulate, perform
the following:
touch /etc/vx/reconfig.d/state.d/root-done
rm /etc/vx/reconfig.d/state.d/install-db
cp /a/etc/vfstab.orig /a/etc/vfstab
cp /a/etc/system.orig /a/etc/system
reboot
Keywords: SEVM, VxVM, Volume Manager, encapsulation
APPLIES TO: Operating Systems/Solaris/Solaris 8, Operating Systems/Solaris/Solaris 7,
Operating Systems/Solaris/Solaris 2.6, Operating Systems/Solaris/Solaris 2.5.1,
Storage/Veritas, Storage/Volume Manager, AFO Vertical Team Docs/Storage
ATTACHMENTS:

Sun Proprietary: Internal Use Only


SunSolve INFODOCs A-9
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Appendix B

Disk Encapsulation Processes

This appendix contains flow charts which outline the processes used to
perform the following tasks:
● Encapsulate non-root or data disks
● Unencapsulate non-root or data disks
● Encapsulated boot disks
● Unencapsulate boot disks by using the vxunroot command
● Unencapsulate boot disks by not using the vxunroot command
● Unencapsulate boot disks when the system is booted from a CD-
ROM
● Perform a basic boot disk unencapsulation
● Perform the Sun Enterprise Services’s best practice procedures for
managing boot disks with the VxVM software

Sun Proprietary: Internal Use Only


B-1
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Start

Encapsulate? Yes Boot Disk? Yes B Page 10 of 26

No

No
A Page 2 of 26

Unencapsulate? Yes Boot Disk? Yes


D Page 11 of 26

No
No

C Page 7 of 26
End

Disk Encapsulation Flowchart

Rev-B 07/18/2002
Page 1 of 26

Sun Proprietary: Internal Use Only


B-2 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
This flow has been selected because a
A disk with existing data must be brought

under VxVM software control.

Selected disk must Answer all prompts


Enter the disk
have sufficient space to appropriate for this
Select disk to group name that
hold the private region encapsulation procedure
encapsulate. this disk shall
and have two unused
join
slices. If encapsulating the boot

disk, do not use default

disk name.

Disk names can be up


Selected disk has If a default disk
sufficient free space and
No
E name was not
to 31 characters in

length.
partitions for previously
encapsulation? selected, enter
If encapsulating the boot
an arbitrary
disk, use the name
name now
rootdisk.
Page 3

Yes
Yes of 26
is the normal

selection for this

Open question. If

vxdiskadm # vxdiskadm Use default changing default

utility. private region No size, all disks in

size? this disk group

must have the

same private

region size.
Select Yes
encapsulation 2
utility.

Exit vxdiskadm
Encapsulate
No utility and reboot
additional disks?
the system.
Select disk to

encapsulate
c#t#d#

End

Answer yes to

continue y
operation
Disk Encapsulation Flowchart

Rev-B 07/18/2002

Page 2 of 26

Yes

Sun Proprietary: Internal Use Only


Disk Encapsulation Processes B-3
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
This flow has been selected because a disk with existing data
E (non root) that does not met minimum encapsulation
requirements, must be brought under VxVM software control.

Select the disk to


be encapsulated

For each slice on


the disk (excluding
slice 2) use
vxdisk define c#t#d#s# type=nopriv
vxdisk to bring
under VxVM Example
softwar control
#vxdisk define c1t3d0s5 type=nopriv
Yes #vxdisk define c1t3d0s6 type=nopriv

Configure
additional slices?

No

Add each of these


"slices" as a disk to
vxdg -g <diskgroup> adddisk <diskname>=c#t#d#s#
an existing disk
group
Example

#vxdg -g datadg adddisk NPdisk05=c1t3d0s5


Yes #vxdg -g datadg adddisk NPdisk06=c1t3d0s6

Configure
additional slices?

No

Disk Encapsulation Flowchart


F Page 4 of 26 Rev-B 07/18/2002
Page 3 of 26

Sun Proprietary: Internal Use Only


B-4 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
F This flow has been selected because a process
executing in flow E was directed here.

Document the
maximum size of
the "disks" just
created. This vxdisk list <diskname> | grep public
information shall
be used to create Example

volumes #vxdisk list NPdisk05 | grep public


public: slice=0 offset=0 len=8196096
Yes
#vxdisk list NPdisk06 | grep public
public: slice=0 offset=0 len=9400320

Document
additional slices?

Use the information


from the previous vxassist -g <diskgroup> make <volumename> <size> \
step to create layout=nostripe alloc="<diskname>"
volumes.
Example

Yes
#vxassist -g datadg make NPdisk05vol 8196096
layout=nostripe \ alloc="NPdisk05"
#vxassist -g datadg make NPdisk06vol 9400320
layout=nostripe \ alloc="NPdisk06"
Create additional
volumes?

No

G Page 5 of 26

Disk Encapsulation Flowchart

Rev-B 07/18/2002
Page 4 of 26

Sun Proprietary: Internal Use Only


Disk Encapsulation Processes B-5
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
G This flow has been selected because a process
executing in flow F was directed here.

vxassist -g <diskgroup> mirror <volumename>


Mirror the volumes layout=nostripe \
alloc="<mirrordiskname>"

Yes Example

#vxassist -g datadg mirror NPdisk05vol layout=nostripe \


Mirror additional alloc="disk01"
volumes? #vxassist -g datadg mirror NPdisk06vol layout=nostripe \
alloc="disk01"

No
No

Mirroring process
complete?

Yes

Remove the original


vxplex -g <diskgroup> -o rm dis <originalplexname>
side of the mirror.

Example

#vxplex -g datadg -o rm dis NPdisk05vol-01


Yes #vxplex -g dtatdg -o rm dis NPdisk06vol-01

Remove additional
mirrors?

No

H Page 6 of 26
Disk Encapsulation Flowchart

Rev-B 07/18/2002
Page 5 of 26

Sun Proprietary: Internal Use Only


B-6 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
H This flow has been selected because a process

executing in flow G was directed here.

vxplex -g <diskgroup> rmdisk <diskname>


Remove the old

vxdisk rm c#t#d#s#
disks from the disk

group

Example

Yes
#vxdg -g datadg rmdisk NPdisk05
#vxdg -g datadg rmdisk NPdisk06
#vxdisk rm c1t3d0s5
#vxdisk rm c1t3d0s6
Remove additional

disks?

No

Note: To return these volumes to full redundancy, re-mirror the volumes


End
using another initialized disk within the disk group.

Disk Encapsulation Flowchart

Rev-B 07/18/2002

Page 6 of 26

Sun Proprietary: Internal Use Only


Disk Encapsulation Processes B-7
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
This flow has been selected because an encapsulated data disk
C (non root) must be unencapsulated. Applications using the volumes
encapsulated on this disk must be shutdown.

Any grown Disk is unable to be


Yes
volumes? unencapsulated.

No

Any volumes
Disk is unable to be
structurally Yes
unencapsulated.
modified?

No

Original
Disk is unable to be
disk Yes
unencapsulated.
replaced?

No
Encapsulated disks use slices six and
seven for public and private regions.
Backup data and restore to
Mirror disks use slices three and four.
Identify disk to be non VxVM managed disk.
unencapsulated.
Use the following commands:
vxprint
format / prtvtoc
No
End

Volumes
Review content of
backed-up?
/etc/vfstab file
to define
pre-encapsulation
configuration
Yes

Disk Encapsulation Flowchart

I Page 8 of 26
Rev-B
Page 7 of 26
07/18/2002

Sun Proprietary: Internal Use Only


B-8 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
I This flow has been selected because a process

executing in flow C was directed here.

Unmirror all volumes on the

disk to be unencapsulated
vxplex -g <diskgroup> -o rm dis <plexname>
Example

#vxplex -g datadg -o rm dis fs1-02


Yes
#vxplex -g dtatdg -o rm dis fs2-02

Remove additional

mirrors?

No

Verify that all mirrors have

been detached.
vxprint -qhtg <diskgroup>

The information needed to execute this step is found in the

/etc/vfstab file and output from the vxprint -qhtg


<diskgroup> command.
Recreate original partitions.
vxmksdpart -g <diskgroup> <plexname> <partition#>
\tags flags
Example
Yes
#/etc/vx/bin/vxmksdpart -g datadg datadg01-03 0
0x00 0x00
Additional
#/etc/vx/bin/vxmksdpart -g datadg datadg01-02 1
partitions to
0x00 0x00
restore?

No

The disk is unable to be

unencapsulated, restore
Partititions No the data from backups to End
Restored?
non VxVM software

managed disk.

Yes
Disk Encapsulation Flowchart

Rev-B 07/18/2002
J
Page 9 of 26 Page 8 of 26

Sun Proprietary: Internal Use Only


Disk Encapsulation Processes B-9
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
This flow has been selected because a process
J
executing in flow I was directed here.

Unmount the volumes

Yes

Unmount
additional
volumes?

No

Edit /etc/vfstab file and


restore pre encapsulation
mount statements.

Re mount filesystems

vxedit -r -f rm <volume>
Recursively remove each Example
volume.
#vxedit -r -f rm fs1
#vxedit -r -f rm fs2
Yes

Remove additional No
Remove disk from vxdg rmdisk <diskname>
volumes?
diskgroup. vxdisk rm c#t#d#
Example

#vxdg rmdisk datadg01


#vxdisk rm c1t3d0
Remove Public and Private
Regions using format utility.

End
Disk Encapsulation Flowchart

Rev-B 07/18/2002
Page 9 of 26

Sun Proprietary: Internal Use Only


B-10 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
This flow has been selected because the
B boot disk must be brought under Volume

Manager control.

Encapsulate

X
and mirror using

Sun ES Yes Page 24 of 26

Best Practice for

boot disks?

No

This option is used during the initial

installation of the Volum e Manager

s o f t wa r e . T o e n c a p s u l a t e t h e b o o t
Encapsulate using

vxinstall?
Yes disk, answer yes to the "Encapsulate
Boot Disk" prompt and answer

presented questions as appropriate for

the installation

No

End
A Page 2 of 26

Disk Encapsulation Flowchart

Rev-B 07/18/2002

Page 10 of 26

Sun Proprietary: Internal Use Only


Disk Encapsulation Processes B-11
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
This flow has been selected because the
D boot disk must be removed from Volume

Manager control.

vxunroot? Yes
K Page 12 of 26

No

Manual? Yes
N Page 15 of 26

No

S
Booted

from Yes Page 19 of 26

CDROM?

No

Basic
Yes
V Page 22 of 26
Functionality?

No This unencapsulation procedure, is designed to help troubleshoot

system problems that require the VxVM software to be shutdown.

It is also a good technique to use for system maintenance tasks


End
that require the VxVM software to be shutdown.

Disk Encapsulation Flowchart

Rev-B 07/18/2002

Page 11 of 26

Sun Proprietary: Internal Use Only


B-12 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
K This flow has been selected because a process
executing in flow D was directed here.

No
The following procedure requires a reboot of the
system. Terminate all production applications
and logoff all non administrative users prior to
Current boot
starting.
disk backup?

Yes

Unm ount all file system s


under the control of VxVM
Properly configured boot disks only contain
software except ( /, /usr,
system partitions. This should not be an issue.
/var, /opt) if resident on
the boot disk.

Record the current


vxprint -qhtg rootdg > /rootdg.print
configuration of rootdg .

Record the current disk


configuration.
vxdisk -o alldgs list > /vxdisk_alldgs.list

vxprint -qhtg rootdg -s | grep -i rootmirror | \


awk '{print $3}' > /rmsub.plex
Detach all plexes associated
for i in `cat ./rmsub.plex`
with the " rootmirror" disk.
>do
>vxplex -g rootdg -o rm dis $i
>done
Veify all " rootmirror "
plexes are detached and vxprint -qhtg rootdg
No removed.

All This is a critical step. All "rootmirror" plexes


"rootmirror" must be removed for all volumes residing on the
plexes detached boot disk. If this step does not complete
and removed? successfully, vxunroot will fail.

Yes
Disk Encapsulation Flowchart

L Rev-B
Page 12 of 26
07/18/2002

Sun Proprietary: Internal Use Only


Disk Encapsulation Processes B-13
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
L This flow has been selected because a process
executing in flow D was directed here.

Remove rootability using the


/etc/vx/bin/vxunroot
vxunroot command.

Successful? No
M Page 14 of 26

Yes

Reboot system

Reboot
No
M Page 14 of 26
successful?

Yes

Verify system
file system mounts.

Using non
VxVM software No
M Page 14 of 26
objects

Yes

End

Disk Encapsulation Flowchart

Rev-B 07/18/2002
Page 13 of 26

Sun Proprietary: Internal Use Only


B-14 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
M This flow has been selected because a process
executing in flow O was directed here.

1st vxunroot Go to manual


time through No still Yes unencapsulation D
flow? fails? process.

No a
Yes

vxunroot Go to CDROM
command No
Reboot
Yes unencapsulation D
failed?
failed? process.

No
Yes Execute
vxunroot
Still using unencapsulation
Verify all non
system volumes VxVM software Yes process again. D
on this disk are objects? Follow

unmounted. instructions
exactly as written
No

Verify all volumes


on this disk have Other
not been grown, or Yes Call support.
problem?
structually
modified.
a No End

Verify all
"rootmirror" System
Still using
Other
mirrors have been Yes VxVM software No No
bootable? problem?
detached and objects?
removed.
End
No Yes
Yes

Execute Call support.


Go to CDROM
vxunroot Go to manual
unencapsulation unencapsulation
unencapsulation
process. process.
process again.

Disk Encapsulation Flowchart

D D D Rev-B 07/18/2002
Page 14 of 26

Sun Proprietary: Internal Use Only


Disk Encapsulation Processes B-15
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
N This flow has been selected because a process
executing in flow D was directed here.

No

The following procedure requires a reboot of the


system. Terminate all production applications
Current boot and logoff all non administrative users prior to
disk backup? starting.

Yes

Unmount all file systems


under the control of VxVM
Properly configured boot disks only contain
software except ( , / /usr, /
system partitions. This should not be an issue.
var, /opt) if resident on the
boot disk.

Record the current


vxprint -qhtg rootdg > /rootdg.print
configuration of rootdg .

Record the current disk vxdisk -o alldgs list > /vxdisk_alldgs.list


configuration.

vxprint -qhtg rootdg -s | grep -i rootmirror | \


awk '{print $3}' > /rmsub.plex
Detach all plexes associated
for i in `cat ./rmsub.plex`
with the " rootmirror" disk.
>do
>vxplex -g rootdg -o rm dis $i
>done
Veify all " rootmirror"
plexes are detached and

No
removed. vxprint -qhtg rootdg

All
rootmirror"
This is a critical step.All "rootmirror" plexes must be
"
plexes detached
removed for all volumes residing on the boot disk. If this step
does not complete successfully, this procedure shall fail.
and removed?

Yes
Disk Encapsulation Flowchart

P Page 16 of 26
Rev-B 07/18/2002
Page 15 of 26

Sun Proprietary: Internal Use Only


B-16 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
P This flow has been selected because a process
executing in flow N was directed here.

Remove rootability
using the following
processes:

Remove the rm /etc/vx/reconfig.d/state.d/


root-done file. root-done
cp /etc/system /etc/system.orig
vi /etc/system
Edit the /etc/
system file and Remove (do not comment) the following two
remove " rootdev" lines:
statements.
rootdev:/pseudo/vxio@0:0
set vxio:vol_rootdev_is_volume=1
/
If there are non system ( ,
swap, /usr, /var) partitions Edit the /etc/ Use either the comments at the end of the
on this disk, restore those vfstab file and /etc/
file or the contents of the
mount statements at this time. restore to it's vfstab.prevm to restore the original
These non system partitions pre boot disk mount statements.
could include /opt, /home or encapsulation state.
other named filesystems. The /etc/vfstab.prevm file can be
copied over the/etc/vfstab file providing
there have been no changes to this systems
storage configuration after the boot disk was
If there are any non encapsulated.
"system" partitions on
this disk, use
vxmksdpart -g <diskgroup> \
vxmksdpart <plexname> <partition#> tags
command to restore.
flags
Example

#/etc/vx/bin/vxmksdpart -g
datadg \
reboot the system datadg01-04 5 0x00 0x00

Reboot
No
R Page 18 of 26
successful?

Yes Disk Encapsulation Flowchart

Rev-B 07/18/2002
Q Page 17 of 26
Page 16 of 26

Sun Proprietary: Internal Use Only


Disk Encapsulation Processes B-17
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Q This flow has been selected because a process

executing in flow O was directed here.

All boot Fix problem. Check /etc/vfstab entries. Make sure

disk file systems No that all non system partitions were properly restored and

mounted? match vfstab entries.

Yes

vxedit -r -f rm rootvol
Recursively remove

vxedit -r -f swapvol
boot volumes from

rootdg .

vxprint -qhtg rootdg


Verify volume

removal
No

Volumes

removed?

Yes

Remove the boot disk

from VxVM software vxdg -g rootdg rmdisk rootdisk


control.

Verify disk removal. vxprint -qhtg rootdg


No

Disk

removed?

Yes

Update the VxVM

Delete public and


software's view of vxdctl enable
installed disks.
private region

partitions using the

format utility. Disk Encapsulation Flowchart

End Rev-B 07/18/2002

Page 17 of 26

Sun Proprietary: Internal Use Only


B-18 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
R This flow has been selected because a process

executing in flow P was directed here.

System
Yes End
bootable?

No

Manual
Go to CDROM
unencapsulation
Yes unencapsulation
D
failed. First time
process.
through flow?

No

CDROM

unencapsulation failed Other


No Yes Call support.
leaving an unbootable problem?

system?

No End

Yes

Boot disk
Reinstall Solaris
backup No End
Operating Environment
available?

Yes

Rebuild boot disk from

backups. Once restored, boot


Pre
from CDROM, disable boot
encapsulation No
disk encapsulation using
backup?
CDROM unencapsulation

procedure.

Yes

Restore boot disk from Disk Encapsulation Flowchart


End
backups
Rev-B 07/18/2002

Page 18 of 26

Sun Proprietary: Internal Use Only


Disk Encapsulation Processes B-19
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
S This flow has been selected because a process
executing in flow D was directed here.

No
The following procedure requires a reboot of the
system. Terminate all production applications
and logoff all non administrative users prior to
Current boot starting.
disk backup?

Yes

Bring system to boot


prom and boot to single
user mode from
CDROM.

Once booted, set TERM


type to vt100 so vi will
work.

fsck root filesystem.

Mount root partition


(slice 0) to /a

Once booted, set TERM


type to vt100 so vi will
work.

cp /a/etc/system /a/etc/system.orig
vi /a/etc/system
Edit the /etc/
system file and
Remove (do not comment) the following two lines:
remove " rootdev"
statements.
rootdev:/pseudo/vxio@0:0
set vxio:vol_rootdev_is_volume=1

Make a backup copy of


/a/etc/vfstab
Disk Encapsulation Flowchart

Rev-B 07/18/2002
T Page 20 of 26 Page 19 of 26

Sun Proprietary: Internal Use Only


B-20 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
If there are non system (/,

swap, /usr, /var) partitions on T This flow has been selected because a process

executing in flow S was directed here.


this disk, do not restore those

mount statements at this time.

These non system partitions

could include /opt, /home or

other named filesystem. Edit the /etc/vfstab file


Use either the comments at the end of the
These mount statements shall and restore to it's
file or the contents of the /etc/vfstab.prevm
be restored later in this pre boot disk
to restore the original mount statements.
process after their partitions encapsulation state.

have been recovered. It is


The /etc/vfstab.prevm file can be copied
acceptable to restore the
over the /etc/vfstab file providing there
mount statements at this time
have been no changes to this systems
if they are commented out so
storage configuration after the boot disk
they will not be executed
was encapsulated.
during the next system reboot.

touch /a/etc/vx/reconfig.d/state.d/ \
Prevent Volume

install-db
Manager from starting on

the next boot.

Remove boot disk rm /a/etc/vx/reconfig.d/state.d/ \


encapsulation flag. root-done

reboot the system

R Page 18 of 26

No

Reboot

successfull?

The system will reboot in a partially unencapsulated

state with /, swap, /usr and /var mounted.

Yes

rm /etc/vx/reconfig.d/state.d/ \
Manually start the install-db
Volume Manager vxiod set 10
software. vxconfigd -m disable
vxdctl enable

List all volumes

vxprint -qhtg rootdg


encapsulated on the boot

disk. Note these for use

later.

Disk Encapsulation Flowchart

U Page 21 of 26
Rev-B 07/18/2002

Page 20 of 26

Sun Proprietary: Internal Use Only


Disk Encapsulation Processes B-21
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
U This flow has been selected because a process

executing in flow S was directed here.

Remove the volumes

that existed on the /usr/sbin/vxedit -rf rm <volume name>


encapsulated disk.
Example

/usr/sbin/vxedit -rf rm rootvol


/usr/sbin/vxedit -rf rm swapvol
/usr/sbin/vxedit -rf rm usr
/usr/sbin/vxedit -rf rm var
/usr/sbin/vxedit -rf rm opt

/usr/sbin/vxdg -k rmdisk <disk name>


Remove the boot disk

from rootdg .

Example

Restore the boot disk's /usr/sbin/vxdg -k rmdisk rootdisk


vtoc to it's pre

encapsulation state. This

defines hard partitions

for the file systems /etc/vx/bin/vxedvtoc -f /etc/vx/reconfig.d/disk.d/ \


resident on the boot disk. <bootdisk c#t#d#>/vtoc /dev/rdsk/<bootdisk c#t#d#s2
Use the vxedvtoc
command and answer Y
Example

when prompted to write

new vtoc .
/etc/vx/bin/vxedvtoc -f /etc/vx/reconfig.d/disk.d/ \
c0t0d0/vtoc /dev/rdsk/c0t0d0s2

CAUTION: Be absolutely sure that the correct

c#t#d# addresses are used in this step or loss

of data could occur. Additionally, the boot disk


Edit the /etc/vfstab file
could be corrupted causing the system to
and restore the mount
become un-bootable.
statements for any non

system filesystems

resident on the boot disk.


Non system mount statements could include /opt / ,

home /export
, and others.

Start all volumes. /usr/sbin/vxedit -rf rm <volume name>

Mount all non system file

systems that may have

been resident on the

boot disk.
Disk Encapsulation Flowchart

Rev-B 07/18/2002
End
Page 21 of 26

Sun Proprietary: Internal Use Only


B-22 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
V This flow has been selected because a process
executing in flow D was directed here.

No
The following procedure requires a reboot of the
system. Terminate all production applications

Current boot and logoff all non administrative users prior to

disk backup? starting.

Yes

Bring system to boot


prom and boot to single
user mode from
CDROM.

Once booted, set TERM


type to vt100 so vi will
work.

fsck root filesystem.

Mount root partition


(slice 0) to /a

Once booted, set TERM


type to vt100 so vi will cp /a/etc/system /a/etc/system.orig
work. vi /a/etc/system

**rootdev:/pseudo/vxio@0:0
Edit the /etc/ **set vxio:vol_rootdev_is_volume=1
system file and
comment "rootdev"
statements. Do not skip this step. This copy is used to
restore the system back to it's original state
once system maintenance or troublrshooting
has been completed.

Make a backup copy of


/a/etc/vfstab cp /a/etc/vfstab /a/etc/vfstab.orig

Disk Encapsulation Flowchart

W Page 23 of 26 Rev-B 07/18/2002


Page 22 of 26

Sun Proprietary: Internal Use Only


Disk Encapsulation Processes B-23
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
W This flow has been selected because a process

executing in flow V was directed here.

/etc/vfsta
Edit the b file

/, swap, /
and restore

usr and /var


/etc/vfstab
to their pre
W hen completed, the file

/dev/vx
encapsulation physical
should have all devices commented

/ swap /usr /var


device mount statements.
out and , , and should be

mounting from their pre-encapsulation physical


Comment out all VxVM

(/dev/
devices.
managed volumes

vx... ). This includes /opt


and /export if they exist.

touch /a/etc/vx/reconfig.d/state.d/ \
Prevent VxVM software

install-db
from starting on the next

boot.

Remove boot disk rm /a/etc/vx/reconfig.d/state.d/ \


encapsulation flag. root-done

Reboot the system

The system is now in an The system is now ready for troubleshooting or

unencapsulated state maintenance.

with /, swap /usr


, and

/var m ounted. VxVM


All VxVM software devices are unavailable, this could

software daem ons are


include /opt /home /export
, , or other non-system file

stopped.
system resident on the boot disk. If these filesystems are

needed, a full unencapsulation of the boot is necessary.

No

Maintenance

finished?

touch /etc/vx/revconfig.d/state.d/root-done
Yes rm /etc/vx/reconfig.d/state.d/install-db
cp /etc/vfstab.orig /etc/vfstab
Restore the system to it's
cp /etc/system.orig /etc/system
encapsulated and VxVM
reboot
software enabled state.

Disk Encapsulation Flowchart

Rev-B 07/18/2002

End Page 23 of 26

Sun Proprietary: Internal Use Only


B-24 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
X This flow has been selected because a process
executing in flow B was directed here.

Recommended Guidelines:

1. Use a maximum of three slices - /, swap and /var (optional).


2. Never configure /usr as a separate slice.
3. Attach mirrors in geographical order not in alphabetical order. vxdiskadm builds and
attaches mirrors in alphabetical order. Do not use vxdiskadm to mirror the boot disk if
slices other than /, swap and /var are present.
4. Replace the encapsulated boot disk with an initialized disk. Replacing the
encapsulated disk with an initialized disk insures that the boot disk and the mirror disk
are identical. This reduces the complexity of the mirrored boot disk configuration.
5. Map the core system volumes to slices/partitions. This should not be necessary if /,
swap and /var are the only slices on the boot disk and you are using VxVM 3.x or
higher.
6. Ensure that all boot disk mirrors are bootable and have a devalias built in the
OpenBoot Prom.
7. Create a clone disk. This disk must be able to be booted from slices (not under VxVM
control, non encapsulated copy of the boot disk) and be able to run VxVM software
utilities (VxVM installed). This disk is used if there is a complete failure of the VxVM
managed boot disk making the system un-bootable.

Save the boot


# mkdir /var/tmp/sysdocs
disk vtoc for
# prtvtoc /dev/dsk/c#t#d#s2 > \
reference if
/var/tmp/sysdocs/rootdisk_prevm.vtoc
needed.

Encapsulate the boot


disk using
vxdiskadm or
vxinstall

Initialize and add the


disk to be used as the # /etc/vx/bin/vxdisksetup -i c#t#d#
rootmirror for # vxdg -g rootdg adddisk rootmirror=c#t#d#
rootdg.

Y Page 25 of 26

Disk Encapsulation Flowchart

Rev-B 07/18/2002
Page 24 of 26

Sun Proprietary: Internal Use Only


Disk Encapsulation Processes B-25
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Y This flow has been selected because a process

executing in flow X was directed here.

# /etc/vx/bin/vxrootmir rootmir
Attach the mirrors in

# vxassist -g rootdg mirror swapvol rootmirror&


the geographic order

# vxassist -g rootdg mirror var rootmirror&


of the encapsulated

disk.

Note: If the original boot disk was configured only using /, swap
and /var slices; mirroring using vxdiskadm is acceptable.

Note: The following hand loop can be executed to display the


progress of the mirroring process:

while true
> do
> vxtask list
> sleep 15
> echo "#####################"
> done
W ait for the mirroring

process to complete

No

Mirroring

Process

Complete?

Yes

vxplex -g rootdg dis rootvol-01


Dissociate the vxples -g rootdg dis swapvol-01
rootdisk plexes vxplex -g rootdg dis var-01
vxedit -fr rm rootvol-01 swapvol-01 var-01

Z
Page 26 of 26

Disk Encapsulation Flowchart

Rev-B 07/18/2002

Page 25 of 26

Sun Proprietary: Internal Use Only


B-26 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Z This flow has been selected because a process
executing in flow Y was directed here.

Remove the
rootdisk from vxdg -g rootdg rootdisk
rootdg

Initialize the
/etc/vx/bin/vxdisksetup -i c#t#d#
rootdisk and add
vxdg -g rootdg adddisk rootdisk=c#t#d#
it back to rootdg

Attach the mirrors in


/etc/vx/bin/cxrootmir rootmir
the geographic order
vxassist -g rootdg mirror swapvol rootmirror&
of the encapsulated
vxassist -g rootdg mirror var rootmirror&
disk.

If needed, create the


# /usr/vx/bin/vxmksdpart -g rootdg rootdisk-03 \
underlying system
6 0x07 0x00
partitions using
(This example creates the overlay partition for /var if configured.)
vxmksdpart.
Note: If /, swap and /var are the only slices configured on the
boot disk, vxdiskadm can be used to create the new mirror and
vxdiskadm will build all the overlay partitions making this step
unnecessary.

Note: Runvxprint -qhtg rootdg prior to executing the


vxmksdpart command to get the subdisk names for the overlay
partitions needing to be built.

Set the dump device


to a non VM disk if # dumpadm -d /dev/dsk/c#t#d#s#
available.

Create the OBP


device aliases if
necessary.

Disk Encapsulation Flowchart


End
Rev-B 07/18/2002
Page 26 of 26

Sun Proprietary: Internal Use Only


Disk Encapsulation Processes B-27
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Appendix C

Example Five-Slice Boot Disk Encapsulation

Encapsulation of a boot disk that has only the /, swap, /usr, and /var
partitions is a straightforward process. The encapsulation process creates
the public and private regions needed by the VxVM software and an
overlap partition for each of the system partitions needed to boot the
system. Unencapsulation is also a straightforward process as well, as long
as the vxunroot utility is successful.

When a boot disk has other slices in addition to /, swap, /usr, and /var,
encapsulation is more complex. The main system partitions (/, swap,
/usr, and /var) are still visible, but the additional partitions, such as /opr
or /home, disappear. A similar result occurs during the data disk
encapsulation process. The data is not moved or overwritten. It is still
accessible by the VxVM software and mounted as a VxVM software
volume.

Sun Proprietary: Internal Use Only


C-1
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Figure C-1 illustrates an example partitioning scheme of a complex, five-
slice boot disk.

Boot Disk Encapsulated Data Disk

rootvol
swapvol
Slice 0 - / Overlay Partition - / (0)
Slice 1 - swap Slice 2 and
Overlay Partition - swap (1)
Slice 6
Overlay Partition - /usr (3) (Public Region)
Slice 2 Slice 3 - /usr
Overlaps
Slice 4 - /var Overlay Partition - /var (4) Full Disk.
/opt is
Encapsulated
Slice 5 - /opt in Public Region
Slice7
Free Space Private Region

Mirror Disk

Preserved
Slice 3 Private Region
rootvol /opt
Overlay Partition - /opt (5)
swapvol
Overlay Partition - / (0)
Slice 2
Slice 4
Overlay Partition - swap (1)
(Public Region)
Overlay Partition - /usr (6)

Overlay Partition - /var (7)

Figure C-1 Five-Slice Boot Disk Encapsulation

Sun Proprietary: Internal Use Only


C-2 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Pre-Encapsulation Configuration Data

Pre-Encapsulation Configuration Data


The following are example configuration data from a five-slice boot disk
prior to encapsulation.

Pre-Encapsulation prtvtoc Command


Output from the following example of the prtvtoc command clearly
shows five slices and sufficient free space for encapsulation.
bash-2.03# prtvtoc /dev/rdsk/c1t2d0s2
* /dev/rdsk/c1t2d0s2 partition map
*
* Dimensions:
* 512 bytes/sector
* 133 sectors/track
* 27 tracks/cylinder
* 3591 sectors/cylinder
* 4926 cylinders
* 4924 accessible cylinders
*
* Flags:
* 1: unmountable
* 10: read-only
*
* Unallocated space:
* First Sector Last
* Sector Count Sector
* 16795107 886977 17682083
*
* First Sector Last
* Partition Tag Flags Sector Count Sector Mount Directory
0 2 00 0 2100735 2100734 /
1 3 01 2100735 2100735 4201469
2 5 00 0 17682084 17682083
3 4 00 4201470 4197879 8399348 /usr
4 7 00 8399349 4197879 12597227 /var
5 0 00 12597228 4197879 16795106 /opt

Sun Proprietary: Internal Use Only


Example Five-Slice Boot Disk Encapsulation C-3
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Pre-Encapsulation Configuration Data

Pre-Encapsulation format Print Utility


A description of the example disk from the format utility delineates each
partition on a cylinder boundary. Notice that there are two free partitions
for use by the encapsulation process to assign to the public and private
regions.
partition> print
Current partition table (original):
Total disk cylinders available: 4924 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks


0 root wm 0 - 584 1.00GB (585/0/0) 2100735
1 swap wu 585 - 1169 1.00GB (585/0/0) 2100735
2 backup wm 0 - 4923 8.43GB (4924/0/0) 17682084
3 usr wm 1170 - 2338 2.00GB (1169/0/0) 4197879
4 var wm 2339 - 3507 2.00GB (1169/0/0) 4197879
5 unassigned wm 3508 - 4676 2.00GB (1169/0/0) 4197879
6 unassigned wm 0 0 (0/0/0) 0
7 unassigned wm 0 0 (0/0/0) 0

Pre-Encapsulation df -k Command
Example output from the df -k command delineates the physical devices
supporting currently mounted file systems.
bash-2.03# df -k
Filesystem kbytes used avail capacity Mounted on
/dev/dsk/c1t2d0s0 1018382 47206 910074 5% /
/dev/dsk/c1t2d0s3 2055705 772430 1221604 39% /usr
/proc 0 0 0 0% /proc
fd 0 0 0 0% /dev/fd
mnttab 0 0 0 0% /etc/mnttab
/dev/dsk/c1t2d0s4 2055705 956248 1037786 48% /var
swap 1222104 16 1222088 1% /var/run
swap 1222104 16 1222088 1% /tmp
/dev/dsk/c1t2d0s5 2055705 2133 1991901 1% /opt

Pre-Encapsulation swap -l Command


Output from the swap -l command for this example shows that slice one
of the boot disk is used as a swap device.
bash-2.03# swap -l
swapfile dev swaplo blocks free
/dev/dsk/c1t2d0s1 118,145 16 2100704 2100704

Sun Proprietary: Internal Use Only


C-4 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Post-Encapsulation Configuration Data

Post-Encapsulation Configuration Data


The following output was captured after encapsulation of a five-slice boot
disk.

Post-Encapsulation prtvtoc /dev/rdsk/c1t2d0s2


Command
The following example displays post-encapsulation partitioning for the
boot disk. The /, /usr, /var, and swap partitions inhabit their original
locations on the boot disk and are represented as overlay partitions.
Partition 5, which holds the /opt file system, is encapsulated into the
public region (slice 6) and does not have an overlay partition assigned.
Since /opt is not needed to boot the system, it is encapsulated just like
any other data partition.
bash-2.03# prtvtoc /dev/rdsk/c1t2d0s2
* /dev/rdsk/c1t2d0s2 partition map
*
* Dimensions:
* 512 bytes/sector
* 133 sectors/track
* 27 tracks/cylinder
* 3591 sectors/cylinder
* 4926 cylinders
* 4924 accessible cylinders
*
* Flags:
* 1: unmountable
* 10: read-only
*
* Unallocated space:
* First Sector Last
* Sector Count Sector
* 2100735 4292866561 4294967295
* 17682084 4279385947 2100734
* 12597228 5081265 17678492
*
* First Sector Last
* Partition Tag Flags Sector Count Sector Mount Directory
0 2 00 0 2100735 2100734
1 3 01 2100735 2100735 4201469
2 5 00 0 17682084 17682083
3 4 00 4201470 4197879 8399348
4 7 00 8399349 4197879 12597227
6 14 01 0 17682084 17682083
7 15 01 17678493 3591 17682083

Sun Proprietary: Internal Use Only


Example Five-Slice Boot Disk Encapsulation C-5
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Post-Encapsulation Configuration Data

Post-Encapsulation format print Utility


An example of output from the format utility shows the partitioning
scheme on cylinder boundaries and mount points for the overlay
partitions. Notice that partition 5, /opt, is unassigned with a zero block
size. It was encapsulated into the disk’s public region.
partition> print
Current partition table (original):
Total disk cylinders available: 4924 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks


0 root wm 0 - 584 1.00GB (585/0/0) 2100735
1 swap wu 585 - 1169 1.00GB (585/0/0) 2100735
2 backup wm 0 - 4923 8.43GB (4924/0/0) 17682084
3 usr wm 1170 - 2338 2.00GB (1169/0/0) 4197879
4 var wm 2339 - 3507 2.00GB (1169/0/0) 4197879
5 unassigned wm 0 0 (0/0/0) 0
6 - wu 0 - 4923 8.43GB (4924/0/0) 17682084
7 - wu 4923 - 4923 1.75MB (1/0/0) 3591

Post-Encapsulation df -k Command
The following example of output from the df -k command shows that
the VxVM software volumes are used for the file systems provided by the
boot disk.
bash-2.03# df -k
Filesystem kbytes used avail capacity Mounted on
/dev/vx/dsk/rootvol 1018382 75398 881882 8% /
/dev/vx/dsk/usr 2055705 805992 1188042 41% /usr
/proc 0 0 0 0% /proc
fd 0 0 0 0% /dev/fd
mnttab 0 0 0 0% /etc/mnttab
/dev/vx/dsk/var 2055705 974276 1019758 49% /var
swap 1180368 16 1180352 1% /var/run
swap 1180424 72 1180352 1% /tmp
/dev/vx/dsk/opt 2055705 53686 1940348 3% /opt

Post-Encapsulation vxprint Command


An example of the vxprint output displays all the relevant VxVM
software information about this encapsulated boot disk, its volumes, and
its mirrors. Information from this command is important if this disk needs
to be manually unencapsulated.

Sun Proprietary: Internal Use Only


C-6 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Post-Encapsulation Configuration Data

bash-2.03# vxprint -qhtg rootdg


dg rootdg default default 0 1020469848.1025.lowtide5

dm rootdisk c1t2d0s2 sliced 3590 17678493 -


dm rootmirror c1t20d0s2 sliced 3590 17678493 -

v opt - ENABLED ACTIVE 4197879 ROUND - gen


pl opt-01 opt ENABLED ACTIVE 4197879 CONCAT - RW
sd rootdisk-03 opt-01 rootdisk 12597227 4197879 0 c1t2d0 ENA
pl opt-02 opt ENABLED ACTIVE 4197879 CONCAT - RW
sd rootmirror-01 opt-02 rootmirror 0 4197879 0 c1t20d0 ENA

v rootvol - ENABLED ACTIVE 2100735 ROUND - root


pl rootvol-01 rootvol ENABLED ACTIVE 2100735 CONCAT - RW
sd rootdisk-B0 rootvol-01 rootdisk 17678492 1 0 c1t2d0 ENA
sd rootdisk-02 rootvol-01 rootdisk 0 2100734 1 c1t2d0 ENA
pl rootvol-02 rootvol ENABLED ACTIVE 2100735 CONCAT - RW
sd rootmirror-02 rootvol-02 rootmirror 4197879 2100735 0 c1t20d0 ENA

v swapvol - ENABLED ACTIVE 2100735 ROUND - swap


pl swapvol-01 swapvol ENABLED ACTIVE 2100735 CONCAT - RW
sd rootdisk-01 swapvol-01 rootdisk 2100734 2100735 0 c1t2d0 ENA
pl swapvol-02 swapvol ENABLED ACTIVE 2100735 CONCAT - RW
sd rootmirror-03 swapvol-02 rootmirror 6298614 2100735 0 c1t20d0 ENA

v usr - ENABLED ACTIVE 4197879 ROUND - gen


pl usr-01 usr ENABLED ACTIVE 4197879 CONCAT - RW
sd rootdisk-05 usr-01 rootdisk 4201469 4197879 0 c1t2d0 ENA
pl usr-02 usr ENABLED ACTIVE 4197879 CONCAT - RW
sd rootmirror-04 usr-02 rootmirror 8399349 4197879 0 c1t20d0 ENA

v var - ENABLED ACTIVE 4197879 ROUND - gen


pl var-01 var ENABLED ACTIVE 4197879 CONCAT - RW
sd rootdisk-04 var-01 rootdisk 8399348 4197879 0 c1t2d0 ENA
pl var-02 var ENABLED ACTIVE 4197879 CONCAT - RW
sd rootmirror-05 var-02 rootmirror 12597228 4197879 0 c1t20d0 ENA

Post-Encapsulation /etc/vfstab File


The /etc/vfstab file in the following example was modified to reflect
the encapsulated volumes. A copy of the original was saved to the
/etc/vfstab.prevm directory.
bash-2.03# more /etc/vfstab
#device device mount FS fsck mount mount
#to mount to fsck point type pass at boot options
#
#/dev/dsk/c1d0s2 /dev/rdsk/c1d0s2 /usr ufs 1 yes -
fd - /dev/fd fd - no -
/proc - /proc proc - no -
/dev/vx/dsk/swapvol - - swap - no -

Sun Proprietary: Internal Use Only


Example Five-Slice Boot Disk Encapsulation C-7
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Post-Encapsulation Configuration Data

/dev/vx/dsk/rootvol /dev/vx/rdsk/rootvol / ufs 1 no -


/dev/vx/dsk/usr /dev/vx/rdsk/usr /usr ufs 1 no -
/dev/vx/dsk/var /dev/vx/rdsk/var /var ufs 1 no -
/dev/vx/dsk/opt /dev/vx/rdsk/opt /opt ufs 2 yes -
swap - /tmp tmpfs - yes -
#NOTE: volume rootvol (/) encapsulated partition c1t2d0s0
#NOTE: volume swapvol (swap) encapsulated partition c1t2d0s1
#NOTE: volume usr (/usr) encapsulated partition c1t2d0s3
#NOTE: volume var (/var) encapsulated partition c1t2d0s4
#NOTE: volume opt (/opt) encapsulated partition c1t2d0s5

Post-Encapsulation Mirror Disk prtvtoc Command


The partitioning scheme in the following example of the mirror disk for a
five-slice boot disk is very different from that of a two-slice boot disk.
Notice that there is a relocation of overlay partitions. Partition 5, which is
/opt, was moved to the front of the disk after the private region. This
preserves the contents of that encapsulated slice and allows recovery of
that partition’s data if the original disk fails and is replaced.
bash-2.03# prtvtoc /dev/rdsk/c1t20d0s2
* /dev/rdsk/c1t20d0s2 partition map
*
* Dimensions:
* 512 bytes/sector
* 133 sectors/track
* 27 tracks/cylinder
* 3591 sectors/cylinder
* 4926 cylinders
* 4924 accessible cylinders
*
* Flags:
* 1: unmountable
* 10: read-only
*
* Unallocated space:
* First Sector Last
* Sector Count Sector
* 17682084 4277288803 3590
* 16798698 883386 17682083
*
* First Sector Last
* Partition Tag Flags Sector Count Sector Mount Directory
0 2 00 4201470 2100735 6302204 <-- /
1 3 01 6302205 2100735 8402939 <-- swap
2 5 01 0 17682084 17682083 <-- overlap/backup
3 15 01 0 3591 3590 <-- Private
4 14 01 3591 17678493 17682083 <-- Public
5 0 00 3591 4197879 4201469 <-- /opt
6 4 00 8402940 4197879 12600818 <-- /usr
7 7 00 12600819 4197879 16798697 <-- /var

Sun Proprietary: Internal Use Only


C-8 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Post-Encapsulation Configuration Data

Post-Encapsulation Mirror Disk format Print Utility


The following example of output from the format utility displays the
partitioning scheme for this disk on cylinder boundaries.
partition> print
Current partition table (original):
Total disk cylinders available: 4924 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks


0 root wm 1170 - 1754 1.00GB (585/0/0) 2100735
1 swap wu 1755 - 2339 1.00GB (585/0/0) 2100735
2 backup wu 0 - 4923 8.43GB (4924/0/0) 17682084
3 - wu 0 - 0 1.75MB (1/0/0) 3591
4 - wu 1 - 4923 8.43GB (4923/0/0) 17678493
5 unassigned wm 1 - 1169 2.00GB (1169/0/0) 4197879 <-- /opt
6 usr wm 2340 - 3508 2.00GB (1169/0/0) 4197879
7 var wm 3509 - 4677 2.00GB (1169/0/0) 4197879

Sun Proprietary: Internal Use Only


Example Five-Slice Boot Disk Encapsulation C-9
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Example Five-Slice Boot Disk Unencapsulation

Example Five-Slice Boot Disk Unencapsulation


Unencapsulating a five-slice boot disk is similar to unencapsulating a two-
or four-slice disk. The most direct and easiest method is to use the
vxunroot command. If that fails, use one of the three remaining manual
methods to successfully complete the unencapsulation.

Unencapsulating Using the vxunroot Command


The unencapsulation process for a five-slice disk is identical to the
vxunroot procedure described in ‘‘Unencapsulating a Boot Disk Using
the vxunroot Utility’’ on page 2-57.

Manually Unencapsulating
The manual unencapsulation process for a five-slice disk is similar to the
manual procedure described in ‘‘Manually Unencapsulating a Boot Disk’’
on page 2-60. The one difference in the process of unencapsulating a five-
slice boot disk is the recovery of the /opt partition; this procedure is
covered as part of the manual unencapsulation procedure.

Additionally, the procedures in ‘‘Unencapsulating When Booted From the


CD-ROM’’ on page 2-64, and ‘‘Performing a Basic or Functional
Unencapsulation’’ on page 2-68 are similar for a five-slice boot disk.
Execute these procedures as written in the referenced sections. Be sure to
recover the /opt partition using the instructions for one of the following
commands:
● The vxmksdpart command – See step 6.c on page 2-62.
● The vxedvtoc command – See step 19.b on page 2-67.

Sun Proprietary: Internal Use Only


C-10 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Appendix D

The Boot Process

This appendix contains an annotated boot process code list that identifies
when each of the VxVM software start-up scripts execute during the
system boot process.
{0} ok boot -v
Rebooting with command: boot -v
Boot device: /sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203713fc9f,0:a File and args: -
v
Size: 339096+90034+75930 Bytes
SunOS Release 5.8 Version Generic_108528-14 64-bit
Copyright 1983-2001 Sun Microsystems, Inc. All rights reserved.
Ethernet address = 8:0:20:7d:5d:60
mem = 262144K (0x10000000)
avail mem = 232955904
root nexus = 8-slot Sun Enterprise 4000/5000
sbus0 at root: UPA 0x2 0x0 ...
sbus0 is /sbus@2,0
sbus1 at root: UPA 0x3 0x0 ...
sbus1 is /sbus@3,0
socal0 at sbus1: SBus1 slot 0x0 offset 0x0 and slot 0x0 offset 0x10000 and slot 0x0
offset 0x20000 SBus level 3 sparc9 ipl 5
socal0 is /sbus@3,0/SUNW,socal@0,0
sf0 at socal0: socal_port 0
sf0 is /sbus@3,0/SUNW,socal@0,0/sf@0,0
sf1 at socal0: socal_port 1
sf1 is /sbus@3,0/SUNW,socal@0,0/sf@1,0
soc0 at sbus0: SBus0 slot 0xd offset 0x10000 Onboard device sparc9 ipl 5
soc0 is /sbus@2,0/SUNW,soc@d,10000
ssd0 at sf1: name w210000203713f582,0, bus address c7
ssd0 is /sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w210000203713f582,0
<SUN9.0G cyl 4924 alt 2 hd 27 sec 133>
/sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w210000203713f582,0 (ssd0) online
ssd2 at sf1: name w210000203713e0b9,0, bus address e2
ssd2 is /sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w210000203713e0b9,0
<SUN9.0G cyl 4924 alt 2 hd 27 sec 133>
/sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w210000203713e0b9,0 (ssd2) online
ssd1 at sf1: name w210000203713f643,0, bus address cc
ssd1 is /sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w210000203713f643,0
<SUN9.0G cyl 4924 alt 2 hd 27 sec 133>
/sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w210000203713f643,0 (ssd1) online

Sun Proprietary: Internal Use Only


D-1
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
ssd3 at sf1: name w210000203713f565,0, bus address e1
ssd3 is /sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w210000203713f565,0
<SUN9.0G cyl 4924 alt 2 hd 27 sec 133>
/sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w210000203713f565,0 (ssd3) online
ssd4 at sf1: name w210000203713fc9f,0, bus address ef
ssd4 is /sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w210000203713fc9f,0
<SUN9.0G cyl 4924 alt 2 hd 27 sec 133>
/sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w210000203713fc9f,0 (ssd4) online
ssd5 at sf1: name w210000203713df01,0, bus address dc
ssd5 is /sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w210000203713df01,0
<SUN9.0G cyl 4924 alt 2 hd 27 sec 133>
/sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w210000203713df01,0 (ssd5) online
ssd6 at sf1: name w210000203713f96d,0, bus address e8
ssd6 is /sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w210000203713f96d,0
<SUN9.0G cyl 4924 alt 2 hd 27 sec 133>
/sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w210000203713f96d,0 (ssd6) online
ssd7 at sf1: name w210000203713f7f0,0, bus address cb
ssd7 is /sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w210000203713f7f0,0
<SUN9.0G cyl 4924 alt 2 hd 27 sec 133>
/sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w210000203713f7f0,0 (ssd7) online
ssd8 at sf1: name w21000020372d5917,0, bus address c9
ssd8 is /sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w21000020372d5917,0
<SUN9.0G cyl 4924 alt 2 hd 27 sec 133>
/sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w21000020372d5917,0 (ssd8) online
ssd9 at sf1: name w210000203713fc0f,0, bus address c6
ssd9 is /sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w210000203713fc0f,0
<SUN9.0G cyl 4924 alt 2 hd 27 sec 133>
/sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w210000203713fc0f,0 (ssd9) online
ssd10 at sf1: name w2100002037140269,0, bus address cd
ssd10 is /sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w2100002037140269,0
<SUN9.0G cyl 4924 alt 2 hd 27 sec 133>
/sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w2100002037140269,0 (ssd10) online
ssd11 at sf1: name w210000203713f579,0, bus address e4
ssd11 is /sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w210000203713f579,0
<SUN9.0G cyl 4924 alt 2 hd 27 sec 133>
/sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w210000203713f579,0 (ssd11) online
ssd12 at sf1: name w210000203716cf3d,0, bus address e0
ssd12 is /sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w210000203716cf3d,0
<SUN9.0G cyl 4924 alt 2 hd 27 sec 133>
/sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w210000203716cf3d,0 (ssd12) online
ssd13 at sf1: name w210000203713f49f,0, bus address ca
ssd13 is /sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w210000203713f49f,0
<SUN9.0G cyl 4924 alt 2 hd 27 sec 133>
/sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w210000203713f49f,0 (ssd13) online
ssd14 at sf0: name w220000203713f582,0, bus address c7
ssd14 is /sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203713f582,0
<SUN9.0G cyl 4924 alt 2 hd 27 sec 133>
/sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203713f582,0 (ssd14) online
ssd15 at sf0: name w220000203713f643,0, bus address cc
ssd15 is /sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203713f643,0
<SUN9.0G cyl 4924 alt 2 hd 27 sec 133>
/sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203713f643,0 (ssd15) online
ssd16 at sf0: name w220000203713e0b9,0, bus address e2
ssd16 is /sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203713e0b9,0

Sun Proprietary: Internal Use Only


D-2 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
<SUN9.0G cyl 4924 alt 2 hd 27 sec 133>
/sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203713e0b9,0 (ssd16) online
ssd17 at sf0: name w220000203713f565,0, bus address e1
ssd17 is /sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203713f565,0
<SUN9.0G cyl 4924 alt 2 hd 27 sec 133>
/sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203713f565,0 (ssd17) online
ssd18 at sf0: name w220000203713fc9f,0, bus address ef
ssd18 is /sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203713fc9f,0
<SUN9.0G cyl 4924 alt 2 hd 27 sec 133>
/sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203713fc9f,0 (ssd18) online
ssd19 at sf0: name w220000203713df01,0, bus address dc
ssd19 is /sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203713df01,0
<SUN9.0G cyl 4924 alt 2 hd 27 sec 133>
/sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203713df01,0 (ssd19) online
ssd20 at sf0: name w220000203713f96d,0, bus address e8
ssd20 is /sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203713f96d,0
<SUN9.0G cyl 4924 alt 2 hd 27 sec 133>
/sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203713f96d,0 (ssd20) online
ssd21 at sf0: name w220000203713f7f0,0, bus address cb
ssd21 is /sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203713f7f0,0
<SUN9.0G cyl 4924 alt 2 hd 27 sec 133>
/sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203713f7f0,0 (ssd21) online
ssd22 at sf0: name w22000020372d5917,0, bus address c9
ssd22 is /sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w22000020372d5917,0
<SUN9.0G cyl 4924 alt 2 hd 27 sec 133>
/sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w22000020372d5917,0 (ssd22) online
ssd23 at sf0: name w220000203713fc0f,0, bus address c6
ssd23 is /sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203713fc0f,0
<SUN9.0G cyl 4924 alt 2 hd 27 sec 133>
/sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203713fc0f,0 (ssd23) online
ssd24 at sf0: name w2200002037140269,0, bus address cd
ssd24 is /sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w2200002037140269,0
<SUN9.0G cyl 4924 alt 2 hd 27 sec 133>
/sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w2200002037140269,0 (ssd24) online
ssd25 at sf0: name w220000203713f579,0, bus address e4
ssd25 is /sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203713f579,0
<SUN9.0G cyl 4924 alt 2 hd 27 sec 133>
/sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203713f579,0 (ssd25) online
ssd26 at sf0: name w220000203716cf3d,0, bus address e0
ssd26 is /sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203716cf3d,0
<SUN9.0G cyl 4924 alt 2 hd 27 sec 133>
/sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203716cf3d,0 (ssd26) online
ssd27 at sf0: name w220000203713f49f,0, bus address ca
ssd27 is /sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203713f49f,0
<SUN9.0G cyl 4924 alt 2 hd 27 sec 133>
/sbus@3,0/SUNW,socal@0,0/sf@0,0/ssd@w220000203713f49f,0 (ssd27) online
/sbus@3,0/SUNW,fas@3,8800000 (fas0):
rev 2.2 FEPS chip
fas0 at sbus1: SBus1 slot 0x3 offset 0x8800000 and slot 0x3 offset 0x8810000 SBus level
3 sparc9 ipl 5
fas0 is /sbus@3,0/SUNW,fas@3,8800000
sd6 at fas0: target 6 lun 0
sd6 is /sbus@3,0/SUNW,fas@3,8800000/sd@6,0
root on /pseudo/vxio@0:0 fstype ufs

Sun Proprietary: Internal Use Only


The Boot Process D-3
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
WARNING: forceload of drv/SUNW,socal failed
fhc0 at root: UPA 0x0 0xf8800000
fhc0 is /fhc@0,f8800000
ac0 board 0 bank 0: base 0x0 size 256mb rstate 2 ostate 1 condition 1
ac0 is /fhc@0,f8800000/ac@0,1000000
fhc1 at root: UPA 0x2 0xf8800000
fhc1 is /fhc@2,f8800000
ac1 is /fhc@2,f8800000/ac@0,1000000
central0 at root: UPA 0x1f 0x0 ...
fhc2 at root: UPA 0x0 0xf8800000
fhc2 is /central@1f,0/fhc@0,f8800000
WARNING: Core Power Supply 3 Failing
sysctrl0 is /central@1f,0/fhc@0,f8800000/clock-board@0,900000
sysctrl0: Key switch is not in the secure position
environ0 is /fhc@0,f8800000/environment@0,400000
environ1 is /fhc@2,f8800000/environment@0,400000
simmstat0 is /fhc@0,f8800000/simm-status@0,600000
sram0 is /fhc@0,f8800000/sram@0,200000
zs0 is /central@1f,0/fhc@0,f8800000/zs@0,902000
zs1 is /central@1f,0/fhc@0,f8800000/zs@0,904000
cpu0: SUNW,UltraSPARC (upaid 0 impl 0x10 ver 0x40 clock 168 MHz)
cpu1: SUNW,UltraSPARC (upaid 1 impl 0x10 ver 0x40 clock 168 MHz)

Starting /etc/rcS.d/S25vxvm-sysboot script

Starting VxVM restore daemon...


VxVM starting in boot mode...
ses16 at sf1: name w5080020000034909,0, bus address d2
ses16 is /sbus@3,0/SUNW,socal@0,0/sf@1,0/ses@w5080020000034909,0
ses17 at sf1: name w508002000003490a,0, bus address b5
ses17 is /sbus@3,0/SUNW,socal@0,0/sf@1,0/ses@w508002000003490a,0
ses18 at sf0: name w508002000003490b,0, bus address d2
ses18 is /sbus@3,0/SUNW,socal@0,0/sf@0,0/ses@w508002000003490b,0
ses19 at sf0: name w508002000003490c,0, bus address b5
ses19 is /sbus@3,0/SUNW,socal@0,0/sf@0,0/ses@w508002000003490c,0
NOTICE: vxvm:vxdmp: enabled path 118/0x20 belonging to the dmpnode 68/0x0

NOTICE: vxvm:vxio: Cannot open disk c1t3d0s2: kernel error 6


NOTICE: vxvm:vxio: Cannot open disk c1t3d0s2: kernel error 6
SUNW,hme0 : Sbus (Rev Id = 22) Found
hme0 at sbus0: SBus0 slot 0x1 offset 0x8c00000 and slot 0x1 offset 0x8c02000 and slot
0x1 offset 0x8c04000 and slot 0x1 offset 0x8c06000 and slot 0x1 offset 0x8c07000 SBus
level 4 sparc9 ipl 7
hme0 is /sbus@2,0/SUNW,hme@1,8c00000
SUNW,hme1 : Sbus (Rev Id = 22) Found
hme1 at sbus1: SBus1 slot 0x3 offset 0x8c00000 and slot 0x3 offset 0x8c02000 and slot
0x3 offset 0x8c04000 and slot 0x3 offset 0x8c06000 and slot 0x3 offset 0x8c07000 SBus
level 4 sparc9 ipl 7
hme1 is /sbus@3,0/SUNW,hme@3,8c00000
configuring IPv4 interfaces: hme0.
configuring IPv6 interfaces: hme0.
Hostname: lowtide

Executing /etc/rcS.d/S35vxvm-startup1 Script

Sun Proprietary: Internal Use Only


D-4 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
VxVM starting special volumes ( swapvol var )...

Executing /etc/rcS.d/standardmounts Script

SUNW,hme0 : Internal Transceiver Selected.


SUNW,hme0 : 100 Mbps Full-Duplex Link Up

Executing /etc/rcS.d/S50devfsadm Script

Executing /etc/rcS.d/S85vxvm-startup2 Script

pseudo-device: devinfo0
devinfo0 is /pseudo/devinfo@0
VxVM general startup...
NOTICE: vxvm:vxio: Cannot open disk c1t3d0s2: kernel error 6
dump on /dev/dsk/c1t0d0s1 size 513 MB
NOTICE: vxvm:vxio: Cannot open disk c1t3d0s2: kernel error 6

Executing /etc/rcS.d/S86vxvm-reconfig Script

The system is coming up. Please wait.

Executing /etc/rc2.d/S20sysetup Script

Starting IPv6 neighbor discovery.


Setting default IPv6 interface for multicast: add net ff00::/8: gateway
fe80::a00:20ff:fe7d:5d60
starting rpc services: rpcbind done.
Setting netmask of hme0 to 255.255.255.0
Setting default IPv4 interface for multicast: add net 224.0/4: gateway lowtide
syslog service starting.
Print services started.
Jun 24 14:39:05 lowtide pseudo: pseudo-device: tod0

Jun 24 14:39:05 lowtide genunix: tod0 is /pseudo/tod@0

Jun 24 14:39:05 lowtide pseudo: pseudo-device: pm0

Jun 24 14:39:05 lowtide genunix: pm0 is /pseudo/pm@0

volume management starting.

Executing /etc/rc2.d/S94vxnm-host_infod Script

Executing /etc/rc2.d/S94vxnm-vxnetd Script

Jun 24 14:39:07 lowtide pseudo: pseudo-device: vol0

Jun 24 14:39:07 lowtide genunix: vol0 is /pseudo/vol@0

Executing /etc/rc2.d/S95vxvm-recover Script

Executing /etc/rc2.d/S96vmsa-server Script

Sun Proprietary: Internal Use Only


The Boot Process D-5
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Starting RMI Registry
Starting VERITAS VM Storage Administrator Command Server
Starting VERITAS VM Storage Administrator Server
The system is ready.

lowtide console login:

Sun Proprietary: Internal Use Only


D-6 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Appendix E

Configuring the VxVM Software

Objectives
This appendix contains information related to VxVM software device
configuration tasks.

Sun Proprietary: Internal Use Only


E-1
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Relevance

Relevance

Discussion – Modules in this course discussed the basic elements and


architecture of the VxVM software. This appendix provides detailed
! information on how to build and assess VxVM software configurations in
?
an environment, and how to determine the effectiveness of those
configurations.

The following questions are relevant to the material presented in this


appendix:
● What are important factors to look at when formulating a VxVM
software solution?
● Can you identify the standards for RAID installations?

Sun Proprietary: Internal Use Only


E-2 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Additional Resources

Additional Resources

Additional resources – The following references provide additional


details on the topics discussed in this module:
● VERITAS Volume Manager™ 3.2 Administrator’s Guide. Mountain
View, California: VERITAS Software Corporation, August 2001,
number 30-000392-011, TechPDF ID 240253.
● VERITAS Volume Manager™ 3.2 SRT Instructor Guide. Mountain View,
California: VERITAS Software Corporation.
● VERITAS Volume Manager™ 3.2 Release Notes. Mountain View,
California: VERITAS Software Corporation, August 2001,
number 30-0003962-011, TechPDF ID 240252.
● SunSolveSM Online INFODOC 40143,
[http://sunsolve.Sun.COM/pub-cgi/search.pl?mode=advanced].
● Man pages for the vxassist and vxdg commands.

Sun Proprietary: Internal Use Only


Configuring the VxVM Software E-3
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Supported RAID Levels

Supported RAID Levels


The VxVM software supports the following levels of RAID:
● RAID 0 – Uses striping
● RAID 1 – Uses mirroring
● RAID 0 + RAID 1 – Uses striping and mirroring
● RAID 1 + RAID 0 – Uses mirroring and striping—referred to as
RAID 10
● RAID 5 – Uses parity striping

This module describes the differences among these RAID configurations.

Layered Volumes
Prior to version 3.2, the VxVM software applied default rules to assign
disks using the vxassist command. The default rules essentially created
the RAID stripe and then the mirror. The VxVM software version 3.0
introduced layered volumes, allowing for RAID 1 + RAID 0 (mirror and
striping) configurations. This feature is now fully implemented in VxVM
software version 3.2 through the vxassist command -o ordered option.

Ordered Allocation of Storage


When the -o ordered option of the vxassist command is used, the
default configuration function creates volumes in the following order:
1. Concatenate disks
2. Form columns
3. Form mirrors

The VxVm software version 3.2 provides configuration control, allowing


the selection of either the default configuration (RAID 0 + 1) or a layered
volumes configuration (RAID 1 + 0), also referred to as a stripe
professional or concat professional configuration.

Sun Proprietary: Internal Use Only


E-4 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
VxVM Software States

VxVM Software States


State information is critical to understanding the health of a configuration.
This section defines the various state information for plexes, volumes, and
disks.

Plex States
Plex state information reflects consistent or inconsistent configurations
and the state of those configurations.

Valid plex states are described in detail in Module 1, ‘‘Introducing the


VERITAS Volume Manager Software Architecture,” in the section ‘‘Plex
State Descriptions’’ on page 1-34.

Plex and Volume Kernel States


The plex and volume kernel states indicate the accessibility of the plex.

Plex kernel states are described in detail in Module 1, ‘‘Introducing the


VERITAS Volume Manager Software Architecture,” in the section ‘‘Plex
Kernel States’’ on page 1-36. Volume kernel states are described in the
same manner as plex kernel state.

Volume States
Volume states consist of the following:
● Clean – Volume is not started; kernel state is disabled, but plexes are
synchronized.
● Active – Volume is started; kernel state is enabled.
● Empty – Volume is not initialized; kernel state is disabled.
● Sycn – Volume is in recovery mode; kernel state is enabled. Also
indicates volume is recovered after boot; kernel state is disabled, and
plexes need to be resynchronized.
● Needsync – Volume requires resynchronization.
● Replay – Volume is in transient state as part of log replay (only valid
for RAID 5).

Sun Proprietary: Internal Use Only


Configuring the VxVM Software E-5
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
VxVM Software States

Disk States
The following example shows output of the vxdisk list command:
DEVICE TYPE DISK GROUP STATUS
c0t0d0s2 sliced disk06 - online
c0t2d0s2 sliced - - error
c1t0d0s2 sliced rootdisk rootdg online
c2t0d0s2 sliced - - error
c3t0d1s2 sliced - - online
c3t0d4s2 sliced rootmir rootdg online
c3t1d1s2 sliced newdg01 newdg online spare
c3t2d0s2 sliced newdg02 newdg online spare
c3t2d1s2 sliced disk01 rootdg online
c3t2d2s2 sliced newdg04 newdg online reserved
c3t3d0s2 sliced newdg05 newdg online nohotuse
c3t3d2s2 sliced newdg07 newdg online invalid
c3t3d3s2 sliced - - error
c3t4d0s2 sliced newdg09 newdg online failing
c3t4d3s2 sliced newdg12 newdg online altused
c3t5d2s2 sliced newdg15 newdg online
- - newdg03 newdg removed was:c3t2d1s2
- - newdg08 newdg removed was:c3t3d3s2
- - newdg11 newdg failed was:c3t4d2s2

As shown in the previous example, disk states are the following:


● Invalid – A private area exists, but the information in it is not a valid
configuration.
● Altused – The alternate configuration copy in the private area is in
use.
● Failed was – Indicates full disk failure.
● Removed was – The vxdiskadm #4 command was executed.
● Error – No private area on this disk.
● Reserved – This disk is not used by the vxassist command to make
new volumes or for relocation.
● Failing – (VxVM software version 2.x and 3.x) This disk incurred a
hardware error in the past.
● Nohotuse – (VxVM software version 3.x) This disk is not used for
relocated data.
● Dgname – (VxVM software version 3.x) The disk group name, shown
only when using the -o alldgs option.

Sun Proprietary: Internal Use Only


E-6 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
The vxprint Command

The vxprint Command


Execution of the vxprint command displays the state of a RAID
configuration, providing an overview and the current state of the
configuration.

Output Header Information


The command output header contains important information. An
example of output from the vxprint command is shown:
bash-2.03# vxprint -htg rootdg
DG NAME NCONFIG NLOG MINORS GROUP-ID
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
V NAME RVG KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO

dg rootdg default default 0 1023554924.1025.lowtide

dm rootdisk c1t0d0s2 sliced 3590 17678493 -


dm rootmirror c1t1d0s2 sliced 3590 17674902 -

v rootvol - ENABLED ACTIVE 4197879 ROUND - root


pl rootvol-01 rootvol ENABLED ACTIVE 4197879 CONCAT - RW
sd rootdisk-B0 rootvol-01 rootdisk 17678492 1 0 c1t0d0 ENA
sd rootdisk-02 rootvol-01 rootdisk 0 4197878 1 c1t0d0 ENA
pl rootvol-02 rootvol ENABLED ACTIVE 4197879 CONCAT - RW
sd rootmirror-01 rootvol-02 rootmirror 0 4197879 0 c1t1d0 ENA

Sun Proprietary: Internal Use Only


Configuring the VxVM Software E-7
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
The vxprint Command

Field Descriptions
This section contains an explanation of the fields in the vxprint output.
For additional information about the vxprint command, see the vxprint
man page.
● Record type dg:
● Disk group name
● Disk ID
● Record type dm:
● Record name
● Underlying disk access record
● Disk access record type (sliced, simple, or nopriv)
● Length of the disk’s private region
● Length of the disk’s public region
● Record type sd:
● Record name
● Associated plex, or dash (-) if the subdisk is dissociated
● Name of the disk media record used by the subdisk
● Device offset in sectors
● Subdisk length in sectors
● Plex association offset – Optionally, this value is preceded by
subdisk column number for subdisks associated to striped
plexes, LOG for log subdisks, or the putil[0] field if the
subdisk is dissociated. The putil[0] field can be non-empty to
reserve the subdisk space for non-volume uses. If the putil[0]
field is empty, it is a dissociated subdisk.
● Subdisk state string:
● ENA – The subdisk is usable.
● DIS – The subdisk is disabled.
● RCOV – The subdisk is part of a RAID-5 plex and has stale
content.
● DET – The subdisk is detached.
● KDET – The subdisk is detached in the kernel due to an
error.

Sun Proprietary: Internal Use Only


E-8 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
The vxprint Command

● RMOV – The media record on which the subdisk is defined


was removed from its disk access record by a utility.
● RLOC – The subdisk has failed and is waiting to be
relocated.
● NDEV – The media record on which the subdisk is defined
has no associated access record.
● Record type sv:
● Record name.
● Associated plex, or dash (-) if the subvolume is dissociated.
● Name of the underlying (layered) volume record used by the
subvolume.
● Number of layers used in the subvolume.
● Subvolume length in sectors.
● Plex association offset – This value is optionally preceded by
subvolume column number for subvolumes associated with
striped plexes.
● Number of active plexes, followed by the number of plexes in
the underlying (layered) volume.
● Subvolume state string:
● ENA – The subvolume is usable.
● DIS – The subvolume is disabled.
● KDET – The subvolume was detached in the kernel due to
an error.
● Record type pl:
● Record name.
● Associated volume, or dash (-) if the plex is dissociated.
● Plex kernel state.
● Plex utility state – If an exception condition is recognized on the
plex (such as an I/O failure, a removed or inaccessible disk, or
an unrecovered stale data condition), that condition is listed
instead of the value of the plex record’s state field.
● Plex length in sectors.
● Plex layout type.

Sun Proprietary: Internal Use Only


Configuring the VxVM Software E-9
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
The vxprint Command

● Number of columns and plex stripe width, or if the plex is not


striped
● Plex I/O mode:
● RW – Read-Write
● WO – Write only
● RO – Read only

For volumes, the output consists of the following fields, from left to right:
● Record type v:
● Record name
● Associated usage type
● Volume kernel state
● Volume utility state
● Volume length in sectors
● Volume read policy
● The preferred plex, if the read policy uses a preferred plex
● Record type dc:
● Record name
● Associated volume, or dash (-) if the data change object (DCO)
is dissociated
● Name of the DCO log volume, or dash (-) if no DCO log
volume is associated with the DCO object
● Record type sp:
● Record name
● Name of the volume which this snap record describes
● Name of the DCO with which this snap record is associated
● Record type rv:
● Record name
● Associated remote link (RLINK) object count
● Remote volume group (RVG) kernel state (derived from various
flags)
● RVG utility state
● RVG primary flag (primary or secondary)

Sun Proprietary: Internal Use Only


E-10 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
The vxprint Command

● Associated data volume count


● The srl volume
● Record type rl:
● Record name
● Associated RVG, or dash (-) if the RLINK is dissociated
● RLINK kernel state (derived from various flags)
● RLINK utility state
● The remote host
● The remote disk group
● The remote RLINK

Stripe Width and Stripe Units


Data is allocated in equal-sized units, called stripe units, that are
interleaved between the columns. Each stripe unit is a set of contiguous
blocks on a disk. The total number of blocks defined by the value stripe
units determine the per-column stripe width. When used during volume
creation, the terms stripe unit and stripe width are synonymous.
Differentiation between the two terms occurs at the command level.

The vxassist command option stripeunit defines stripe size. The


vxmake command option stwidth (for stripe width) defines stripe size.
The vxassist command builds the plex and volume in a single
command. The vxmake command builds individual plexes that are then
associated to volumes using additional iterations of the vxmake command.

The following example commands create a stripe of four columns with a


stripe size of 32 kilobytes:
# vxassist make stripevol 80g layout=stripe stripeunit=32k ncols=4
# vxmake plex pl-01 80g layout=stripe stwidth=32k ncols=4 \
sd=disk01-01,disk02-01,disk03-01,disk04-01

Sun Proprietary: Internal Use Only


Configuring the VxVM Software E-11
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Complex RAID Levels

Complex RAID Levels


This section describes the concepts and commands used to build the
concat-pro and stripe-pro volumes, also known as RAID 1 + 0, or RAID 10.

In previous releases, when mirroring was used, the mirroring had to occur
above striping. In the VxVM software version 3.2, mirroring can occur
both above and below striping. Mirroring put below striping mirrors each
column of the stripe. If the stripe is large enough to have multiple
subdisks per column, each subdisk can be individually mirrored. Instead
of forming subdisks into plexes and then mirroring the plexes, RAID 1 + 0
provides a separate plex for each disk that is then mirrored individually.

Figure E-1 shows a comparison between RAID 0 + 1 and RAID 1 + 0


configurations.

RAID 0+1 Volume RAID1+0 Volume


Plex
Mirror
Plex Plex Stripe Volumes
Mirror
Subplex Subplex

Stripe sd sd

Stripe Volumes
mirror
Mirror
Stripe
subplex subplex

sd sd
sd sd

Figure E-1 RAID 0 + 1 and RAID 1 + 0 Comparison

The layout in Figure E-1 is referred to as layer volumes or a VERITAS stripe-


pro or VERITAS concat-pro layout. This layout enhances redundancy and
reduces recovery time in case of an error. In a mirror-stripe layout, if a
disk fails, the entire plex is detached, thereby losing redundancy on the
entire volume. When the disk is replaced, the entire plex must be brought
up to date. Recovering the entire plex can take a substantial amount of
time.

Sun Proprietary: Internal Use Only


E-12 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Complex RAID Levels

If a disk fails in a stripe-mirror layout, only the failing subdisk must be


detached, and only that portion of the volume loses redundancy. When
the disk is replaced, only a portion of the volume needs to be recovered.
Compared to mirroring plus striping, striping plus mirroring offers a
volume more tolerant to disk failure and quicker recovery time.

The following example demonstrates RAID 1 + 0 using the -o ordered


option:
# vxassist -g newdg3dg -o ordered make vol01 1g layout=stripe-mirror ncol=2 disk04
disk02 disk01 disk03
# vxprint -htr
.
.
.

The previous example shows stripes of RAID 0 combined with two


subvolumes of a RAID 1 mirror, as shown in Figure E-2.

Volume - vol01
Plex - vol01-03
vol01-S01

Stripe
vol01-S02

Figure E-2 Top Layers of a RAID 1 + 0 Volume

Sun Proprietary: Internal Use Only


Configuring the VxVM Software E-13
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Complex RAID Levels

Matching the diagram in Figure E-2 on page E-13 against the output of the
example vxprint command shows the following objects:
● Volume (v) – vol01
● Plex (pl) – vol01-03
● Subvolume – vol01-S01
● Subvolume – vol01-S02

These objects provide the stripe. Notice that even though mirroring is
implemented, it is not visible to the single upper layer plex. In essence, this
configuration is a single stripe that is mirrored.

There are two subvolumes with a total of four subdisks in the plex. From
the perspective of the plex, the capacity is equal to two subdisks. The
other subdisks provide the data redundancy. Underneath the plex, logical
subvolumes provide the mirroring as shown in Figure E-3.

Plex
vol01-S01
vol01-L01
vol01-P01 vol01-P02
Disk 04 Disk 01

Mirror

vol01-S02
vol01-L02
vol01-P03 vol01-P04

Disk 02 Disk 03

Mirror

Figure E-3 Lower Layers of a RAID 1 + 0 Volume

The subvolumes each contain two subplexes which are concatenated as


part of a mirror. Each subplex can be detached or resynchronized
individually, allowing for faster recovery times.

Sun Proprietary: Internal Use Only


E-14 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Complex RAID Levels

The following examples show a RAID 1 + 0 configuration without using


the -o ordered option:
# vxassist -g testdg make vol01 1g layout=stripe-mirror ncol=2
# vxassist -g testdg make vol01 1g layout=stripe-mirror ncol=2 disk04 disk02 disk01
disk03

Both operations provide the following result:


# vxprint -htr (partial output)
v vol01 - ENABLED ACTIVE 2097152 SELECT vol01-03 fsgen
pl vol01-03 vol01 ENABLED ACTIVE 2097152 STRIPE 2/128 RW
sv vol01-S01 vol01-03 vol01-L01 1 1048576 0/0 2/2 ENA
v2 vol01-L01 - ENABLED ACTIVE 1048576 SELECT - fsgen
p2 vol01-P01 vol01-L01 ENABLED ACTIVE 1048576 CONCAT - RW
s2 disk01-02 vol01-P01 disk01 0 1048576 0 c0t1d0 ENA
p2 vol01-P02 vol01-L01 ENABLED ACTIVE 1048576 CONCAT - RW
s2 disk03-02 vol01-P02 disk03 0 1048576 0 c1t9d0 ENA
sv vol01-S02 vol01-03 vol01-L02 1 1048576 1/0 2/2 ENA
v2 vol01-L02 - ENABLED ACTIVE 1048576 SELECT - fsgen
p2 vol01-P03 vol01-L02 ENABLED ACTIVE 1048576 CONCAT - RW
s2 disk02-02 vol01-P03 disk02 0 1048576 0 c1t8d0 ENA
p2 vol01-P04 vol01-L02 ENABLED ACTIVE 1048576 CONCAT - RW
s2 disk04-02 vol01-P04 disk04 0 1048576 0 c1t10d0 ENA

Additional Layout Options


Additional attributes are used with the vxassist command to control the
data layout in the volume.

One attribute is the col_switch attribute, which specifies how to


concatenate space on the disks into columns. For example, the following
command creates a mirrored-stripe volume with two columns:
# vxassist -o ordered make new 10g layout=mirror-stripe ncol=2
col_switch=3g,2g disk01 disk02 disk03 disk04 disk05 disk06 disk07 disk08

Sun Proprietary: Internal Use Only


Configuring the VxVM Software E-15
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Complex RAID Levels

This command allocates spacing, as shown in Figure E-4.

Mirrored Stripe Volume


Striped Plex

Column 1 Column 2 Column 3

disk01-01 disk03-01 disk05-01

disk02-01 disk04-01 disk06-01

Mirror Mirror Mirror


Column 1 Column 2 Column 3

disk07-01 disk09-01 disk11-01

disk08-01 disk10-01 disk12-01

Striped Plex

Figure E-4 Mirrored Volume with Multiple Subdisks

Storage is also allocated based on controllers and enclosures with ordered


allocation. For example, the following command creates a three column,
mirrored-stripe volume across controllers:
# vxassist -o ordered make mirvol 80g layout=mirror-stripe ncol=3 \
ctlr:c1 ctlr:c2 ctlr:c3 ctlr:c4 ctlr:c5 ctlr:c6

This command allocates space for column 1 from disks on controllers c1,
for column 2 from disks on controller c2, and so on.

Sun Proprietary: Internal Use Only


E-16 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Complex RAID Levels

Additional Mirror Attributes


The VxVM software provides additional attributes for controlling
mirroring, using the vxassist command:
# vxassist make volume length layout=layout mirror=[attribute]

Mirror attributes include the following:


● The attribute mirror=ctlr specifies placement of disks in one
mirror on a different controller than disks in other mirrors within the
same volume.
● The attribute mirror=target specifies mirroring of volumes
between identical target IDs on different controllers.
● The attribute mirror=enclr specifies placement of disks in one
mirror in a different enclosure than disks in other mirrors within the
same volume.

The following command creates a mirrored volume with two plexes on


different controllers:
# vxassist make volmnt 10g layout=mirror nmirror=2 mirror=ctlr ctlr:c1 ctlr:c2

In this example, the disks in one plex are all attached to controller c1, and
the disks in the other plex are all attached to controller c2. If a controller
fails, only one side of the mirror is lost.

Sun Proprietary: Internal Use Only


Configuring the VxVM Software E-17
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Disk Layout Practices

Disk Layout Practices


Consider the following recommended guidelines for file system layouts:
● Start simple.
Start with a simple disk layout; then add complexity. Learn the
layout needed to identify the tool set requirements, and start further
tuning.
● More spindles are better.
The ability to get top performance from a system depends on
avoiding contention and evenly spreading throughput.
● Bigger is not always better.
Do not confuse capacity and performance. Often one feature must be
compromised to achieve the other. Doubling the capacity of storage
does not increase the performance speed.
Capacity is measured in gigabytes while performance is measured in
I/O per second.
● Look at the big picture.
Learn the system environment before focusing too much on where a
problem is located. Look at the whole environment, and apply efforts
across the entire environment.
● Trade-offs must be made.
There is no perfect solution, and there is always more than one
solution. This usually results in a mutually-exclusive choice between
things which cannot be obtained at the same time.
● Performance is never the only consideration.
Availability, reliability, and manageability are very high on the list of
concerns in system layout, as are price of ownership (not to be
confused with purchase price alone) and scalability (that is, room to
grow).

Sun Proprietary: Internal Use Only


E-18 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Disk Layout Practices

Guidelines for RAID Array Layouts


This section contains recommendations for configuring arrays.
● RAID 0 Mirroring – Production file systems are mirrored. Staging
and development file systems are not.
Proper backup procedures must be implemented. When possible,
disks are mirrored across controllers. Experience shows that when
this is done correctly, either a disk or a controller can fail and the
system remains fully operational.
Mirror across arrays and label the arrays A and B. Doing this allows
a system to lose an entire array and still stay on line. A typical
response to this situation is use the VxVM software and rename the
disks to include the array identifier. For example, rename disk22 to
diskA22 and the mirror to diskB22.
● RAID 1 Striping – Striping is highly effective when used correctly to
solve a known and understood problem. Experience shows that
when this is used incorrectly, however, it compounds problems and
makes problem resolution difficult.
Implement striping in a standardized manner, where used, across all
platforms. This reduces administration overhead.
● RAID 5 – Parity disks are used at the hardware level only or on read-
only oriented file systems.

Guidelines for Complex File System Layouts


To lay out a complex file system, a general work practice is to first create
simple, single-volume, single-disk file system. When the simple system is
in place, take performance measurements and use them to create the
required I/O RAID 1 file system.

Use the following guidelines for layout strategies:


● Small file systems (under 2 gigabytes) are generally mapped to a
single disk.
● Medium file systems (between 2 and 8 gigabytes) are usually striped
over four disks.
● Large file systems (8 gigabytes and above) are usually striped over
four to eight disks.

Sun Proprietary: Internal Use Only


Configuring the VxVM Software E-19
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Disk Layout Practices

Striping Considerations
The question of how to perform striping is a subject of hot debate with no
correct answer. This is partly due to the fact that striping can either help
or hurt performance, depending on the workload, the number of stripes
per disk, concurrent access, and how well the application uses I/O.

Striping Characteristics

The objective of striping is to reduce head contention. Potentially,


especially with high capacity disks, striping can increase head contention
by increasing concurrent access, resulting in inconsistent allocation of the
stripe or poor data layouts (multiple heavy I/O stripes on the same
disks).

After striping is set up, it can be more difficult to reconfigure than


unstriped environments. This can result in additional time the system
administrator needs to set up and maintain the striped environment.

Another consideration is the human overhead associated with configuring


and managing a striped I/O subsystem. This impact is far higher on a
busy system than on an idle one. Operators and administrators must be
trained on the striping setup and the impact each disk has on the system
performance.

Guidelines for Striping

Use the following guidelines to determine striping layout need:


● The primary goal for striping is to identify large file systems which
require striping. Large file systems with high-intensity sequential
reads or writes require striping.
Place those file systems which do not require striping in a single disk
volume.
● When striping, you should use increments of disks available on the
system. Usually, that is a four-column or six-column stripe. Using
symmetrical striping avoids creating roving hot spots.
● A good starting point is a four-way stripe set with a 64 kilobyte
stripe unit. The stripe should become a standard implementation.
RAID 1 + 0 stripes can be potentially larger because recovery times
are not as dramatically impacted.

Sun Proprietary: Internal Use Only


E-20 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Disk Layout Practices

● Avoid a stripe width of greater than eight columns. Risk of


contention goes up and the performance gains level off in systems
with greater than eight-column stripes.
● The stripe size should always be a multiple of the database block
size. It is grossly inefficient to split an I/O over multiple I/O
requests.
When performing I/Os that require multiple blocks, it is best to
ensure that each I/O can be satisfied from one disk. Examples of this
type of I/O requests are table scans, backups, and disaster recovery
restore operations. Chose a stripe size that is a multiple of the file
system block (for example, in Oracle this would be a multiple of
db_block_multiblock_read_count init.ora parameter).
Random access, non-table scanning applications tolerate a smaller
block size. Do not set the stripe size so small that backups and
restores are inefficient.
Table scanning applications benefit from large stripe sizes. Good
stripe sizes vary from 128 kilobytes for online transaction processing
(OLTP) applications to several megabytes for data warehouse
applications.

Sun Proprietary: Internal Use Only


Configuring the VxVM Software E-21
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Manipulating Disk Layouts

Manipulating Disk Layouts


This section describes procedures for working with disk layouts.

Online Re-Layout
Online re-layout allows reconfiguration of RAID layouts on currently
configured volumes. A volume can be re-laid out or converted to another
layout.

Online re-layout is supported for the following operations:


● RAID 5 to mirror
● Mirror to RAID 5
● Mirror to Concat
● Concat to RAID 5
● RAID 5 to Concat
● Adding or removing parity
● Adding or removing columns
● Changing stripe width

Re-Laying Out Volumes

To re-layout volumes, use the following format for the vxassist


command with the relayout option:
# vxassist relayout vol01 layout=stripe,log nmirror=2 ncolumn=5 stwidth=256K

Only one plex is used if you are re-laying out to RAID 5. See the
vxassist man pages for additional information.

Sun Proprietary: Internal Use Only


E-22 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Manipulating Disk Layouts

Converting RAID Configurations

The following commands convert a RAID configuration to a new layout:


# vxassist -g new-dg convert vol01 layout=stripe-mirror
# vxassist -g new-dg convert vol01 layout=mirror-stripe
# vxassist -g new-dg convert vol01 layout=concat-mirror

Growing File Systems


To grow a volume that has a file system on it, you must grow the volume
first, and then the file system.

Using the vmsa GUI

The vmsa GUI allows a user to grow either a volume or a file system.
Depending on whether the user selects a volume or file system, the
selected item is either grown or shrunk. For example, selecting the volume
enables a resize menu entry to grow only the volume. Selecting the file
system enables a resize menu entry to grow both the volume and the file
system.

Using the Command Line

To grow an existing volume, perform the following procedures:


1. Display the current size of the volumes. Type the following:
# vxprint -g test01 -t vol01
V NAME USETYPE KSTATE STATE LENGTH READPOL PREFPLEX
v vol01 fsgen ENABLED ACTIVE 40960 SELECT -
2. Determine the current size of the file system.
a. For UFS, type the following:
# fstyp -v /dev/vx/dsk/test01/vol01 | head -10 | grep ncg
ncg 5 size 20480 blocks 19183
The number after size is the size of the file system, in kilobytes. To
translate to sectors, multiply by 2.
b. For VxFS, type the following:
# fstyp -v /dev/vx/dsk/test01/vol01 | grep nau
bsize 1024 size 20480 dsize 91392 ninode 0 nau 3

Sun Proprietary: Internal Use Only


Configuring the VxVM Software E-23
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Manipulating Disk Layouts

Multiply the number after bsize by the number after size. That
determines the size of the file system, in bytes. To translate to sectors,
divide that number by 512.
1024 (bsize) x 20480 (size) / 512 = 40960 sectors
This file system size is 40960 sectors, which matches the volume size.
3. Determine the largest size an existing volume can be grown.
Type the following:
# vxassist -g rootdg maxgrow vol01 disk01 disk02 disk03
Volume vol01 can be extended by 12244992 to 12285952 (5999Mb)
If you do not specify the disks, the vxassist command uses the
disks in the diskgroup.
4. Grow the volume to the required size. Type the following:
# vxassist -g rootdg growto vol01 10639360 disk01 disk02 disk03
This command only grows the volume. Remember to grow the file
system as well, if needed.

Note – The vxassist command also has a growby option. See the
vxassist man pages for more information.

Manually Growing the File System

Determine to which size to grow the file system. Usually, this is the size of
the volume. Proceed as follows:
1. Determine the size of volume with the following command:
# vxprint -g rootdg -t vol01
V NAME USETYPE KSTATE STATE LENGTH READPOL PREFPLEX
v vol01 fsgen ENABLED ACTIVE 10639360 SELECT -
2. Grow the file system. Type –
# /usr/lib/fs/ufs/mkfs -F ufs -M /mnt /dev/vx/rdsk/rootdg/vol01 10639360

The mkfs command has an option to grow a file system instead of making
one. To use this option, the file system must be mounted. You must also
use the full path to the mkfs program; do not use /usr/sbin/mkfs.

Note – The above process is described in detail by Sun Solve INFODOC


14881.

Sun Proprietary: Internal Use Only


E-24 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Manipulating Disk Layouts

To grow a file system, execute the following command for VxFS:


# /usr/lib/fs/vxfs/fsadm -b 10639360 -r /dev/vx/rdsk/rootdg/vol01 /mnt

Examples of Grown File System

Below are examples of grown file systems. Notice the change in the offset
positions in the first example and the change in the disk offset in the
second example.

An example of a stripe before layout is shown.


# vxprint -ht -g newdg vol04

V NAME USETYPE KSTATE STATE LENGTH READPOL PREFPLEX


PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
v vol04 fsgen ENABLED ACTIVE 154224 SELECT vol04-01
pl vol04-01 vol04 ENABLED ACTIVE 154224 STRIPE 4/128 RW
sd newdg01-01vol04-01 newdg01 0 51408 0/0 c4t0d0 ENA
sd newdg03-01vol04-01 newdg03 0 51408 1/0 c4t2d0 ENA
sd newdg05-01vol04-01 newdg05 0 51408 2/0 c4t4d0 ENA
sd newdg07-01vol04-01 newdg07 0 51408 3/0 c4t0d1 ENA

An example of a stripe after layout is provided.


# vxprint -ht -g newdg vol04

V NAME USETYPE KSTATE STATE LENGTH READPOL PREFPLEX


PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE

v vol04 fsgen ENABLED ACTIVE 614400 SELECT vol04-01


pl vol04-01 vol04 ENABLED ACTIVE 616944 STRIPE 4/128 RW
sd newdg01-01vol04-01 newdg01 0 51408 0/0 c4t0d0 ENA
sd newdg02-01vol04-01 newdg02 0 102816 0/51408 c4t1d0 ENA
sd newdg03-01vol04-01 newdg03 0 51408 1/0 c4t2d0 ENA
sd newdg04-01vol04-01 newdg04 0 102816 1/51408 c4t3d0 ENA
sd newdg05-01vol04-01 newdg05 0 51408 2/0 c4t4d0 ENA
sd newdg06-01vol04-01 newdg06 0 102816 2/51408 c4t5d0 ENA
sd newdg07-01vol04-01 newdg07 0 51408 3/0 c4t0d1 ENA
sd newdg08-01vol04-01 newdg08 0 102816 3/51408 c4t1d1 ENA

A second example of a stripe before layout is shown.


# vxprint -ht -g newdg vol04
V NAME USETYPE KSTATE STATE LENGTH READPOL PREFPLEX
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE

v vol04 fsgen ENABLED ACTIVE 205632 SELECT vol04-01


pl vol04-01 vol04 ENABLED ACTIVE 205632 STRIPE 4/128 RW

Sun Proprietary: Internal Use Only


Configuring the VxVM Software E-25
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Manipulating Disk Layouts

sd newdg12-01vol04-01 newdg12 0 51408 0/0 c4t0d0 ENA


sd newdg13-01vol04-01 newdg13 0 51408 1/0 c4t0d1 ENA
sd newdg08-01vol04-01 newdg08 0 51408 2/0 c4t2d4 ENA
sd newdg02-01vol04-01 newdg02 0 51408 3/0 c4t3d1 ENA

Another stripe after layout example is shown.


# vxprint -ht -g newdg vol04
V NAME USETYPE KSTATE STATE LENGTH READPOL PREFPLEX
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE

v vol04 fsgen ENABLED ACTIVE 409600 SELECT vol04-01


pl vol04-01 vol04 ENABLED ACTIVE 411552 STRIPE 4/128 RW
sd newdg12-01vol04-01 newdg12 0 51408 0/0 c4t0d0 ENA
sd newdg12-01vol04-01 newdg12 358848 51408 0/51408 c4t0d0 ENA
sd newdg13-01vol04-01 newdg13 0 102816 1/0 c4t0d1 ENA
sd newdg08-01vol04-01 newdg08 0 51408 2/0 c4t2d4 ENA
sd newdg08-03vol04-01 newdg08 358848 51408 2/51408 c4t2d4 ENA
sd newdg02-01vol04-01 newdg02 0 102816 3/0 c4t3d1 ENA

Sun Proprietary: Internal Use Only


E-26 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Changing Disk Group Configurations

Changing Disk Group Configurations


To reorganize disk groups, move disks and objects between the disk
groups. Generally this is driven by organizational change, online
maintenance requirements, or when the capacity of the private region is
reached. Changing the disk group has less impact than growing the
private region.

To change configurations, you must ensure that the vxdg command


options interact with vxconfigd to validate, identify, and populate new
configurations. These commands are described in the following
paragraphs.

Disk Group Reconfiguration Commands


Use the following syntax to move objects from one disk group to a new
disk group:
vxdg [-o expand] split sourcedg targetdg [object ...]

Caution – Using the same target name as an existing deported disk group
destroys that group.

To join disk groups, use the following syntax:


vxdg [-o expand] join sourcedg targetdg [object ...]

To preview a move, use the following syntax:


vxdg [-o expand] listmove sourcedg targetdg [object ...]

To move objects from one imported disk group to another, use the
following syntax:
vxdg [-o expand] move sourcedg targetdg [object ...]

For each of these vxdg commands, the option -o expand includes all
disks from volumes sharing subdisks.

For a complete list of options for the vxdg command, see the man pages.

Sun Proprietary: Internal Use Only


Configuring the VxVM Software E-27
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Changing Disk Group Configurations

Example Disk Group Reconfiguration Commands

The following examples demonstrate the use of commands to modify disk


groups.
● The following example commands each split a disk group and then
restart the volumes in the new disk group:
# vxdg split old-dg new-dg disk01 disk02 disk03
# vxrecover -g new-dg -m vol01
# vxvol -g new-dg start vol01
● The following commands join a disk group:
# vxdg join new-dg old-dg
# vxrecover -g old-dg -m vol01
● This command displays potential objects to move, including shared
subdisks –
# vxdg -o expand listmove old-dg new-dg
● This command moves objects from one disk group to another:
# vxdg -o expand move old-dg new-dg vol04

Disk Group Reconfiguration Recovery


If a problem occurs during the reconfiguration of a disk group, such as a
system crash, the VxVM software release 3.2 provides recovery
capabilities.

First, the VxVM software release 3.2 attempts to recover automatically by


either completing the operation or reversing the operation, depending on
the progress of the reconfiguration. If the recovery is successful, the
VxVM software imports the affected disk group and starts the volumes.

If the VxVM software recovery is not successful, perform the following


procedures to recover the disk group manually:
1. Use the vxprint command to examine the configuration of both
disk groups. Objects in disk groups for which the move operation is
incomplete have a TUTIL0 field set to MOVE. For example:
# vxprint -h new-dg
Disk group: newdg

TY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0


dg new-dgnewdg - - - - MOVE -

Sun Proprietary: Internal Use Only


E-28 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Changing Disk Group Configurations

The VxVM software uses the TUTIL0 and PUTIL0 fields to lock the
affected objects during transition.
2. Enter the following command to complete the move:
# vxdg recover new-dg

vxvm: vxdg: ERROR: diskgroup: Disk group does not exist


3. If the recovery fails, check to see if the disk group was imported onto
another host. If it was imported, deport it from that host, and import
it onto the current host.
4. If all the required objects already exist in either the source or target
disk group, use the following command to reset the MOVE flags in
that disk group:
# vxdg -o clean recover new-dg
5. Use the following command on the other disk group to remove the
objects that have TUTIL0 fields marked as MOVE :
# vxdg -o remove recover old-dg
6. If only one disk group is available to be imported, use the following
command to reset the MOVE flags on this disk group:
# vxdg -o clean recover old-dg

Listing the PUTIL and TUTIL Fields

The PUTIL and TUTIL fields are used as locks on volumes, plexes,
subdisks, and disks during most types of configuration changes and
recoveries. For example, if attaching a plex to a volume, the plex TUTIL0
field is set to ATT. Once the attach is complete, the TUTIL0 field is cleared
automatically.

If a user attempts to perform a function on a entity with a set PUTIL or


TUTIL flag, the following message is received:
vxvm:vxplex: ERROR: Volume volname is locked by another utility

The PUTIL fields are permanent and stay set even after a reboot. The
TUTIL fields are temporary and do not survive a reboot.

Sun Proprietary: Internal Use Only


Configuring the VxVM Software E-29
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Changing Disk Group Configurations

To list the PUTIL and TUTIL fields, use the following command:
# vxprint -h vol01
TY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0
v vol01 fsgen ENABLED 2050272 - ACTIVE ATT1 -
pl vol04-01 vol01 ENABLED 2050272 - ACTIVE - -
sd newdg02-02vol04-01 ENABLED 2050272 0 - - -
pl vol01-01 vol01 ENABLED 2050272 - TEMPRMSD ATT -
sd newdg11-01vol01-01 ENABLED 2050272 0 - - -

Clearing the PUTIL and TUTIL Fields

To clear the PUTIL and TUTIL lock flags:


1. Ensure that no recover command is in process. Execute one of the
following commands:
# ps -ef | grep vx
# vxtask -l list
If there is a vxvol, vxplex or vxrecover operation in process, the
PUTIL and TUTIL fields are probably set for a reason.
2. Use the following syntax to clear the flags:
# vxmend -g dgname clear all objectname
Replace objectname with the volume, plex, subdisk, or disk which
has a PUTIL or TUTIL flag set.

Reconfiguration Considerations
The disk group move, split, and join features have the following
limitations:
● Disk groups involved in a move, split, or join must be version 90 or
greater.
● The reconfiguration must involve physical disks.
● Objects to be moved must not contain open volumes.
● Moved volumes are initially disabled following a disk group move,
split or join. Use either vxrecover -m or vxvol startall to restart the
volumes.
● Data change objects (DCOs) and snap objects that are dissociated by
persistent fast resynchronization cannot be moved between disk
groups.
● Sun StorEdge Volume Replicator (VR) objects cannot be moved
between disk groups.

Sun Proprietary: Internal Use Only


E-30 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Changing Disk Group Configurations

● For a disk group reconfiguration to succeed, the source and target


disk group must be able to store copies of the configuration database
after the move.
The configuration database in the target disk group must also be able
to store all the object information in the enlarged disk group.
● Splitting or moving a volume into a different disk group changes the
volume record ID.

Sun Proprietary: Internal Use Only


Configuring the VxVM Software E-31
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Hot-Relocation

Hot-Relocation
Hot-relocation allows a system to relocate data automatically in a
redundant configuration, in the event of a subdisk failure. There are,
however, a number of restrictions for hot-relocation use:
● Subdisks must be in a redundant configuration (mirror or RAID 5).
● Space must be available to contain the recovered data.
● Hot-relocation fails if:
● Space is only available in the same plex of the mirror as the
failed sub-disk.
● Space is only available on a plex that contains a RAID 5 log.
● The failed subdisk is in the same plex as a DRL log.
● RAID 5 logs or DRL logs are created, not relocated.

This section discusses the hot-relocation process and how to perform it.

Hot-Relocation Process
To execute relocation, the hot-relocation deamon vxrelocd handles four
distinct operations, in the following order:
1. Failure detection
2. Notification
3. Relocation
4. Recovery

The vxrelocd daemon monitors events that signify a subdisk or plex


failure. When a failure occurs, data space is selected from hot-relocation-
reserved disks or from available free space in the disk group, and a
resynchronization begins. The failed disk is marked as failing. The
deamon notifies designated users and then reconstructs the objects in the
new location; the new subdisks retain the existing names. The last
operation is to recover the data.

Figure E-5 on page E-33 shows the interaction of various processes during
these procedures.

Sun Proprietary: Internal Use Only


E-32 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Hot-Relocation

vxrelocd

vxnotify
failure
detected

vxrelocd
correct, vxrelocd
action

mailx
determine
notify available
vxconfigd space
access
disk

updates
configuration vxassist
builds
objects

vxrecover
recovers data

Figure E-5 Hot-Relocation Process Interaction

Failure Detection

The vxrelocd deamon is a script running in the background, interacting


with vxnotify, to monitor three types of failure:
● Disk failures using the failing flag in DM records
● Plex failures due to uncorrectable I/O errors
● Subdisk failures in a RAID 5 volume

Sun Proprietary: Internal Use Only


Configuring the VxVM Software E-33
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Hot-Relocation

After a failure is detected, the vxrelocd deamon takes object-specific


actions and can include a kernel-initiated change that alters the
configuration. If a configuration change is required, the vxconfigd
deamon is notified with the request update. The vxconfigd deamon, in
turn, writes out the configuration change to the database copies.

Notification

The vxrelocd deamon notifies users (by default, the root user) using the
mailx command, providing information about the failure and the status
of relocation and recovery. The file /etc/rc2.d/s95-vxvm-recover
contains a list of users to notify.

To add users to the notification list, modify the


/etc/rc2.d/s95-vxvm-recover file as follows:
vxrelocd root username1 username2 &

Changes to this file take effect at the next reboot or when the vxrelocd
command is executed from the command line. To execute this command
from the command line, kill the running vxrelocd deamon first, but be
careful not to kill the deamon in the middle of a relocation process.

Error notification takes the following form:


Date: Wed, 6 Feb 2002 12:04:27 GMT
From: root
Subject: Volume Manager failures on host fred
To: root

Volume vol02 Subdisk disk04-02 relocated to disk06-03, but not yet recovered.

Sun Proprietary: Internal Use Only


E-34 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Hot-Relocation

Relocation

The vxrelocd daemon determines available space by looking for the


spare flag in the DM record.

To display the spare flag use either the vxdisk or vxprint command, as
shown in the following example:
# vxdisk list
DEVICE TYPE DISK GROUP STATUS
c0t0d0s2 sliced rootdisk rootdg online
c1t12d0s2 sliced altboot rootdg online
c1t13d0s2 sliced newdg01 newdg online
c1t14d0s2 sliced newdg02 newdg online
c1t15d0s2 sliced newdg03 newdg online spare

If a spare flag is not set then the vxrelocd deamon uses available space in
the disk group to build the VxVM object. To exclude a disk from use in
hot-relocation, set the nohotuse flag as follows:
# /etc/vx/bin/vxedit -g disk_group set nohotuse=on disk_name

If a disk fails, then remove it from space allocation as follows:


# vxedit set failing=on fail_disk

To rejoin space allocation:


# vxedit set failing=off fail_disk

After space is determined, the vxrelocd deamon uses the vxassist


command with the move option to create the new objects, which retain the
existing names.

Recovery

Recovery is the last step in the hot-relocation process. After new objects
are moved, the vxrelocd deamon calls the vxrecovery command to
recover the data. Two fields are added to the DM record to identify the
original location of the object: the orig_dmname= and orig_dmoffset=
fields. These assist in manually restructuring the original object.

Sun Proprietary: Internal Use Only


Configuring the VxVM Software E-35
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Hot-Relocation

Hot-Relocation Configuration
The vxdiskadm utility has new options which support hot-relocation,
available as of the VxVM software version 3.1.

A sample of the vxdiskadm menu display is as follows:


Volume Manager Support Operations
Menu: VolumeManager/Disk
1 Add or initialize one or more disks
2 Encapsulate one or more disks
3 Remove a disk
4 Remove a disk for replacement
5 Replace a failed or removed disk
6 Mirror volumes on a disk
7 Move volumes from a disk
8 Enable access to (import) a disk group
9 Remove access to (deport) a disk group
10 Enable (online) a disk device
11 Disable (offline) a disk device
12 Mark a disk as a spare for a disk group
13 Turn off the spare flag on a disk
14 Unrelocate subdisks back to a disk
15 Exclude a disk from hot-relocation use
16 Make a disk available for hot-relocation use
17 Prevent multipathing/Suppress devices from VxVM’s view
18 Allow multipathing/Unsuppress devices from VxVM’s view
19 List currently suppressed/non-multipathed devices
list List disk information
q Exit from menus

Alternatively, use the vxedit command to set the spare flag from the
command line. Use the following command:
# vxedit set spare=on disk_name

Sun Proprietary: Internal Use Only


E-36 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Hot-Relocation

Unrelocating
If you try to manually move a relocated subdisk using the vxsd
command, the following message is displayed:
vxvm:vxsd: ERROR: Relocate trace information in subdisk disk04-01 not empty. Use -r to
retain or -d to discard it

You can then decide whether to retain the orig_dmname information or


discard it. The VxVM software version 3.1 provides the vxunrelocate
command for use to restore a relocated subdisk to its original location.

Sun Proprietary: Internal Use Only


Configuring the VxVM Software E-37
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Hot-Spares

Hot-Spares
The hot-spare process is similar to hot-relocation, but there are significant
differences between the two. The primary functional difference is disk
selection in the event of a failure; hot-relocation is able to relocate
subdisks, whereas the hot-spare process relocates entire disks. With hot-
relocation, the event of a failure to a subdisk no longer impacts all
volumes on the physical disk, unless it is the disk that fails. Hot-relocation
is the recovery process available in the VxVM software, starting with
version 2.3.

In the hot-spare process, the /etc/vol/sparelist file is maintained to


determine locations of available spare disks. In the hot-relocation process,
inherent policies determine where subdisks are relocated. The primary
policy for both the hot-spare and hot-relocation processes is to locate
recovered data based on the spare flag and drive locality. The closer the
drive is to the failed drive, the greater the preference.

Comparison of Hot-Relocation and Hot-Spares


While hot-relocation is more flexible than the hot-spare function, it also
has the potential to disrupt carefully designed layouts.

For example, a mirror configuration can be set up across two arrays. In


the event of a failure, if sufficient disk space is not available, then the hot-
relocation process can potentially recover the failed plex across multiple
disks over both arrays in small chunks. If a 1 gigabyte subdisk fails and
the only free space available in the disk group are 100 megabyte chunks
on 10 separate disks, the VxVM software concatenates 10 subdisks
together to form the 1 gigabyte failed subdisk, thus spreading the data out
onto 10 separate disks.

Sun Proprietary: Internal Use Only


E-38 VERITAS Volume Manager Troubleshooting
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B
Hot-Spares

Activating the Hot-Spare Function


Hot-relocation can be unconfigured in the
/etc/rc2.d/S95-vxvm-recover file and the hot-spare function
implemented. To implement the hot-spare function, change
vxrelocd root & to vxsparecheck root & in this file.

Hot-relocation can be configured to use only disks marked as spare. Back


up the /etc/vx/bin/vxrelocd script; then edit the file and change
“spare=yes” to “spare=only” in this script.

Note – Be aware that any upgrade removes these modifications.

Sun Proprietary: Internal Use Only


Configuring the VxVM Software E-39
Copyright 2003 Sun Microsystems, Inc. All Rights Reserved. Enterprise Services, Revision B

Potrebbero piacerti anche