Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
Supporting
BMC Performance Assurance, version 7.5.10 BMC Capacity Management, versions 8.1.00 and 8.2.00
April 2011
www.bmc.com
Copyright 2011 BMC Software, Inc. BMC, BMC Software, and the BMC Software logo are the exclusive properties of BMC Software, Inc., are registered with the U.S. Patent and Trademark Office, and may be registered or pending registration in other countries. All other BMC trademarks, service marks, and logos may be registered or pending registration in the U.S. or in other countries. All other trademarks or registered trademarks are the property of their respective owners. IBM, AIX, WPAR, LPAR, DLPAR, SPLPAR, PowerVM, Power5, Power6 and others are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. IT Infrastructure Library is a registered trademark of the Office of Government Commerce and is used here by BMC Software, Inc., under license from and with the permission of OGC. ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office, and is used here by BMC Software, Inc., under license from and with the permission of OGC. Linux is the registered trademark of Linus Torvalds. Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners. UNIX is the registered trademark of The Open Group in the US and other countries. The information included in this documentation is the proprietary and confidential information of BMC Software, Inc., its affiliates, or licensors. Your use of this information is subject to the terms and conditions of the applicable End User License agreement for the product and to the proprietary and restricted rights notices included in the product documentation.
Customer support
You can obtain technical support by using the BMC Software Customer Support website or by contacting Customer Support by telephone or e-mail. To expedite your inquiry, see Before contacting BMC.
Support website
You can obtain technical support from BMC 24 hours a day, 7 days a week at http://www.bmc.com/support. From this website, you can
s s s s s s s s
read overviews about support services and programs that BMC offers find the most current information about BMC products search a database for issues similar to yours and possible solutions order or download product documentation download products and maintenance report an issue or ask a question subscribe to receive proactive e-mail alerts when new product notices are released find worldwide BMC support center locations and contact information, including e-mail addresses, fax numbers, and telephone numbers
product information product name product version (release number) license number and password (trial or permanent)
operating system and environment information machine type operating system type, version, and service pack or other maintenance level such as PUT or PTF system hardware configuration serial numbers related software (database, application, and communication) including type, version, and service pack or maintenance level
s s s
sequence of events leading to the issue commands and options that you used messages received (and the time and date that you received them) product error messages messages from the operating system, such as file system full messages from related software
Send an e-mail message to customer_support@bmc.com. Use the Customer Support website at http://www.bmc.com/support.
Contents
Chapter 1 Introduction to Virtual Systems 9 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Overview of VMware virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Overview of Microsoft virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Overview of AIX hardware partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Overview of HP virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Overview of Oracle Partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Introduction to BMC Performance Assurance for Virtual Servers . . . . . . . . . . . . . . . . 16 Chapter 2 Implementing Performance Assurance in a VMware Environment
19 20 20 21 21 21 22 22 24 25 30 31 37 37 44 48 50 61 61 64 66 78 88 88 89 91 92
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Benefits of using BMC Performance Assurance in this environment . . . . . . . . . . Installation and configuration considerations for a VMware environment . . . . . . . . Installation notes for a VMware environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Overview of implementation process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Setting up VMware ESX and vCenter Server proxy data collection . . . . . . . . . . . . . . Methods of collecting data from ESX servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Requirements for VMware ESX and vCenter Server proxy hosts . . . . . . . . . . . . . Configure a VMware ESX proxy host . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Configure a vCenter Server proxy host . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Collecting data from a Virtual Machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Viewing results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Displaying VMware in Analyze . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Displaying VMware CPU Usage in Visualizer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Displaying VMware CPU Use in Perceiver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . FAQs for CPU reporting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Modeling a VMware Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Working with VMware systems in Predict . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . View the Predict report for VMware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Performing What if...? investigations of VMware systems . . . . . . . . . . . . . . . . . FAQs for Predict Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Evaluating the virtual environment in Perceiver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Step 1: View overall guest activity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Step 2: Identify guests of interest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Step 3: Review resource utilization for a guest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Step 4: Explore guest activity on a single Virtual Host . . . . . . . . . . . . . . . . . . . . . .
Contents
Chapter 3
97
Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 Support for Microsoft Virtual Servers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 Support for Hyper-V (Microsoft 2008 Virtualization Server) . . . . . . . . . . . . . . . . . 98 Installation considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 Installation considerations for a Microsoft Hyper-V environment . . . . . . . . . . . . 99 Installation considerations for a Microsoft Virtual environment . . . . . . . . . . . . . 100 Managing performance of a Microsoft Virtual Server . . . . . . . . . . . . . . . . . . . . . . . . . 101 Collecting and processing data from a Microsoft Virtual Server . . . . . . . . . . . . . 101 Viewing the results for Virtual Server data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 Viewing Virtual Server CPU Usage in Visualizer . . . . . . . . . . . . . . . . . . . . . . . . . . 108 Viewing Virtual Server data in Perceiver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 Managing performance in a Microsoft Hyper-V environment . . . . . . . . . . . . . . . . . . 110 Setting up data collection for Hyper-V . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 Viewing results for a Hyper-V environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 Viewing Hyper-V CPU usage in Visualizer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 Viewing Microsoft Hyper-V results in Perceiver . . . . . . . . . . . . . . . . . . . . . . . . . . 113 Virtualization Planning support for MS Hyper-V . . . . . . . . . . . . . . . . . . . . . . . . . 126 Chapter 4 Implementing Performance Assurance on an AIX System with Hardware Partitions
127
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 Support for AIX hardware partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 Support for AIX workload partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 Installation Considerations for AIX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 Installation considerations for AIX hardware partitions . . . . . . . . . . . . . . . . . . . . 129 Installation considerations for AIX workload partitions . . . . . . . . . . . . . . . . . . . . 130 Configuration Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 Enable Remote Command Execution on the HMC. . . . . . . . . . . . . . . . . . . . . . . . . 131 Configuring the AIX Environment using the Perform Configuration File . . . . . 132 Manually configuring the AIX Environment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 Verify that the HMC was added to the list of known hosts . . . . . . . . . . . . . . . . . 135 Managing the performance of AIX hardware partitions . . . . . . . . . . . . . . . . . . . . . . . 136 Collecting data from AIX hardware partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136 Displaying data from AIX Partitioned Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138 Viewing AIX partition metrics in Investigate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138 Key Analyze reports to review for AIX partitions . . . . . . . . . . . . . . . . . . . . . . . . . 140 Displaying AIX hardware partition CPU usage in Visualizer . . . . . . . . . . . . . . . 144 Displaying AIX hardware partition CPU use in Perceiver . . . . . . . . . . . . . . . . . . 146 FAQs for CPU reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 How does SMT affect the number of CPUs? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 Where do I start? What view is "correct"? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148 How do I interpret the name of the physical system? . . . . . . . . . . . . . . . . . . . . . . 160 When I look at "per processor" usage, it's very uneven. Why? . . . . . . . . . . . . . . . 160 When I look at "per logical processor" usage, it's very uneven. Why? . . . . . . . . 162 Modeling AIX partitioned systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 Evaluate the model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
Review the Physical System Summary report . . . . . . . . . . . . . . . . . . . . . . . . . . . . Perform What if...? investigations of AIX partitioned systems . . . . . . . . . . . . Scenario for modeling data from an AIX partition. . . . . . . . . . . . . . . . . . . . . . . . . Additional scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . FAQs for Predict Modeling of AIX partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Managing the performance of workload partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . Collecting and processing data from a workload partition . . . . . . . . . . . . . . . . . Viewing metrics in Investigate from a workload partition . . . . . . . . . . . . . . . . . . Displaying WPAR CPU Use in Perceiver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Displaying AIX partitions with Live Partition Mobility . . . . . . . . . . . . . . . . . . . . General FAQs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . What are best practices for performance reporting and analysis for the AIX environment?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chapter 5 Implementing Performance Assurance on HP Systems with Hardware Partitions
165 167 168 170 171 181 181 188 190 191 193 193
197 199 199 200 200 200 200 201 201 201 207 211 212 213 213 214 215 217 222 225 227 228 228 229 229 231 232 234 235
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Support for hardware partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Support for HP Integrity VM systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Installation considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Installation considerations for HP-UX partitions . . . . . . . . . . . . . . . . . . . . . . . . . . Installation considerations for HP Integrity VM systems . . . . . . . . . . . . . . . . . . . Managing the performance of HP-UX partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Data collection considerations for HP hardware partitions . . . . . . . . . . . . . . . . . Displaying data from HP hardware partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . Modeling HP-UX virtual systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Overview of HP Integrity VM systems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . HP IVM Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Data collection considerations for HP IVM systems . . . . . . . . . . . . . . . . . . . . . . . Displaying data from HP IVM partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Viewing HP IVM metrics in Investigate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Viewing HP IVM data in Visualizer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Viewing HP IVM data in Perceiver. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Viewing HP IVM data in Analyze . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Virtualization Planning support for HP IVM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Reporting capacity usage for HP IVM systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Reporting CPU for a HP IVM system. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Reporting CPU allocation for a HP IVM system . . . . . . . . . . . . . . . . . . . . . . . . . . Modeling HP IVM systems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Implementation of best practices for HP IVM modeling and reporting . . . . . . . Evaluate the Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Review the Physical System Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Perform What if...? investigations of HP IVM partitioned systems. . . . . . . . . Modeling process summary of HP IVM systems . . . . . . . . . . . . . . . . . . . . . . . . . . Chapter 6 Implementing Performance Assurance on a Oracle Partitioned System
237
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
Contents 7
Support for Solaris Containers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238 Support for Solaris Dynamic System Domains (DSDs) . . . . . . . . . . . . . . . . . . . . . 239 Support for Solaris Logical Domains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239 Installation considerations for Solaris . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240 For installing in Solaris 10 environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240 For installing in Solaris DSD environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240 For installing in Solaris LDom environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241 Managing performance of Solaris Containers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241 Collecting and processing data from Solaris Container systems . . . . . . . . . . . . . 241 Displaying data in Visualizer for Solaris Container systems . . . . . . . . . . . . . . . . 249 Displaying data in Perceiver for Solaris Container systems . . . . . . . . . . . . . . . . . 251 Finding out more using Investigate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253 Managing performance of Oracle partitioned (DSD) systems. . . . . . . . . . . . . . . . . . . 257 Displaying Data in Analyze from Oracle Partitioned Systems . . . . . . . . . . . . . . . 257 Modeling Data from a Oracle Partitioned Environment . . . . . . . . . . . . . . . . . . . . 257 Managing performance of Oracle Logical Domains . . . . . . . . . . . . . . . . . . . . . . . . . . . 259 Displaying data in Investigate from Oracle LDoms . . . . . . . . . . . . . . . . . . . . . . . . 259 Displaying Oracle LDom CPU usage in Visualizer . . . . . . . . . . . . . . . . . . . . . . . . 259 Displaying Solaris LDom CPU Use in Perceiver. . . . . . . . . . . . . . . . . . . . . . . . . . . 260 Modeling data from a Oracle LDom environment . . . . . . . . . . . . . . . . . . . . . . . . . 262 Managing performance with Chip Multi-Threading . . . . . . . . . . . . . . . . . . . . . . . 265 Modeling an UltraSPARC processor. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266 Chapter 7 Implementing Performance Assurance on Xen Systems 269
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269 Support for Xen nodes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270 Benefits of using BMC Performance Assurance in this environment . . . . . . . . . 270 Installation and configuration considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271 Collecting and processing data from Xen systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271 How do I collect and process the data? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271 Viewing results in Analyze . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279 Viewing results in Perceiver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281 Xen Partition Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281 Xen Host Summary by Resource . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285 Xen Partition Summary by Resource . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287 Modeling Xen servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289 Working with Xen systems in Predict. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289 View the Predict report for Xen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292 Virtualization Planning support for Xen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294 Index 295
Chapter
Overview
Many companies are using virtualization and partitioning technologies to create application servers that can share physical resources to reduce hardware costs, enhance scalability, and simplify systems management. In effect, these are virtual systems that appear to be isolated from other servers that share the underlying hardware. You can create a virtual system by using virtualization software, such as VMware on Intel-based hardware platform, or by using hardware partitions in UNIX environments. In a VMware or Microsoft Virtual Server environment, a physical system hosts a group of logical systems, or virtual machines. Operating systems and applications are configured to run inside a virtual machine that is managed by the Host Server. This approach allows system resources to be dynamically allocated to any of the virtual machines based on need, providing mainframe-class capacity usage and control of server resources. In a typical UNIX hardware partitioned system, each operating system image has hardware associated with it. This hardware includes the CPU, memory, I/O ports, and network interface card. The operating system runs on the partitioned system, and the applications run inside each image of the operating system. The hardware partition essentially provides the separation of the operating system images.
Chapter 1
You can then change the resources allocated to each operating system image by moving the resources between the partitions or reassigning them within the physical machine. Some vendors also provide the capability for dynamic partitions and partitions that share resources. The chapters in this book explain how to implement BMC Performance Assurance in each of the following virtual system environments:
s s s s s
VMware Microsoft Virtual Server and Microsoft 2008 Virtualization Server (Hyper-V) IBM AIX HP-UX Oracle Solaris
TIP
Keep in mind when moving any virtual system from one physical system to another in Predict, that you can move systems only within the same physical system type. For example, a a VMware virtual machine can be moved only to another VMware host, not to an AIX server.
The following sections provide some background information for each of these environments.
10
The VMware vCenter Server (formerly VMware VirtualCenter) provides a scalable platform that enables IT administrators control the virtual environment. It provides virtualization management by centrally managing VMware servers and VMs.
With the introduction of AIX version 5.3, there are three types of logical partitions that can be implemented on servers built on POWER5 (p5) and POWER6 (p6) architecture:
s s s
dedicated processors assigned to AIX Logical Partitions (LPARs) Dynamic Logical Partitions (DLPARs) AIX systems running in shared processor logical partitioned mode (SPLPARs) on IBM p5 and p6 servers with Micro-Partitioning.
With the introduction of AIX 6.1, the software-based Workload Partitions (WPARs) can be created on any p5 and p6 hardware that supports the operating system.
Partition type
DLPAR
Dynamic logical partition Introduced with AIX 5.2, a DLPAR configuration enables the addition or removal of resources (for example CPUs, adapters, or memory) without rebooting the operating system or partition. This configuration improves system availability as well as resource utilization. As with an LPAR, the hardware resources in a DLPAR are divided into partitions where the resources associated with the partition are used only by that partition. Each DLPAR must have at least one CPU, 1GB memory, and one I/O adapter. This partition type is dynamic: the allocation of resources to the individual partitions can be changed on demand, while the AIX system image is running.
12
Table 1
SPLPAR
Partition type
Chapter 1
13
Overview of HP virtualization
Overview of HP virtualization
On HP systems, there are various types of partitions: physical partitions (nPars), virtual partitions (vPars), and Integrity Virtual Machines (Integrity VMs).
Multiple nPars running on a server, with each nPar running one instance of the HP-UX operating system. Multiple vPars running directly on a server, with each vPar running one instance of the operating system. Multiple nPars running on a server, some of which have vPars configured. Each non-virtual-partitioned nPar runs one instance of the HP-UX and each vPar in the nPar runs one instance of the operating system.
14
In both the nPar only and vPar examples, all of the partitions are actual systems that can be started and stopped. In the vPar on an nPar scenario, the nPar is no longer a discreet system, as the resources it contains are allocated to the virtual partitions; it cannot run an instance of the operating system. The following table lists the BMC Performance Assurance support for each hardware partition platform.
Hardware Platform Supported for vPar PA-RISC Itanium
s s
Not supported
HP-UX 11.31
Integrity VMs
HP Integrity VM is a virtualization technology that provides operating systems isolation and supports shared processors. A single HP Integrity server running Integrity VM can support multiple guest instances on the server, each with its own separate guest operating system. Integrity VM runs only on HP Integrity servers, and supports several operating systems as guests (for example, HP-UX, Microsoft Windows Server, and RedHat Linux).
Hardware partitions known as Dynamic System Domains (DSDs), which are implemented on the SunFire architecture of servers. Partitions known as Containers, or zones, available on Solaris 10 systems. Solaris LDom
Chapter 1
15
Resources can be re-allocated across individual DSDs without rebooting the partitions, enabling mission-critical applications to continue running while the computing environment is being upgraded. BMC Performance Assurance supports Solaris Dynamic System Domains as implemented on the SunFire architecture (SunFire 12K, 15K, 20K, and 25K) with a System Controller running System Management Services software version 1.3, 1.4.1 or later, under a limited support agreement. Since the SunFire architecture provides a variety of site-specific configurations, BMC Software will provide this support on a case-by-case basis with the specified customer.
Solaris LDoms
Logical domains (LDoms) enable you to create virtual machines that take advantage of the large thread scale offered by the CoolThreads server platforms. With LDoms, the logical processors are associated with threads and threads are associated with specific cores. With LDoms you can create up to 128 virtual servers on one server. Logical domains speed deployment, reduce disk capacity requirements, and create higher availability domains (using redundant virtual networks and disks). You can also reallocate resources such as CPU, virtual disks, and virtual networks on the fly to meet the demands of the workloads and services running in the LDom.
16
BMC Performance Assurance is the premiere product for comprehensive performance management of virtualized server environments. BMC Performance Assurance offers the following capabilities in environments that are moving towards virtualization:
s s s
Help identify candidates for virtualization (before deployment) Help identify candidates for consolidation Manage host-system and virtual machine performance (after deployment)
This guide describes how to implement BMC Performance Assurance in the various Virtual Server environment.
NOTE
Analyzing and modeling data collected from the virtual system environments described in this book use the version 7.5.00 BMC Performance Assurance Console on Microsoft Windows. Similar support is available on UNIX consoles.
Chapter 1
17
18
Chapter
This chapter presents the following topics: Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Benefits of using BMC Performance Assurance in this environment . . . . . . . . . . Installation and configuration considerations for a VMware environment . . . . . . . . Installation notes for a VMware environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Overview of implementation process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Setting up VMware ESX and vCenter Server proxy data collection . . . . . . . . . . . . . . Methods of collecting data from ESX servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Requirements for VMware ESX and vCenter Server proxy hosts . . . . . . . . . . . . . Configure a VMware ESX proxy host . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Configure a vCenter Server proxy host . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Collecting data from a Virtual Machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Viewing results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Displaying VMware in Analyze . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Displaying VMware CPU Usage in Visualizer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Displaying VMware CPU Use in Perceiver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . FAQs for CPU reporting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Modeling a VMware Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Working with VMware systems in Predict . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . View the Predict report for VMware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Performing What if...? investigations of VMware systems . . . . . . . . . . . . . . . . . FAQs for Predict Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Evaluating the virtual environment in Perceiver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Step 1: View overall guest activity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Step 2: Identify guests of interest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Step 3: Review resource utilization for a guest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Step 4: Explore guest activity on a single Virtual Host . . . . . . . . . . . . . . . . . . . . . . 20 20 21 21 21 22 22 24 25 30 31 37 37 44 48 50 61 61 64 66 78 88 88 89 91 92
Chapter 2
19
Overview
Overview
With BMC Performance Assurance, you can manage the performance of VMware ESX servers. You can analyze performance information for the virtualized systems (Windows and Linux) and obtain a system-level view of the VMware ESX server for server sizing, management, and daily reporting. BMC Performance Assurance provides the ability to collect VMware ESX server configuration and system-wide statistics, analyze this data with the Analyze component, and report on the data using the Investigate and Visualizer components. For an overview, refer to Overview of VMware virtualization on page 10.
NOTE
Analyzing and modeling data collected from the virtual system environments described in this book use the version 7.5.10 BMC Performance Assurance console on Microsoft Windows. Similar support is available on UNIX consoles.
20
Select a Microsoft Windows server to act as a VMware proxy host computer. Run the installation program according to the instructions in BMC Performance Assurance Getting Started guide, installing the Perform Agent on the selected Microsoft Windows server. Configure the VMware proxy host according to the instructions in the Configure the VMware ESX proxy host in Perform Agent on page 28 section and in the Implementing Proxy Data Collection appendix in the BMC Performance Assurance Getting Started guide.
WARNING
Do not install the Perform Agent on the VMware service console operating system (Red Hat Linux).
Chapter 2
21
22
Table 2 on page 23 shows the differences in the two methods and demonstrates the advantages of using the vCenter Server to collect data. BMC Best Practices recommends using only vCenter Server to collect data from ESX servers. Table 2
Feature Configuring collection
vCenter Data Collection Requires you to configure a user account with read privileges on vcenter server only.
Requires you to configure a user account with read privileges on every ESX server from which you want to collect data. In addition, requires you to configure a user account with read privileges on vcenter to collect cluster and resource pool details.
Availability
Host and guest metrics will be available as long as the host is up and running. Collection is not impacted by the connectivity to vCenter except for cluster and resource pool information. The default is 1 minute for host/VM and 15 minutes for cluster/pool. Performance metrics for Cluster and resource pool are available only if the vCenter is at level 3 or above.
Depends solely on the connectivity to vCenter. No data will be collected if vCenter is down; there is a single point of failure. The default is 15 minutes. Collection from vCenter is not affected by the level set in vCenter. All metrics are available.
Collection granularity vCenter 'Level' for data collection Supported metrics Supportability metrics
Same set of configuration and statistical Same set of configuration and statistical metrics as vCenter data collection. metrics as ESX host data collection. No specific supportability metrics available.
s
Collector will record vCenter connection failure status in the 'collector information' metric group. Collector records ESX host status to the 'VMware Host Configuration' metric group. Data availability metric in udr provides the number of seconds for which data is available for the spill interval In the database , field "collector_type" in the Agent table identifies the kind of collection (ESX or vCenter).
Chapter 2
23
Table 2
Feature
Very limited additional system resources needed BMC recommends limiting to collecting to a maximum of 100 VMware hosts per proxy host. Investigate - Sampling interval of 1 minute will make the data refresh much slower than OS collectors that have 10 sec sampling PrintUDR - works as before Analyze & Predict - works like other platforms , no change to the workflow
Very limited additional system resources needed BMC recommends limiting to collecting to a maximum of 150 VMware hosts per proxy host. Investigate - not recommended for real-time monitoring s The default of 15 minute sampling is not very useful for the real-time investigate use case PrintUDR - works as before Analyze & Predict - works like other platforms, no change to the workflow. Must have 7.5.10 console to process vCenter data.
Use Cases
s s
s s
WARNING
The only supported data collection method for VMware ESX 3.5/4.0 and 3.5i/4.0i is a VMware ESX or vCenter Server proxy host. Local installation on a VMware ESX 3.5/4.0 Service Console is not supported due to a known defect in the VMware Perl Scripting API, which prevents data collection using a locally installed Perform Agent on VMware ESX 3.5/4.0. There is also a known memory leak in the VMware Perl Scripting API.
Verify the version of VMware Verify the version of Microsoft .NET Framework Install and configure the proxy host
24
For a VMware ESX proxy host see, Configure a VMware ESX proxy host on page 25. For a vCenter Server proxy host see, Configure a vCenter Server proxy host on page 30
To grant account privileges on VMware 1 Start the VMware Infrastructure Client and log on to the ESX server. 2 Select the ESX host. 3 Select the Permissions tab in the right panel. 4 Right-click the context menu and select Add Permissions.
Chapter 2
25
5 In the Assign Permissions dialog box, click Add. 6 Select the account you want to use in the list and click Add. 7 Click OK. 8 Ensure that the account is using at least the Read-Only role to provide adequate
permissions.
To run EsxUser.exe to obtain user accounts 1 When running this tool without any argument, the following is displayed:
Figure 1 esxUser.exe command line window
2 Use command line switches to perform operations on the VMware ESX server:
Table 3
Switch /list /add /grantRead /deprive
26
Table 3
Switch /rm
NOTE
VMware does not recommend changing the vCenter statistics level to 3. According to VMware, "Level 1 has very low overhead on both the VirtualCenter server as well as the ESX Server hosts. Levels 2 - 4 have slightly greater overhead on ESX, but can adversely impact VirtualCenter performance if there are more than 10 ESX Server hosts." For more information see http://www.vmware.com/pdf/vi3_monitoring_statistics_note.pdf.
Cluster Level
s s s s s s
CPU Usage Mem Usage CPU MHz Usage Mem Swap Used Mem Granted Mem Shared
s s s s s
Mem Active Mem Zero Mem Vmmemctl Mem Consumed Mem Overhead
CPU MHz Usage Mem Granted Mem Shared Mem Active Mem Zero
s s s s
Chapter 2
27
2 Click Add New to configure a new domain for proxy collection. The Logical
Domain Setup dialog box appears.
3 In the Logical Domain field, enter the name of the domain from which the proxy
host will collect data. The domain can have any unique name. It is used to group all of the agentless ESX servers together.
4 Select Enable this domain to include the domain in the proxy collection. 5 Specify the following information in the ESX Server Account Information area:
s
User Name: Name of the user account that will be used to manage the ESX
servers. The account must have (at least) Read permissions in order for the VMware VI objects to collect data from the ESX servers. (By default, only root and vpxuser have this privilege on the ESX servers.)
s
Password and Confirm Password: The password associated with the user name.
6 In Agentless ESX Server Configuration, select one of two discovery methods for
locating ESX servers.
s
Use ESX Server List - Uses the list of ESX servers configured in the agentlessComputers.cfg file. For information on populating this file see, the
Preparing for proxy data collection section in the Implementing Proxy Data Collection chapter of the BMC Performance Assurance Getting Started guide.
s
Discover from the VC - Uses the list of ESX servers discovered via a vCenter server if it is defined for the logical domain. If you select this option and do not specify a vCenter, no ESX server will be returned for discovery.
28
NOTE
Regardless of the option you use, you have to define a vCenter in order to collect cluster and resource pool data.
7 Click Apply to save your changes. A message appears notifying you that the user
information has been created in the registry. Click OK. The Edit List button becomes available.
8 Depending on the option you chose in step 6, click Edit List if you want to:
s
Edit the agentlessComputers.cfg file. For instructions about editing the file, see the Preparing for proxy data collection section in the Implementing Proxy Data Collection chapter of the BMC Performance Assurance Getting Started guide. Specify the vCenter Server machine that will supply the list of ESX servers and the data related to clusters and resource pools. For further instructions, see the following procedure, To specify a vCenter Server machine.
9 Click Exit to close the Logical Domain Setup dialog box and return to the VMware
Proxy tab. The new domain appears in the Logical Domain Setup list.
NOTE
In7.5.10 no more than one vCenter can be defined for any logical domain, while in 7.5.00 multiple vCenters can be defined for a logical domain.
10 You can add additional domains or click OK to exit PATROL - Perform Agent. A
message appears asking you to wait while the agent is being restarted. A second message appears stating that the restart was successful. Click OK again. PATROLPerform Agent Setup closes. If for any reason it is not successful, you must restart the application manually (from Start => Settings => Control Panel => PATROL - Perform Agent).
To specify a vCenter Server machine 1 From within the Logical Domain Setup dialog box, click Edit List. 2 In Virtual Center List Setup dialog box you can specify one vCenter Server
machine for each logical domain. Enter the following account details:
s
Virtual Center: Name of the vCenter Server machine from which you want the list of ESX servers and data related to clusters and resource pools. User Name: Name of the user account used to access vCenter Server.
Chapter 2
29
Password and Confirm Password: The password associated with the user name.
3 Click Add to save your settings. A message appears asking you to wait while the
account information is being verified. A second message appears informing you that the account information has been created. Click OK.
1 Click the vCenter Proxy tab in PATROL - Perform Agent. 2 Click Add New to configure a new domain for proxy collection. 3 In the vCenter Setup dialog box, select Enable this vCenter for data collection to
include vCenter Server in the proxy collection.
4 Specify the following domain information in the vCenter Account Information area:
s
vCenter Name: Name of the vCenter Server machine from which the proxy host will collect data. User Name: Name of the Windows domain or local account that will be used to access vCenter Server. Password and Confirm Password: The password associated with the user name.
30
5 (optional) Select Use the Configuration File instead of the vCenter to Discover and
Collect Data from the ESX Servers, to collect data only from those ESX servers listed in the agentlessComputers.cfg file. (See the Preparing for proxy data collection
section in the Implementing Proxy Data Collection chapter of the BMC Performance Assurance Getting Started guide).
6 (optional) Click Edit Configuration File to open the agentlessComputers.cfg file for
editing.
NOTE
BMC Best Practices recommends using vCenter Server to discover and collect data from the ESX servers.
7 Click Apply to save your changes. A message appears asking you to wait while the
account information is being verified. A second message appears informing you that the account information has been created. Click OK.
8 Click Exit to return to the vCenter Proxy tab. The new vCenter Server proxy
appears in the vCenter Setup list.
NOTE
The vCenter Server name listed here is treated the same as the domain for a Windows or VMware ESX proxy.
9 You can add additional vCenter Servers or click OK to exit PATROL - Perform
Agent. A message appears asking you to wait while the agent is being restarted. A second message appears stating that the restart was successful. Click OK again. PATROL-Perform Agent Setup closes. If for any reason the restart is not successful, you must restart the application manually (from Start => Settings => Control Panel => PATROL - Perform Agent).
Chapter 2
31
The following tasks describe the recommended process for collecting and processing data from VMware systems.
Task Create a policy that includes the VMware Host Server. Set up an Automator script. Set up and schedule the Manager script. Verify the data collection and processing. Refer to page 32 page 33 page 34 page 36
NOTE
The tasks described in the following sections are for the Windows version of the BMC Performance Assurance console. Similar support is available for the UNIX console. For instructions on how to complete these tasks using the UNIX console, see BMC Performance Assurance Implementation Guide for UNIX.
1 In the Investigate snap-in, right-click Policies and select New Policy. This starts the
New Policy Wizard.
2 On the Welcome page, enter a name for the policy and accept the default location
for the console data repository. The default location is C:\Program Files\BMC Software\Patrol3\BEST1\NTC\Repository. Click Next. This starts the New Computer wizard.
3 Make sure discovered from network is selected in the drop-down list, and then click
the Discover/Add button.
4 On the Discover Computers window, specify an IP address range and click Start
Discovery.
The discovery process could take a while depending on the number of computers in your network. After it is complete, you will have a list of all computers within the specified IP address range. You can sort by column headings. For example, if you only want to collect data from computers with the 7.5.10 Agent installed, click the Agent Version column head. This sorts the discovered computers by Agent version. You can then easily select computers with the 7.5.10 Agent for collection. Alternatively, you can select non-7.5.10 computers and delete them from the list.
32
5 Click the Agent Version column to sort by Agent type. 6 Highlight the computer you are interested in and click the Add button. To select
more than one computer, hold down the Shift key when making your selections.
TIP
If you have installed the Perform Agent and system collector on one or more VMs, group the Host Server and other instrumented VMs in the same policy.
7 Click OK. 8 On the Computers to Add to Policy window, click the check-boxes to select the
individual computers from which to collect data.
TIP
Run your script against a small or duplicate database first, previewing and testing it before running it against the production database.
For more information, refer to the Visualizer online Help or the Visualizer Implementation Guide.
1 From the Automator File menu, choose one of the default scriptname.b1a files to
open the Select a Database from the Catalog dialog box. New Daily Script (daily.b1a) New Weekly Script (weekly.b1a) New Monthly Script (monthly.b1a)
2 Select one of the databases from the Database selection list. When you click OK,
Visualizer creates a script and names it with capital letters.
Chapter 2
33
For example, if you select demo.b1a, after you click OK, the newly created script is named DEMO.B1A.
3 Review the events in the script and modify as necessary. To change an event,
highlight the event line in the script. Then do one of the following: Choose Edit => Modify. From the keyboard, press Alt+Enter.
4 Choose Run => Preview Mode. 5 Choose Run => Script Now. Click Yes when you see the following prompt.
Preview mode is enabled C the database will not be altered in any way. Are you sure you want to preview the script: scriptname now?
6 Review the log and correct any errors in the script. 7 From Automator, choose File => Save As.
The script is now available to be included in the Manager script.
1 Right-click the Manager snap-in, right-click Scripts, and choose New Script from the
menu. This starts the New Script Wizard. Read the Welcome page and click Next to proceed to the Manager Script File Name page.
2 Enter a name for your script. Use the default folders provided, unless you have set
up alternative locations for script files and reports. Click Next.
3 On the Operation window, select Collect New Data. Leave the default selection of
System and applications data.
4 Click Add to browse to the policy that has the computers you identified in Task 1:
Set up a policy that includes the target VMware systems on page 32. On the Select Policy File window, select the policy file and click Open.
34
5 On the Operation window, click Next. The Analyze and Predict window is
displayed. The Analyze and Predict window lets you decide how you will use the data that you collect. You can generate reports for Analyze, Predict, and Applications (if you selected that option on the previous window). You can also generate Visualizer files and create workload characterization files.
7 On the Collect or Analyze Interval window, use the up and down arrows in the
From and To fields to establish the collection interval.
BMC Software strongly recommends that you enable a gap of five to ten minutes between the current Collect request end time and the next collect request start time to allow the existing VMware collectors to exit completely before the new collectors start in the Manager run. An example you could specify a Manager run from 12:00 a.m. to 11:55 p.m. This prevents intense resource contention that can cause the collector to fail.
8 Select the time zone, and then click Next. Changing the time zone setting might be
necessary if data is being collected from computers that are in time zones that differ from the time zone of the console computer.
9 On the Automator Script Option window, click Browse. Use the Automator Script
Option page to specify the Automator script to be executed during the Manager run. Automator is a Visualizer tool that runs user-built scripts to populate a Visualizer database. The Automator script you specify will use Visualizer files you generated with Predict and Analyze to populate the Visualizer database. (See the Visualizer Implementation Guide for information about using Automator scripts.)
Chapter 2
35
10 On the Select Automator File dialog, browse to the Automator script you created in
Task 2: Set up the Automator script on page 33, and then click Open.
11 On the Automator Script Options window, click Next. 12 On the Schedule a run of the script window, select the Schedule the script to run
checkbox. Click the up and down arrows to set the time to run the script. Set the recurrence options for the script in the Run script section.
1 In the BMC Performance Assurance console scope pane, open the Manager snap-in. 2 Click Status Report.
The Perform - Data Collection Status opens in the results pane. The available Manager scripts are in the left portion of the report. To view the status of data collection, click on a script. You can also click Schedule in the scope pane to display the scripts in the results pane. Then right-click on the script for which you want to view reports and select View Status Report.
36
Viewing results
See the online help, available from the help button on the top right portion of the Status Report, or refer to the appendix on Troubleshooting Tools in the BMC Performance Assurance Getting Started for more information about using these reports.
Viewing results
You can access performance data about the VM sessions by using any of the following components, using the Perform Microsoft Windows console:
s s s s
For Investigate, refer to the online Help to create drill-downs on the VMware sessions or the Host Server. For more information, refer to the chapter on VMware metrics in BMC Performance Assurance System Metrics for Microsoft Windows and UNIX, Volume Two. The following sections describe how to display and interpret the data using Analyze, Visualizer, and Perceiver.
VMware Summary VMware Memory VMware Disk VMware Network VMware Per Processor
In these reports, each VM is identified by its display name used by VMware, and the guest operating systems are also displayed. Each operating system instance, including the VMware service console operating system, is shown as an independent computer. For more information about these reports, refer to the Analyze online Help.
Chapter 2
37
First, you must create a workload characterization file that includes the VMware service console, and optionally the virtual machines, as computers. An Analyze workload characterization contains parameters for
s s s s
Creating a performance model for use by Predict Viewing charts of performance data Creating performance reports Creating input files for Visualizer
When you run Analyze, it uses the workload characterization to evaluate collected performance data, and create a performance model, reports, and, optionally, Visualizer input files. To generate the VMware reports, perform the following steps:
1 Right-click Analyze in the Tree view, and choose Open Workload Characterization. 2 Choose the appropriate workload characterization (.an) file for the VMware
system.
3 Review the displayed interval or set another interval. 4 Run Analyze by clicking on the green arrow in the menu bar, or right-click on the
Workload Characterization in the Tree view and choose Start Analyze.
5 Expand the Reports item and then expand the VMware selection.
The following sections discuss the various VMware reports.
38
Figure 2
The following table describes each of the columns in the report. You may need to scroll to the right to see some of the report columns.
Column VMware Console Name Display Name Description The name of the VMware service console that provides all of the management functions for the VMs. For reporting purposes, this name is used to refer to the physical host machine. This value is used to refer to the VM. Each VM is identified by its display name specified in the VMware configuration file. The display name is not guaranteed to be unique on the VMware server. BMC Performance Assurance does not support duplicate VM display names within a VMware server. VMs with the same display name are treated as one. The operating system type installed as a guest OS on the VM. Each VM can run Microsoft Windows or Linux as guest OSs. The number of physical processors on the ESX server system. This value is shown only for the top-level server information. The number of logical processors on the ESX server system. This value is shown only for the top-level server information. The number of CPUs allocated to the VM. This value is shown for each VM. The total reported overhead usage for all VMs, due to virtualization. The CPU utilization attributed to the VMware host on behalf of the VMs.
Chapter 2
39
Description The percentage of CPU time used by a VM. The CPU usage shown in this column is a physical value; it represents the total CPU usage of the amount of CPU that has been allocated for the VM. The number of CPU shares assigned to a VM. The guaranteed minimum CPU percentage reserved for this VM. The maximum CPU percentage for a VM. The valid range of values is from zero (0) to the number representing the total physical CPU resources. The maximum may be greater than 100 for SMP virtual machines that may use more than one full physical CPU. The percentage of time during the measurement session that the VM was ready to process transactions, but could not get scheduled to run on a physical CPU. The percentage (out of 100%) of the maximum physical CPU resource allocated to the VM that it is currently using. The amount of RAM (in MB) for the physical machine. This value is shown only for the top-level VMware service console row. The memory required for single copy of memory shared between VMs.
Swapped Memory The sum of all currently swapped memory for running VMs. This value is equivalent to swapout-swapin. Guest Memory Config Amount of memory allocated for the VM (in MB).
40
Figure 3
Memory Shares Shared Memory Memory Used Overhead Memory Memory Swap In Memory Swap Out
Swapped Memory The amount of memory swapped into and out of the VMFS partition swap file for a VM.
Chapter 2
41
The following table describes each of the columns in the report. You may need to scroll to the right to see some of the report columns.
Column VMware Console Name Display Name Description The name of the VMware service console that provides all of the management functions for the VMs. For reporting purposes, this name is used to refer to the physical host machine. This value is used to refer to the VM. Each VM is identified by its display name specified in the VMware configuration file. This display name is not guaranteed to be unique on the VMware server. BMC Performance Assurance does not support duplicate VM display names within a VMware server. Virtual machines with the same display name are treated as one. The host target logical unit number (LUN) for the disk. The number of disk shares assigned to a VM on the target device specified by the host target LUN. The number of disk reads (per second) for a VM on the target device specified by the host target LUN. The amount of data read (per second) for a VM on the target device specified by the host target LUN.
42
Description The number of disk writes (per second) for a VM on the target device specified by the host target LUN. The amount of data written in KB (per second) for a VM on the target device specified by the host target LUN.
Figure 5
Chapter 2
43
Figure 6
1 Click Database=>Select and select the database into which you populated the
VMware data.
2 Click Database=>Detail and choose the data for the graph. Ensure that you select
ESX from the Virtual Type drop-down list.
44
4 Double-click the row representing the service console to see the bar chart that
shows the physical CPU usage, by VM, on the server.
Chapter 2
45
Another graph to look at is the VMware CPU by Host/VM graph. An important metric to view in this report is %Host.
1 From the Graphics menu, choose VMware => VMware CPU by Host/VM graph.
Figure 8 VMware CPU Use Hierarchy
2 Right-click the graph and select the %Host view to see the view of CPU
consumption across all virtual machines, relative to the capacity of the entire system.
3 From the Graphics menu, select VMware => VMware Cluster/Host. This graph
shows the resource consumption of all the hosts running under a cluster for the selected set of clusters.
46
Figure 9
VMware Cluster/Host
4 From the Graphics menu, select VMware => VMware Cluster/Host/VM. This graph
shows the resource consumption of all the hosts running under a cluster and all the VMs running under that host for the selected set of clusters. Figure 10 VMware Cluster/Host/VM
Chapter 2
47
5 From the Graphics menu, select VMware => VMware Cluster/Resource Pool/VM.
This graph shows the resource consumption of all the Resource pools configured under a cluster and all the VMs running in that resource pool for the selected set of clusters. Figure 11 VMware Cluster/Pool/VM
48
Figure 12
If you select a host, you get the following table and charts:
Metric Description Displays the physical CPU capacity used by each VM as a percentage of the physical processors in the server and normalized to a maximum of 100%.
CPU Utilization by VM
CPU User by VM
Displays percentage of CPU used by each VM when the processor is executing in user mode, calculated based on the number of physical processors in the host. Displays the percentage of CPU used by each VM when the processor is executing in system mode, calculated based on the number of physical processors in the host. Displays the percentage of (virtual) CPU used by each VM in this host calculated based on the number of Virtual CPUs configured for the VM. Displays the number of virtual processors that are configured for all the virtual machines for the selected host. Displays the host capacity with respect to spec rating for the selected VMware host.
CPU System by VM
Virtual CPU Capacity Used Virtual Processors Configured by VM CPU Rating Used
For more information on Perceiver views and metrics, see the Evaluating the virtual environment in Perceiver, section in this guide, refer to the online Help, or the BMC Performance Perceiver Getting Started Guide.
Chapter 2
49
The first perspective is the only view which allows you to see the entire environment and assess overall capacity consumption. Any of the VMware Visualizer graphics can be used to display this view. Analyze and Predict VMware Summary reports contain similar information, as do the VMware views in Perceiver. For CPU reporting reports and views already described in this chapter see:
s
Analyze s Analyze VMware Summary Report on page 39 Predict s Predict VMware Summary report on page 65 Visualizer s VMware Host/VM Graph on page 45 s VMware CPU Use Hierarchy on page 46 s VMware Cluster/Host on page 47 s VMware Cluster/Host/VM on page 47 s VMware Cluster/Pool/VM on page 48 Perceiver s CPU Profile from the VMware Host Summary by Resource on page 49
The number of processors being used as the host capacity is the number of physical processors. The number of logical CPUs are also reported (e.g. Hyperthreaded_Processors), but are not used as a basis for available CPU capacity. The number of physical processors can be viewed in a couple of VMware graphics, but again, here's how this information is displayed in the customized VMware CPU Use Hierarchy graphic shown in Figure 8 on page 46. The 400 %Physical_Max indicates that 4 Physical_Processors (7.5 equivalent is Host Processors) are configured throughout this day. And this maximum applies to the entire host, and thus is the overall perspective applied to each virtual machine.
50
Figure 13
However, the VMware CPU Use Hierarchy does not show how the individual VMs are performing. The overall physical system view shows relative consumption of the CPU by each, but it doesn't relate it to their allocations of CPU capacity. Thus, another view is needed for this, describing how much is each VM is using with respect to its allocated capacity. Again, the customized VMware CPU Use Hierarchy graphic can be used to see how much of each VM's total virtual processing (%Virtual_Used or in the non-customized graphic, %VProc) capacity is in use. See Figure 14 on page 52.
Chapter 2
51
Figure 14
An integrated view of the VMware host using Visualizer is also available, simultaneously showing both allocationsPhysical processors (total for the system) and Virtual processors (for that virtual machine)and what portion of the allocated capacity is in use. Metrics of interest in this graph (Figure 15 on page 53) are % Virtual and % Host. See also, Figure 37 on page 90. which shows similar information in a Perceiver view.
52
Figure 15
Another useful view is across all virtual machines within a host system. This graphic, VMware CPU VM % Capacity on page 54, shows total physical processor capacity utilized, and average virtual utilization. metrics are % Virtual and % Host. Also, using this graphic over time, either for individual virtual machines or for the host system as a whole, provides a complete overview of what's happening and what the long-term trends are. Of course you can drill down as appropriate to find out what the underlying cause is for a significant change in utilization. Similar charts are available in Perceiver, including Virtual CPU Capacity Used chart on page 94.
Chapter 2
53
Figure 16
Details of how CPU performance is affected by the particular combination of resources, VM resource demands, and VM configuration parameters are reported by the customized VMware CPU Use Hierarchy graphic (see VMware CPU Use Hierarchy on page 51) as well as the default Visualizer graphic, VS, VMware CPU Use Hierarchy.
54
Figure 17
In this graph:
s
%Ready is the percentage of time during the measurement session that the VM was
ready to process transactions, but could not get scheduled to run on a physical CPU Degrd_VM is the percentage of CPU response time due to competition from other VMs Degrd_All is the percentage of CPU response time due to competition within the VM and between this VM and other VMs
In this example, the degradation percentage is very small, so there is very little wait introduced due to competition for CPU resources. This information is also presented for 7.5 Visualizer databases (via the VMware CPU Host/VM graphic), with some differences:
s
% Ready has been normalized to 100%, so that VMs with different virtual processor
configurations can be directly compared with each other s the underlying CPU measurements are highlighted in VMware CPU States Cluster/Host/VM graphic s values for individual VMs can be displayed for one (or more) hosts using
VMware CPU by VM % Ready
s
values for individual VMs can be displayed for one (or more) clusters using Degradation reporting requires a console patch and updated vgd file
s
Chapter 2
55
NOTE
The degradation percentages are calculated by Predict, so will not appear in these graphics if Predict response times have not been requested as part of Manager processing. These percentages are also output during interactive modeling,
VMware CPU Use Hierarchy on page 46 VMware Cluster/Host on page 47 VMware Cluster/Host/VM on page 47 VMware Cluster/Pool/VM on page 48
In Perceiver see:
s s s s s
VMware Cluster Summary - Cluster Overview VMware Cluster Summary - Resource Usage by Host VMware Cluster Summary - Resource Usage by Virtual Machine VMware Resource Pool Summary - Resource Pool Overview VMware Resource Pool Summary - Resource Usage by Virtual Machine.
Why is there a difference between Cluster CPU and Memory Utilization reported by 7.5.10 and the 7.5.00 views?
Beginning with Release 7.5.10, metrics are available directly for the cluster and are presented in:
s
VMware Capacity Cluster: reports cluster CPU and Memory configured and used, as well as remaining capacity projections in %, number of additional VMs, amount of additional work that could be supported; also reports the average VM resource consumption VMware Balance Cluster/Host: reports how well-balanced the CPU and Memory load is across hosts within the cluster; also reports the average VM resource consumption
56
Cluster utilization is obtained directly from the VMware API, which documents that cluster utilization includes all of the VM activity, but no "overhead" activity . This would be the same as the results obtained from summing all of the VMs, such as shown in:
s
VMware CPU Cluster/Host/VM : reports VM use and configuration, aggregating to both the host and cluster levels (examples shown above); reporting as %, GHz, processors, queue length
However, resource utilization metrics can also be reported for the cluster by summing the activity for the hosts in that cluster, as shown in the following views11:
s s
VMware CPU Cluster/Host: reports host and cluster CPU configured and used VMware CPU Capacity Cluster/Host: reports host and cluster CPU usage as %, GHz, SPEC, as well as remaining capacity projections
This method depends on having all hosts within the cluster properly instrumented with collectors, but has the advantage that both VM and overhead activity is accounted for. So any report based on the directly available values for the cluster (or the sum of the VMs) will understate current total utilization and thus overstate available capacity. The exact amount can be determined by using any of the 7.5.00 views, and comparing them with the 7.5.10 views. For example, the summed view reports almost 50% cluster CPU utilization where the directly available cluster metric only reports about 30% CPU utilization. Figure 18 Summed view vs. directly available cluster metric
Chapter 2
57
In addition, you can report on exactly what the breakdown is between overhead and the VMs using other alternatives such as VMware CPU Cluster/Host: Overhead Processors or VMware Virtualization Overhead Cluster/Host: % CPU. Host utilization obtained by summing the VMs would be similarly affected.
NOTE
Remember that the host-level (for example, outside) view is only available from reports and graphics labeled with "VMware", and that all other reports and graphics show the view from inside the virtual machine(s) which have been instrumented with a system collector . In general, the metrics from inside a virtual machine and at the host level (e.g. outside) will not agree due to the virtualization considerations described above.
When interpreting performance metrics from within a virtual machine, here are a few pointers:
58
(a) if there is very little competition for resources among virtual machines, the data will represent reality - if there is a lot of competition, data accuracy will be affected The classic example of this is that if a virtual machine requires 25% of a CPU, but there are other virtual machines competing with it, the virtual machine will report 100% CPU utilization (instead of 25%), because it is using the CPU during all of the time allotted to it. (b) when viewing process data directly or indirectly (via transactions/workloads), process metrics may be differentially affected by virtualization. This means that as competition among virtual machines increases, some processes will have their CPU more overstated than other processes . And this means that precise breakdown of CPU utilization across workloads cannot be determined. You can still identify which applications are running in the VM, just not exactly how much resource each is consuming. Figure 19 on page 59 shows an example of comparing the outside and inside view for selected guest on an ESX server. The top graphic shows the outside view, i.e. how much of the total host CPU is utilized, which is about 40% across all guests. The bottom graphic shows the inside view, which is what does the guest OS think its utilization is, which is 100% for two guests, and around 5% for the other two guests. The two guests showing 100% for the inside view are using around 10% in reality. Figure 19 Comparison 1 - outside and inside view for selected guest
Chapter 2
59
Figure 20 on page 60 shows more outside views of what's really happening. The heaviest usage VMs are not coming close to their virtual processor allocation, using only about 20 and 30% of virtual capacity (top graphic). But competition for CPU is clearly evident in the bottom graphic, which shows %Ready as around 75% for both VMs. Figure 20 Comparison 2 - outside and inside view for selected guest
Analyze: VMware Per Processor report Visualizer: VMware CPU Host per Processor and VMware CPU Host % per Processor graphics Perceiver: VMware Views, VMware ESX per CPU Statistics
The most likely explanation is that Processor 0 is used exclusively for System Services, and if that work consumes a lot of CPU, the relative CPU utilization of Processor 0 will reflect that.
60
How does Hyper-Threading affect the number of CPUs? Will I see double the number of processors I really have?
Hyper-threading does double the number of processors reported on under VMware. BPA presents the total number of processors as logical processors. Note also that VMware is instrumented with virtual-aware CPU accounting for Hyper-threading, so the sum of all processor time does not exceed the number of physical processors. However, CPU utilization for all other performance objects (VM, ESX host, ESX cluster) should always be viewed with respect to physical processors, which is presented consistently in the graphics shown in Where do I start with CPU reporting? on page 50. CPU queue length is also expressed relative to the number of physical processors. The Predict model always presents the physical processor view.
Chapter 2
61
Figure 21
The host view is prefixed with VH to identify it as the Virtual Host. Any VMs which are instrumented (have the Perform Agent and collector installed on the OS) are displayed as standalone computers.
verify or correct the information in the model identify any resources that might be exceeding your service level objectives. The guideline and threshold apply only to the VMware ESX Host Server.
TIP
A model containing a VMware ESX Host Server and its associated VMs is much like a model containing physical machines. You can change the VMware host or VM configuration settings (such as CPU, memory, CPU shares, and so on), prior to baselining.
For more information on calibrating the model and creating a valid baseline, refer to the Predict online Help.
62
Type of system you are viewing VMware Host Server Virtual machines
Key attributes shown on the Configuration tab for that system type Host name (using the VMware console OS name), CPU model, the number of physical processors, and memory size. Display name, number of processor configured, memory size configured, CPU share and measured CPU utilization (in X*100% where X is number of processors configured). The VMware Host Server name is also displayed.
Figure 22 shows the Properties for a VMware Host Server. Figure 22 Properties page for a VMware system
For either the VMware host or the VM itself, you can click the Virtual System Allocation button on the Configuration tab to see how the system resources are being allocated across the VMs, as shown in Figure 23. You can reassign the physical system resources among the virtual machines using this dialog box.
Chapter 2
63
Figure 23
64
Expand the Reports tree in the scope pane. Expand the Computer tree. Select the VMware Summary report. Predict displays the report in the results pane, as shown in Figure 24. Predict VMware Summary report
Figure 24
The following table describes the columns in this report. You may need to scroll to the right of the report to see some of these columns.
Column VMware Host Description The name of the VMware service console that provides all of the management functions for the VMs, prefixed by VH. For reporting purposes, this name is used to refer to the physical VMware Host Server. This value is used to refer to the VM. Each VM is identified by its display name specified in the VMware configuration file. This display name is not guaranteed to be unique on the VMware server. The model number of the hardware. The hardware vendor. For the VMware Host, this is the total number of physical processors. For Virtual Machines, this is the number of virtual processors.
VM System
Chapter 2
65
Description The CPU utilization attributed to the VMware Host Server (specifically the System Services console which runs on the host). For the VMware Host, this is the percent of CPU utilization out of online processors * 100. For the Virtual Machine, this is the percent of CPU utilization out of physical online processors * 100, with a maximum value of the number of virtual processors * 100. For the VMware Host this equals the online processors * 100. The total disk I/O for the VMware Host is the sum of the I/Os for all the Virtual Machines running on that host. For the VMware Host this is the total physical memory. For the Virtual machines, this is the total virtual memory. The memory for the VMware Host is not the sum of the virtual memory of the Virtual Machines. The percentage of CPU wait time due to all virtual machines (including itself) on the VMware ESX Host Server. The percentage of CPU wait time due to activity from other virtual machines.
Util
Predict also creates a Visualizer file that contains the VMware performance results. See the Visualizer online Help for descriptions of the VMware reports and for information on working with them. A few key items to review in the report:
s
The VMware Host total line shows the total physical processors expressed as a percentage (for example, 400% translates to four physical processors). Therefore the CPU Utilization reflects the total across all VM, relative to a maximum of 400% in the example of four processors. The Processor value for each VM shows the number of virtual processors, which translates to 100% for one processor. The Overall and Other VM Response Time Wait percentages show the effect on CPU response time due to competition among VMs. The higher the percentage for these values, the more degraded the performance is as compared to a system running a native OS with dedicated resources. The System SVC field shows the amount of CPU consumed by the System Services console, which services the entire Virtual Host system.
For more information on modeling and What if...? analysis, refer to the Predict online Help or the BMC Performance Assurance for Microsoft Windows Implementation Guide for Microsoft Windows. The following section explores a sample What if...? scenario involving VMware systems.
NOTE
VMware virtual machines can be moved or assigned only to another VMware Host Server. Converting a standalone computer to a VM is not supported, but workloads from a standalone computer can be moved to a VM.
Example scenario: How do I take existing work and move it to a new Virtual Host?
In this example, a company is looking to reduce the number of dedicated redundant servers by managing the failover of their 24 physical servers to a single ESX Server. They have decided they want the ability to have up to four servers simultaneously failover to one ESX Server, but do not know how big the VMware ESX Server and individual virtual machines need to be. The following example demonstrates how BMC Performance Assurance can be used to accurately size an ESX Server to accommodate the workloads from four physical machines. For general information on workloads, baselines, and models, see the online Help or the BMC Performance Assurance for Microsoft Windows Implementation Guide for Microsoft Windows.
Collecting, Analyzing, and Modeling the Data to Size the VMware ESX Server 1 Collect performance data from the machines of interest. In this example
performance data was collected from all of the 24 physical servers. To understand the performance of the systems and capacity, you need to continually collect performance data. In this example data was collected over a period of time which included the periods of peak demand. This peak demand period relates to the four most highly utilized servers in the server farm.
TIP
For details on collecting data using the Perform Agent and Perform collector, refer to Chapter 2 of the BMC Performance Assurance for Microsoft Windows Implementation Guide for Microsoft Windows.
Chapter 2
67
2 Using Analyze, define workloads that represent each of the four machines.
Create separate Analyze workload characterization files for each machine. Workload characterization enables you to aggregate disparate processes from one or more computers into a logical group. These groups (workloads) can define a business application, group of users, or in this example, all the work on a physical computer. You can then use Predict to calculate the throughput and response time for each transaction in the workload. Create an ALL transaction and workload class containing all processes using the .* regular expression. (The ALL workload encapsulates all of the work being done on the physical machine.)
3 After you've collected performance data over a meaningful period and created
workloads for each machine in the study, you can use the Analyze component of BMC Performance Analyzer to identify the period of peak demand. In each workload characterization file (.an), identify the peaks for each physical server, and create a model that represent the worst case behavior, in terms of performance. To do so:
A Select the peak interval (normally a 1-2 hour time period) for each machine in
your Analyze workload characterization file, as shown in the following figure.
TIP
Individual machines encounter peak utilization at different times (intervals). That is why you the need to create multiple workload characterization files.
that provides scheduled and ad-hoc graphical reports on application and system performance, based upon data collected by the Perform Agent and collector.
68
4 Use Visualizer to estimate the size of the target VMware ESX server. With
Visualizer, you can draw performance graphs which enable you to understand the performance of the physical system(s). Since the analysis interval chosen was during peak utilization, any graph drawn in Visualizer reflects peak values. Understanding the required capacity during this peak demand period helps you accurately size and configure the VMware ESX Server. Draw a stacked CPU utilization graph from the four physical machines you want to migrate to the VMware ESX Server as virtual machines. The following figure shows the Stacked CPU chart used to determine peak capacity requirements. Refer to the Visualizer online Help for specific instructions.
TIP
It is important to view the Y axis in SPECINT rating, as opposed to utilization percentage. The SPECINT rating is a way to normalize the different processing power of different CPUs. This example uses SPEC INT RATE 2000. At peak demand a SPECINTRATE2000 of 4.5 serves as the peak value, meaning the CPUs we choose for our new VMware ESX Server must collectively have a minimum SPEC INT RATE 2000 value of 4.5 to adequately support these workloads.
Chapter 2
69
6 Create a new model with a VMware ESX Server and a staging computer to serve as
a container for the saved workloads from the four physical machines.
70
A Create a new model by right-clicking Predict and selecting New model. The New
model wizard walks you through the steps of creating a new model. In this example, the new model is named Sizing.
B Create the computer to serve as a container for the saved workloads from the
four physical machines. To do so, right-click the Computer object in the tree and selecting New Computer. The New Computer wizard walks you through the steps of creating the computer. This computer will be used as a container for the saved workloads from your physical machines. In this example, the new computer is named Staging.
C Create a new VM host system by right-clicking the Computer object in the tree
and selecting New VMware Host System, as shown below.
The New VMware Host System wizard walks you through the steps of creating the new system.
s
In this example, on the VMware Host System Name page, the system is named VMwareHost_1 and the Performance rating selected is SPEC CINT 2000 RATE.
NOTE
BMC Performance Assurance supports multiple performance ratings. Ensure the performance rating used in Analyze and Visualizer matches the performance rating in Predict.
On the Model Selection page of the wizard, choose the CPU model. It is important to select CPUs that have a SPEC INT RATE 2000 performance rating that is greater than or equal to what was identified in the Visualizer Node CPU Utilization graph. In this example, the CPU model you choose for your ESX server must have a SPEC CINT RATE (performance rating) of at least 4.5 to match the rating shown in the Visualizer graph in step 4 on page 69.
Chapter 2
71
On the VMware Host System Configuration page, the system has a total memory configuration of 2048 Mbytes, and two online processors. After you have created the new VMware host system, you can verify the configuration of the system by right-clicking the VMwareHost_1 computer and selecting Properties.
1 Create virtual machines (VMs) on the VMwareHost_1 server for each physical
system.
A Right-click the VMwareHost_1 computer, and select New Virtual System. The
New Virtual System Wizard is displayed.
B Click Next on the welcome page. The General Information page is displayed. C Specify a name, a brief description, select the operating system, and click Next. D Complete the system configuration information for the new VM, as shown
below, and click Finish.
TIP
Do not be concerned about configuring each VM with the correct amount of virtual processors, memory, or CPU shares at this point in the process. That information is unknown at this time. Only after modeling the ESX Server and virtual machines, running the four workloads, will you identify the optimum configuration, given your performance objectives.
72
2 Import the saved workloads to the Staging computer. Right click the Workloads
object in the Sizing model, and select Import Workloads. On the Import Workload dialog, select the target computer (in this example the Staging system), the four workloads in the Objects in Library window, and click OK, as shown in the following figure.
Chapter 2
73
The four workloads are imported into the newly created Sizing model, and are contained in the Staging system. They will eventually be moved to the individual VMs that were just created on VMwareHost_1.
3 Move the workloads to the individual VMs. For example, to move the workload
PhyNode_1 to the virtual machine VM VirtualNode_1 running on ESX Server VMwareHost1:
A Right-click the Workloads in the model, and select Move workload. B On the Move Workload Computer Options dialog, select Virtual System. C On the Move Workloads to Virtual System dialog, choose the target VMware
host, the target VM, and the utilization adjustment and click OK.
D Repeat the process for each of the workloads you want to move. 4 Evaluate the model to ensure that performance is acceptable. In this example,
right-click the Sizing model and choose Evaluate Model.
When reviewing the VMware Summary Report in Predict pay special attention to the values in Overall Response Time Wait column and the Other VM Response Time Wait column. Overall Response Time Wait is the percentage of CPU wait time due to all virtual machines (including itself) on the ESX server. Other VM Response Time Wait indicates the percentage of CPU wait time due to activity from other virtual machines. Comparing overall wait time and wait time due to other VMs helps you identify if you have balanced the VMs on a the ESX Server properly. The CPU shares and utilization both impact the wait time.
74
TIP
Focus more on the relationship between the two values and not the values themselves. Look for indications that a VM is monopolizing the server:
s s
If you see a VM with high overall wait time but low wait time due to others, it is the dominant VM on the server. On an under utilized system, this may not be an issue. If you see a VM with high overall wait time and high wait time due to others, then there is another VM dominating the system. In this case, you may want to move that dominant VM to another server, so that other VMs can receive better service.
Ultimately, the competition between virtual machines and the work inside the virtual machines affect your overall transaction response time.
What to do next
After you have reviewed the report, consider the following course of action:
s
If the wait time for the VM is significantly higher than the other VMs on the server, use the virtual machine properties page to reallocate the number of virtual processors and CPU shares until performance is acceptable. To access this dialog, right-click the VM under the Computers object, and select Properties. For example, the Overall Response Time Wait column and the Other VM Response Time Wait column in the Predict VMware Summary report may have indicated that the VM spends a significant amount of time waiting for resources. The CPU shares and CPU utilization settings on the virtual machine properties page both impact these wait time statistics. The number of CPU shares assigned to a VM defines whether or not the VM has a higher or lower priority to CPU resources than the other VMs. The more shares a VM has the more CPU cycle opportunities the VM has at its disposal. The fields in the CPU Utilization group box control the minimum and maximum amount of CPU resources that can be allocated to the VM. By increasing these percentages, in conjunction with increasing the number of CPU shares, you can decrease the amount of time the VM will spend waiting for resources.
Chapter 2
75
If the overall wait time and wait time due to other VMs in the Predict VMware Summary report indicated that the VMs are not properly balanced on the ESX Server properly, use the Virtual System Allocation dialog to reallocate resources for all of the virtual machines. You may have found that one of the VMs is over allocated in terms of CPU shares, and this is affecting the performance of the new VM you are modeling. To access this dialog, click the Virtual System Allocation button on the Configuration tab of the VMware Host Server.
If the performance is still unacceptable, you may need to consider changing the configuration of the ESX Server (for example, to upgrade or downgrade CPU) or moving the application to a different server. To access the VMware Host System properties page, right-click the VM Host Server under the Computers object, and select Properties.
76
What would happen to performance if I move a virtual machine from one VMware server to another, to see the effect on CPU utilization and response time wait. What would happen to performance if I move a virtual machine from one VMware server to another, to see the effect on CPU utilization. What if I move an existing workload from an existing stand-alone system to an existing virtual machine? What if I create a new VMware server by specifying server name, CPU model and system service utilization, and move an existing virtual machine there? What would happen to overall performance? What if I upgrade a CPU on the VMware host machine? What effect would it have on response time?
Chapter 2
77
The outside view is prefixed with VH to identify it as the Virtual Host. Any virtual machines which are instrumented are displayed as "regular" computers (for example, "tstcvm1a" is the System Services console). Even though both views are displayed simultaneously, they are not linked; that is, if a change is made to tstcvm1a, it will not be reflected in performance results for VHtstcvm1a (and vice versa). So as a practical matter, typical Predict usage would focus on either the Virtual Host or one or more virtual machines, but not both within the same modeling session.
78
In order to check that the Virtual Host has been properly identified, check that the VMware host computer has an appropriate Identified Model. You will also see the fully identified VH in Predict, by observing the VH configuration (Properties => Configuration for Windows console; Nodes => Edit => Configuration for UNIX console). Figure 27 Virtual Host Configuration
Chapter 2
79
A CPU identification issue which is unique to VMware is that the System Services console runs Linux, classified as UNIX for hardware table purposes, and the Virtual Host is running VMware, which is classified as NT (i.e. Windows) for hardware table purposes. The reality is that both are running on the same physical hardware, but the Analyze/Predict hardware table lookup process would look for the same CPU hardware benchmarked on both UNIX and NT. But almost no systems are benchmarked on both operating systems, so the following message will be issued when Analyze/Predict establishes that the same performance rating will be used for both:
RNING-ALTPERFRAT No performance rating available for cpu model XEON_MP@3660 vendor INTEL operating system UNIX. Alternate performance rating of operating system WINDOWS for this cpu-model will be used.
If you wish to avoid seeing this message, a new hardware table entry can be made for the User Hardware table by modifying the existing one. First, the original hardware table entry:
CREATE CPU-MODEL XEON_MP@3660 VENDOR INTEL DESCRIPTION Copied from ProLiant ML570G3 XeonMP 3.66GHz 1MBL2 667MHzbus 1core/chip CATEGORY system MIN-PROCESSORS 1 MAX-PROCESSORS 4 PROCESSORS 1 PERFORMANCE-RATING SPECINT2000 1386 NT PERFORMANCE-RATING SPECINTRATE2000 16.1 NT PROCESSORS 2 PERFORMANCE-RATING SPECINTRATE2000 30.6 NT PROCESSORS 4 PERFORMANCE-RATING SPECINTRATE2000 57.1 NT # Source : Equivalence of XEON_MP@3660 with HP ML570G3@3660 made by Support # Date : February 2006
Then the new entry which needs to be added to the best1user.hrw file:
MODIFY CPU-MODEL XEON_MP@3660 VENDOR INTEL PROCESSORS 1 PERFORMANCE-RATING SPECINT2000 1386 UNIX PERFORMANCE-RATING SPECINTRATE2000 16.1 UNIX PROCESSORS 2 PERFORMANCE-RATING SPECINTRATE2000 30.6 UNIX PROCESSORS 4 PERFORMANCE-RATING SPECINTRATE2000 57.1 UNIX # Source : Equivalence of XEON_MP@3660 with UNIX and NT ratings made by Support
80
In the previous example reports, there is no published rating on spec for a XEON_MP@3660, so both a User Hardware Table entry and User Translation Table entry are required (in addition to the optional entry which avoids the message). Please contact Customer Support for assistance with CPUs which are not published by spec.org, and thus are not in the default Hardware Table.
Chapter 2
81
The VMware Host total line shows the total physical processors expressed as a % (% Utilization out of), e.g. 400% translates to 4 physical processors, and the CPU Utilization there is the total across all virtual machines, relative to a maximum of 400%. The Processor value for each VM shows the number of virtual processors, which should be translated to %, e.g. 100% for 1 processor. Also shown are Overall and Other VM Response Time Wait % , which show the effect on CPU response time due to competition among VMs. The larger the %, the more degraded performance is as compared with running a native OS with dedicated resources. System SVC shows the amount of CPU consumed by the System Services console, which services the entire Virtual Host. This is displayed as a separate virtual machine in Visualizer rather than as an embedded part of the Host in Predict.
NOTE
Remember that all of the traditional interactive Predict reports (for workloads, relative response time, response time breakdown, etc.) show performance results for virtual machines which have been instrumented with system collectors only (see What about the view from inside a Virtual Machine? on page 58).
How does Predict "alert" for Virtual Hosts and Virtual Machines?
The traditional computer provides resources for workloads which generate a particular amount of CPU utilization, and the CPU Utilization objectives for that computer specify when alerts should be generated in Predict, e.g. yellow alert at 70% utilization and red alert at 80% utilization per processor. A Virtual Host works identically to the traditional computer. Virtual Machines do not have individual performance objectives, but are messaged when they no longer have adequate resources to complete their work:
WARNING-CUTMAXVS CPU on Virtual System 564d53ef49e2010a863db86dfa5da0d5 is exceeding its MAX-CPU-UTILIZATION and the model is evaluated with a cutback on its CPU Utilization.
The report shows the tstcmdsvr1 virtual machine at 100% utilization, its maximum due to its configuration with 1 virtual processor.
82
Figure 29
The overall Virtual Host is not constrained yet (i.e. 178% out of 400%), so it would be possible to satisfy the VMs requirements by increasing its virtual processor allocation from 1 to 2. Figure 30 VMware Host Properties
Chapter 2
83
How do I use Scenario Planner to change the amount of work in a Virtual Machine?
Scenario Planner works similarly for Virtual Hosts as it does for a traditional computer. Overall growth can be specified, and that will affect all virtual machines. In this example, zero overall growth is specified. Figure 31 Growth Rates
Alternatively, individual virtual machine growth may be specified (similar to workload growth for traditional computers). In this example, a single VM's requirements will grow by 500, then 1000%.
84
Figure 32
The results are displayed in the Period-Predict VMware Summary report within Scenario Planner. Other Scenario Planner reports display results for individual VMs which have been instrumented with system collectors (see What about the view from inside a Virtual Machine? on page 58).
A complex workload structure just makes for a lot of work, so it should be avoided. Build baseline model(s) containing each computer which has work on it that is being moved to VMware .
2. Select the workloads to be moved to a Virtual System A. Workloads, right-click a workload, Workload Export (Windows console) B. Workloads, select a workload, File -> Save to Library (UNIX console)
Chapter 2
85
This step should be repeated for each baseline model built in Step 1. 3. Work with a baseline model of an existing Virtual Host. 4. Add any new Virtual System(s) to a Virtual Host as desired . If work is to be moved to existing Virtual System(s), this step can be skipped. A. Computers, right-click a VMware Host -> New Virtual System (Windows console) B. Nodes, Edit -> Create -> VMware Virtual System (UNIX console) 5. Windows console only , right click Workloads -> Move workloads. A. Select Virtual Systems. B. To select workloads to move either 1. Use ADD Workloads from Library to import workloads that have been exported from other models 2. Otherwise check the box next to each workload you want to move C. Select the VMware guest you want to move work to (under VMware System Available) D. Click Add>> 6. Optional addition for "Move workload" step (above) is to add any desired overhead, (use the Utilization adjustment field in the Move Workload dialog). The % is added in proportion to the work being moved to the virtual system, e.g. 100% utilization adjustment will double the utilization of the incoming workload. 7. Evaluate the model Remember that after a workload has been moved, its work is combined with all of the other work for that Virtual System, and cannot be specifically tracked as a separate workload. The workload remains in the model, but no longer generates any resource requirements, so appears as follows.
86
Figure 33
Figure 34
Chapter 2
87
NOTE
BMC Performance Perceiver includes predefined views for many virtualization platforms. The following example illustrates some aspects of the support for the VMware virtual environment.
In addition to overall VMware host resource usage, the resource usage for each of the guests is reported. As with stand-alone systems, the views provide system identification, configuration and hardware performance rating information for the VMware systems. These details are critical in supporting the sizing and consolidation requirements. The following sections walk you through an example of how you might use these predefined views to assess the overall performance of the virtual environment in your data center.
1 In the task pane, click the VMware Virtual Machine Overview category. 2 Select one of the available views. This example uses Virtual Machine Overview. 3 In the selector pane, select a group from the Group menu.
In this example, the Viewer Administrator has created a group called DataCenter2, that includes all the VMware hosts and guests systems in the data center.
4 In the selector pane, select a time interval from the Time menu.
This view displays the average distributed values for the computers in your data center by metric. It also includes a ranking table that shows where the computers being measured fall in increments of 10 percentage points, based on a blended value of the essential metrics (Physical CPU Capacity Used, Virtual CPU Capacity Used, IO Rate, Memory Utilization, Network Rate).
88
Figure 35
The distribution charts in this example show that while guest CPU utilization seems to be low overall, there are a few guests with very high CPU utilization. The charts also indicate that there may be a growing problem with memory utilization.
Chapter 2
89
In this example, the top four guests shown in the table are displaying high memory utilization, with the next two being above guidelines, and the top two guests are displaying both high virtual CPU and high memory utilizations. After these guest systems are identified in the ranking table, use the Virtual Machine Metrics view to see a more complete picture of the overall resource utilization. The Virtual Machine Metrics view (Figure 37) displays multi-server charts for the essential metrics (Physical CPU Capacity Used, Virtual CPU Capacity Used, IO Rate, Physical Memory Used, Network Rate) in your data center, by group of VMs. Figure 37 Virtual Machine Metrics view
From these charts, we see that there appears to be sufficient physical CPU capacity in this group. However, it is apparent that the guest resource utilization for both virtual CPU and memory is not balanced among the virtual hosts available in the data center. The next step is to isolate the virtual hosts with high guest resource utilization.
NOTE
The Network rate chart is not shown in the Figure 37 example.
90
1 In the task pane, click the Virtual Machine Summary by Resource view category. 2 Select one of the available views. This example uses CPU Profile. 3 In the selector pane, select a group from the Group menu. This example uses a
group called VM Systems.
4 Select engrhes50vms5 in that group from the Computer menu. 5 In the selector pane, select a time interval from the Time menu.
The CPU Profile view displays key CPU-related metrics and system identification and configuration information for the individual guests in your data center. Figure 38 Virtual Machine Summary by Resource - CPU Profile view
This profile shows that the guest is using all of the virtual CPU allocated to it for long periods of time, but is using less than 10% of the physical CPU capacity of the Host Server.
Chapter 2
91
Another key piece of information from this view is the name of the VMware Host Server on which it runs (shown in the VM System Identification section) and the information shown in the VMware Host System CPU Configuration section.
6 Click Memory Profile to see the memory utilization statistics for the guest.
Figure 39 Virtual Machine Summary by Resource - Memory Profile view
Consumed memory for the guest is high, over 90% at times, while shared memory is around 4.5%, and balloon memory (amount of reclaimed memory available) is near zero. The memory swapping rate is also near zero. The next step is to look at the overall resource utilization for the VMware Host Server on which this guest is running. From the VM Host System CPU Configuration section shown in the CPU Profile (Figure 38), we see that the Host Server is lex-rdesx-06.
1 In the task pane, click the VMware Host Summary by Resource category. 2 Select one of the available views. This example uses the CPU Profile. 3 In the selector pane, select a group from the Group menu. This example uses a
group called DataCenter2.
4 Select lex-rd-esx-06 in that group from the Computer menu. 5 In the selector pane, select a time interval from the Time menu.
The CPU Profile view displays key CPU-related metrics and system identification and configuration information for the individual guests in your data center. Figure 40 Virtual Host Summary by Resource - CPU Profile
This view confirms that the guests identified earlier are the largest consumers of the CPU resources on the VMware Host Server. Zooming in on the Virtual CPU Capacity Used chart shows the activity for every guest on the Host Server.
Chapter 2
93
Figure 41
Because several of the guests running on this Host Server were also identified as high memory utilization consumers on the Virtual Machine Overview (Figure 35 on page 89), click the Memory Profile view. This view shows the following graphs:
s s s s s s s
Memory Utilization by VM Memory Overhead by VM Zero Memory by VM Balloon Memory by VM Memory Swapping Rate by VM Memory Swapped by VM Memory Configured by Partition The Memory Utilization by VM graph (Figure 42) shows that the guests that are large CPU and virtual memory consumers are not consuming a large percentage of the memory resources on the VMware Host Server. It also shows that the overall memory consumption for the Host Server is around 50%.
94
Figure 42
Based on this examination, the problem does not appear to be related to the capacity of the VMware Host Server, but rather with the resource allocations for a few of the guests on that Host Server. Your options for a next step include:
s
changing the resource shares for the guests, to have a greater percentage of the physical resources on the VMware Host Server available to them. using the Virtualization Planning feature to conduct a guest rebalancing study.
s s s s s
VMware Cluster Summary - Cluster Overview VMware Cluster Summary - Resource Usage by Host VMware Cluster Summary - Resource Usage by Virtual Machine VMware Resource Pool Summary - Resource Pool Overview VMware Resource Pool Summary - Resource Usage by Virtual Machine.
Chapter 2
95
96
Chapter
This chapter presents the following topics: Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 Support for Microsoft Virtual Servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 Support for Hyper-V (Microsoft 2008 Virtualization Server) . . . . . . . . . . . . . . . . . 98 Installation considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 Installation considerations for a Microsoft Hyper-V environment . . . . . . . . . . . . 99 Installation considerations for a Microsoft Virtual environment . . . . . . . . . . . . . 100 Managing performance of a Microsoft Virtual Server . . . . . . . . . . . . . . . . . . . . . . . . . 101 Collecting and processing data from a Microsoft Virtual Server . . . . . . . . . . . . . 101 Viewing the results for Virtual Server data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 Viewing Virtual Server CPU Usage in Visualizer . . . . . . . . . . . . . . . . . . . . . . . . . 108 Viewing Virtual Server data in Perceiver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 Managing performance in a Microsoft Hyper-V environment . . . . . . . . . . . . . . . . . . 110 Setting up data collection for Hyper-V . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 Viewing results for a Hyper-V environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 Viewing Hyper-V CPU usage in Visualizer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 Viewing Microsoft Hyper-V results in Perceiver . . . . . . . . . . . . . . . . . . . . . . . . . . 113
Introduction
With BMC Performance Assurance, you can manage the performance of Hyper-V (Microsoft 2008 Virtualization Server) and Microsoft Virtual Servers. You can analyze performance information for the virtualized systems (guests) and obtain a systemlevel view of the Host Server for server sizing, management, and daily reporting.
Chapter 3
97
s s
collect Hyper-V specific metric groups and metrics (see tables for the list of metrics collected) analysis of key metrics review key metrics in BMC Performance Perceiver Datacenter Overview charts and detailed metrics in Hyper-V specific charts conduct CPU modeling of Hyper-V systems
You must collect data from the Microsoft 2008 Virtualization Server operating system that represents the Host Server (parent partition) by installing the Perform Agent on the parent partition or preferably using proxy collection from a Windows proxy host.
NOTE
Analyzing and modeling data collected from the virtual system environments described in this book use the version BMC Performance Assurance console on Microsoft Windows. Similar support is available on UNIX consoles.
98
Installation considerations
Installation considerations
Installation considerations for a Microsoft Hyper-V environment
To implement BMC Performance Assurance in a Hyper-V environment:
s
Select a Microsoft Windows server (running .NET Framework 2.0 service pack 1 or later) to act as a proxy host computer. Run the installation program according to the instructions in BMC Performance Assurance Getting Started guide, installing the Perform Agent on the selected Microsoft Windows server. Configure the computer to act as a proxy host according to the instructions in the Implementing Proxy Data Collection appendix in the BMC Performance Assurance Getting Started guide.
The proxy host collects data remotely from other Microsoft 2008 Virtualization Servers. This method provides a robust list of metrics for both the host servers and the guests running on those servers. Note the following additional installation considerations for using BMC Performance Assurance in a Hyper-V environment.
s
You must install the Perform Agent on the Microsoft 2008 Virtualization Server operating system that represents the Host Server (parent partition) by installing the Perform Agent on the parent partition, although BMC Software recommends the use of a proxy host. Do not install the Perform Agent on any of the guest operating systems. If you do, then the guest operating system is viewed as a standalone system by BMC Performance Assurance.
Chapter 3
99
Run the installation program according to the instructions in BMC Performance Assurance Getting Started guide. You must install the Perform Agent on the Microsoft Virtual Server host system to obtain system-level information and for accurate system resource analysis and reporting. Do not install the Perform Agent on any of the guest operating systems. If you do, then the guest operating system is viewed as a standalone system by BMC Performance Assurance. To accurately report the guest operating system information, you must ensure that Microsoft Virtual Machine Additions is installed on all guest operating systems. Virtual Machine Additions is a Microsoft Virtual Server feature that can improve the performance of the guest operating system, and the interoperability between the guest and the Host Server.
1 Open the Administration Website. 2 In the navigation pane, under Virtual Machines, point to Configure and then click
the appropriate virtual machine.
3 In Status, point to the virtual machine name, and then click Turn On. 4 After the virtual machine has started, point to the virtual machine name, and then
click Remote Control.
6 After the guest operating system is loaded, press the HOST KEY to release the
mouse pointer, and then in the lower-left corner under Navigation, click Configure
virtual_machine_name.
8 Under Status, point to the virtual machine name, and then click Remote Control.
100
9 Click in the Remote Control window to return to the guest operating system. The
Virtual Machine Additions installation wizard starts. Proceed through the wizard.
10 After the wizard is complete, you will be prompted to restart the virtual machine
to complete the installation.
Virtual Server 2005 Virtual Machine Configuration, which includes the VM configuration file name, the Virtual Server version, display name, guest operating system, server type, power status, the guests universally unique identifier (UUID), memory size, number of virtual CPUs, disk reads, disk writes, network bytes received, network bytes sent. Virtual Machine CPU statistics, which includes the guests UUID, the virtual machine configuration file name, the guest display name, CPU Utilization, minimum CPU allocation, maximum CPU allocation, CPU shares, and uptime. Virtual Server 2005 Host Configuration, which includes the logical CPU number, core CPU number, memory available, and total memory.
Chapter 3 Implementing Performance Assurance in a Microsoft Virtual Environment 101
Virtual Server 2005 Virtual Network, which includes the guest name, Adapter Name, virtual network name, virtual network configuration file name, and various packet and data transfer information. Virtual Server 2005 Virtual Disk, which includes the guest name, disk image name, disk type, disk size, host free disk space, size in guest and size on host.
These metric groups are collected by default. For complete descriptions of these metric groups, see the chapter on Virtual Server Metrics in BMC Performance Assurance System Metrics for Microsoft Windows and Unix, Volume Two.
NOTE
According to the best practices guideline in the Virtual Server 2005 Administrator's Guide from Microsoft, Hyper-threading should be disabled to avoid poor server performance under heavy computing workloads.
NOTE
The tasks described in the following sections are for the Windows version of the BMC Performance Assurance console. Similar support is available for the UNIX console. For instructions on how to complete these tasks using the UNIX console, see BMC Performance Assurance Implementation Guide for UNIX.
Task 1: Set up a policy that includes the target Microsoft Virtual Server systems
To collect data automatically using a Manager script you must have an Investigate policy. The following steps describe the basic process for creating a policy. See the Investigate online Help for assistance with wizard screens or dialog boxes.
102
1 In the Investigate snap-in, right-click Policies and select New Policy. This starts the
New Policy Wizard.
2 On the Welcome page, enter a name for the policy and accept the default location
for the console data repository. The default location is C:\Program Files\BMC Software\Patrol3\BEST1\NTC\Repository. Click Next. This starts the New Computer wizard.
3 Make sure discovered from network is selected in the drop-down list, and then click
the Discover/Add button.
4 On the Discover Computers window, specify an IP address range and click Start
Discovery.
The discovery process could take a while depending on the number of computers in your network. After it is complete, you will have a list of all computers within the specified IP address range. You can sort by column headings. For example, if you only want to collect data from computers with the 7.5.00 Agent installed, click the Agent Version column head sorts the discovered computers by Agent version. You can then easily select computers with the 7.5.00 Agent for collection. Alternatively, you can select non-7.5.00 computers and delete them from the list.
5 Click the Agent Version column to sort by Agent type. 6 Highlight the computer you are interested in and click the Add button. To select
more than on computer, hold down the Shift key when making your selections.
7 Click OK. 8 One the Computers to Add to Policy window, click the check-boxes to select the
individual computers from which to collect data.
WARNING
Make sure you include only the Microsoft Virtual Server host computers in the policy; do not include VMware service console computers in the policy. If you mix the two types of virtual machine computers only the VMware data is processed.
Chapter 3
103
TIP
Run your script against a small or duplicate database first, previewing and testing it before running it against the production database.
For more information, refer to the Visualizer online Help or the Visualizer Implementation Guide.
1 From the Automator File menu, choose one of the default scriptname.b1a files to
open the Select a Database from the Catalog dialog box. New Daily Script (daily.b1a) New Weekly Script (weekly.b1a) New Monthly Script (monthly.b1a)
2 Select one of the databases from the Database selection list. When you click OK,
Visualizer creates a script and names it with capital letters. For example, if you select demo.b1a, after you click OK, the newly created script is named DEMO.B1A.
3 Review the events in the script and modify as necessary. To change an event,
highlight the event line in the script. Then do one of the following: Choose Edit => Modify. From the keyboard, press Alt+Enter.
4 Choose Run => Preview Mode. 5 Choose Run => Script Now. Click Yes when you see the following prompt.
Preview mode is enabled C the database will not be altered in any way. Are you sure you want to preview the script: scriptname now?
6 Review the log and correct any errors in the script. 7 From Automator, choose File => Save As.
104
1 Right-click the Manager snap-in, right-click Scripts, and choose New Script from the
menu. This starts the New Script Wizard. Read the Welcome page and click Next to proceed to the Manager Script File Name page.
2 Enter a name for your script. Use the default folders provided, unless you have set
up alternative locations for script files and reports. Click Next.
3 On the Operation window, select Collect New Data. Leave the default selection of
System and applications data.
4 Click Add to browse to the policy that has the computers you identified in Task 1:
Set up a policy that includes the target Microsoft Virtual Server systems on page 102. On the Select Policy File window, select the policy file and click Open.
5 On the Operation window, click Next. The Analyze and Predict window is
displayed. The Analyze and Predict window lets you decide how you will use the data that you collect. You can generate reports for Analyze, Predict, and Applications (if you selected that option on the previous window). You can also generate Visualizer files and create workload characterization files.
7 On the Collect or Analyze Interval window, use the up and down arrows in the
From and To fields to establish the collection interval.
8 Select the time zone, and then click Next. Changing the time zone setting might be
necessary if data is being collected from computers that are in time zones that differ from the time zone of the console computer.
Chapter 3
105
9 On the Automator Script Option window, click Browse. Use the Automator Script
Option page to specify the Automator script to be executed during the Manager run. Automator is a Visualizer tool that runs user-built scripts to populate a Visualizer database. The Automator script you specify will use Visualizer files you generated with Predict and Analyze to populate the Visualizer database. (See the Visualizer Implementation Guide for information about using Automator scripts.)
10 On the Select Automator File dialog, browse to the Automator script you created in
Task 2: Set up the Automator script on page 104, and then click Open.
11 On the Automator Script Options window, click Next. 12 On the Schedule a run of the script window, select the Schedule the script to run
checkbox. Click the up and down arrows to set the time to run the script. Set the recurrence options for the script in the Run script section.
13 Click Finish to save the script and set the schedule. Task 4: Verify the data collection
Using the Manager component has the added advantage of allowing you to view collection status with the Status Report feature. To access the Status Report:
1 In the BMC Performance Assurance console scope pane, open the Manager snap-in. 2 Click Status Report.
106
The Perform - Data Collection Status opens in the results pane. The available Manager scripts are in the left portion of the report. To view the status of data collection, click on a script. You can also click Schedule in the scope pane to display the scripts in the results pane. Then right-click on the script for which you want to view reports and select View Status Report.
See the online help, available from the help button on the top right portion of the Status Report, or refer to the appendix on Troubleshooting Tools in the BMC Performance Assurance Getting Started guide for more information about using these reports.
For Investigate, refer to the online Help to create drill-downs on the Host Server. For more information, refer to the chapter on Virtual Server metrics in BMC Performance Assurance System Metrics for Microsoft Windows and Unix, Volume Two. The following sections describe how to display and interpret the data using Visualizer and Perceiver.
Chapter 3
107
1 Click Database=>Select and select the database into which you populated the
Virtual Server data.
2 Click Database=>Detail and choose the data for the graph. Ensure that you select
Virtual Server from the Virtual Type drop-down list.
3 From the Graphics menu, choose Virtual Server => Virtual Server Hierarchy.
The following table provides descriptions for some of the key metrics available on this graph. These metrics do not include numbers from other applications running on the
computer outside of the Microsoft Virtual Server. Metric %Total TotaIIO Notes Total percent of Microsoft Virtual Server activity per processor. Total I/O per guest, measured in four kilobytes pages per second. To see the total I/O for the Host Server, click in the total line at the top of the graph. Network usage, in kilobytes per second, attributed by guest system. The net packets transferred per second. The packet size varies by application.
Shows the total quantity, in megabytes, of physical RAM on the Host Server.
Amount of shared memory, in megabytes. This metric does not apply in the Microsoft Virtual Server environment; it applies to VMware only. Amount of memory attributed to overhead, in megabytes. This is the amount of memory attributed to the Host Server on behalf of the guests. This metric does not apply in the Microsoft Virtual Server environment; it applies to VMware only. Amount of memory used for swapping, in megabytes. This metric does not apply in the Microsoft Virtual Server environment; it applies to VMware only.
OvrhdMem
SwapdMem
108
Metric MemUsed
Notes Total amount of memory used in megabytes. This metric does not apply in the Microsoft Virtual Server environment; it applies to VMware only.
MemCnfg
The amount of memory configured for the guest. The memory settings for guests are specified when an administrator configures a guest operating system in a virtual machine, and are stored in the Virtual Machine Configuration (VMC) file. Each guest can have up to 3.6 GB of memory configured, although the available memory for each guest depends on the amount of physical memory on the host.
Chapter 3
109
If you select a Microsoft Virtual Server host, you get the following table and charts for all virtual machines.
Metric Description
Display name Node name IP address OS type OS version Number of virtual processors Amount of CPU shares Amount of allocated memory in MB
CPU Activity by virtual machine IO Activity by virtual machine Network KBytes by virtual machine
Displays the CPU time in seconds used by each virtual computer. Displays the total IO by each virtual computer. Displays the network rate (in KB) of each virtual computer.
The guest operating system in the view is only available if the virtual machine addition is installed on the guest and the guest operating system is running. Microsoft Virtual Machine Additions is an add-on component that enhances the functionality of the guest operating system by providing features including mouse control, heartbeat generator, registry keys, time synchronization, and better performance. For information on installing the Microsoft Virtual Machine Additions component, see Installation considerations on page 99. For more information on Perceiver views and metrics, refer to the online Help or BMC Performance Perceiver Getting Started.
110
Select a Microsoft Windows server to act as a proxy host computer. Run the installation program according to the instructions in BMC Performance Assurance Getting Started guide, installing the Perform Agent on the selected Microsoft Windows computer. Configure the proxy host according to the instructions in the Implementing Proxy Data Collection appendix in the BMC Performance Assurance Getting Started guide. Set up a Manager script to automatically collect, transfer, and process data from the Hyper-V environment. For more information on setting up Manager scripts, see BMC Performance Assurance for Microsoft Windows Implementation Guide for Microsoft Windows.
1 Click Database=>Select and select the database into which you populated the
VMware data.
2 Click Database=>Detail and choose the data for the graph. Ensure that you select
Virtual Host from the Virtual Type drop-down list. Chapter 3 Implementing Performance Assurance in a Microsoft Virtual Environment 111
112
Another graph to look at is the Hyper-V Processor Use graph. From the Graphics menu, choose Hyper-V => Hyper-V Processor Use. Figure 45 Hyper-V Processor Use
Partition Overview
Description: This view displays the average distributed values for the Hyper-V partitions in your data center by metric. It also includes a ranking table that shows where the partitions being measured fall in increments of 10 percentage points, based on a blended value of the essential metrics (Physical CPU Capacity Used, Virtual CPU Capacity Used, IO Rate, Physical Memory Used, Network Rate).
Chapter 3
113
Figure 46
Partition Metrics
Description: This view displays multi-partition charts for the essential metrics (Physical CPU Capacity Used, Virtual CPU Capacity Used, IO Rate, Physical Memory Used, Network Rate) in your datacenter, in line charts by group of partitions. Physical CPU Capacity Used.
114
Figure 47
Partition Overview
Description: This view displays the average distributed values for the Hyper-V root partitions in your data center by metric. It also includes a ranking table that shows where the partitions being measured fall in increments of 10 percentage points, based on a blended value of the essential metrics (Physical CPU Capacity Used, Virtual CPU Capacity Used, IO Rate, Physical Memory Used, Network Rate).
Chapter 3
115
Figure 48
Partition Metrics
Description: This view displays multi-partition charts for the essential metrics (Physical CPU Capacity Used, Virtual CPU Capacity Used, IO Rate, Physical Memory Used, Network Rate) in your datacenter, in line charts by group of root partitions.
116
Figure 49
CPU Profile
Description: This view displays the CPU metrics for each partition in the selected Hyper-V host. The line Total if applicable, displays the value of the metric at the host level. The difference between the stacked partition(s) and the host represents the virtualization overhead for that metric.
Chapter 3
117
Figure 50
Memory Profile
Description: This view displays the memory profile for each partition in the selected Hyper-V host. The line Total if applicable, displays the value of the metric at the host level. The difference between the stacked partition(s) and the host represents the virtualization overhead for that metric.
118
Figure 51
IO Usage
Description: This view displays the IO metrics for each partition in the selected host. The line Total if applicable, displays the value of the metric at the host level.
Chapter 3
119
Figure 52
Processor Profile
Description: This view displays the per processor utilization for the selected host.
120
Figure 53
CPU Profile
Description: This view displays charts for a selected Hyper-V partition designed to provide you with a detailed profile of its CPU capacity activity. This view also includes two tables containing information on partition system identification and CPU configuration of the Hyper-V host.
Chapter 3
121
Figure 54
Memory Profile
Description: This view displays charts for a selected Hyper-V partition designed to provide a detailed profile of its memory usage. The memory usages in this view are reported as a percentage of memory configured for the partition.
122
Figure 55
IO Usage
Description: This view displays charts for IO metrics for a selected VMware virtual machine.
Chapter 3
123
Figure 56
124
Figure 57
Chapter 3
125
Figure 58
126
Chapter
This chapter presents the following topics: Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Installation Considerations for AIX. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Configuration Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Enable Remote Command Execution on the HMC . . . . . . . . . . . . . . . . . . . . . . . . Configuring the AIX Environment using the Perform Configuration File. . . . . Manually configuring the AIX Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Verify that the HMC was added to the list of known hosts . . . . . . . . . . . . . . . . . Managing the performance of AIX hardware partitions . . . . . . . . . . . . . . . . . . . . . . . Collecting data from AIX hardware partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . Displaying data from AIX Partitioned Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Viewing AIX partition metrics in Investigate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Key Analyze reports to review for AIX partitions . . . . . . . . . . . . . . . . . . . . . . . . . Displaying AIX hardware partition CPU usage in Visualizer . . . . . . . . . . . . . . . Displaying AIX hardware partition CPU use in Perceiver . . . . . . . . . . . . . . . . . . FAQs for CPU reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . How does SMT affect the number of CPUs? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Where do I start? What view is "correct"? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . How do I interpret the name of the physical system? . . . . . . . . . . . . . . . . . . . . . . When I look at "per processor" usage, it's very uneven. Why? . . . . . . . . . . . . . . When I look at "per logical processor" usage, it's very uneven. Why? . . . . . . . . Modeling AIX partitioned systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Evaluate the model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Review the Physical System Summary report . . . . . . . . . . . . . . . . . . . . . . . . . . . . Perform What if...? investigations of AIX partitioned systems . . . . . . . . . . . . Scenario for modeling data from an AIX partition. . . . . . . . . . . . . . . . . . . . . . . . . Additional scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . FAQs for Predict Modeling of AIX partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Managing the performance of workload partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 129 130 131 132 134 135 136 136 138 138 140 144 146 147 147 148 160 160 162 164 165 165 167 168 170 171 181
127
Overview
Collecting and processing data from a workload partition. . . . . . . . . . . . . . . . . . 181 Viewing metrics in Investigate from a workload partition . . . . . . . . . . . . . . . . . . 188 Displaying WPAR CPU Use in Perceiver. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190 Displaying AIX partitions with Live Partition Mobility . . . . . . . . . . . . . . . . . . . . 191
Overview
BMC Performance Assurance supports data collection on AIX partitions running on the POWER5 (p5) and POWER6 (p6) servers. This data can be used for analysis, modeling, and Visualizer reporting. You can view performance data by physical system with breakdown by logical systems with either dedicated or shared processors using Analyze, Visualizer, and Perceiver. You can also perform Whatif... analysis of CPU configuration changes using Predict, and include total physical and logical system utilization, which can be used for basic capacity planning exercises on partitioned servers.
NOTE
Analyzing and modeling data collected from the virtual system environments described in this book use the version 7.5.10 BMC Performance Assurance console on Microsoft Windows. Similar support is available on UNIX consoles.
the individual AIX operating system (partitioned or stand alone) level, which provides resource utilization information presented in Analyze and Visualizer reports can be used for reporting, or as input for charge back calculations the server level, based on system configuration data, which lets you analyze and model the full system view
Perceiver provides a number of views that enable you to see performance statistics for both the operating system level and the physical partition level,
128
Investigate support for the WPAR Configuration and WPAR Statistics metric groups. Data Analysis that reads and processes the key metrics, identifies and provides the relationships between objects to the database and the model. Presentation of key metrics in BMC Performance Perceiver views and detailed metrics in IBM WPAR specific charts Modeling of WPAR systems, each WPAR is represented as a workload in the model. Visualizer charts that present WPAR specific views. Collection, analysis, modeling and presentation of WPARs running within IBM hardware partitions (LPAR, DLPAR and SPLPAR).
You must install the Perform collector on a logical system partitions running AIX to gather detailed workload information from the operating system. This ensures proper identification of SMT as well as access to the metrics required to properly report on LPAR resource consumption.
129
For complete installation instructions, refer to the BMC Performance Assurance Getting Started guide.
s
You must also enable the appropriate security access from at least one logical partition to the Hardware Management Console (HMC), so that the collector can gather physical system configuration information. You can place the LPARs in separate Manager policies, but all LPARs must be populated to the same Visualizer database (Visualizer 4.2.06 or later recommended). If you choose to split LPARs from the same physical system across multiple policies, at least one LPAR for each physical system in each policy must be configured to collect the HMC configuration data. To collect Disk Statistics from the AIX partitions, the libperfstat level must be at ML1 for AIX 5.3. To enable this access to the HMC, complete the configuration steps described in the following section.
You must install the Perform Agent in the global zone. You cannot install the Perform Agent inside a non-global WPAR. Visualizer graph support requires version 4.2.05 with latest patch. BMC Performance Perceiver support requires version 7.5.10 and higher.
Configuration Requirements
BMC Performance Assurance collects the configuration data from the HMC remotely, using SSH, which is the only method the HMC allows for remote command execution. This Physical Partition Configuration information collected from the HMC includes information about the HMC, the systems managed by the HMC, and the LPARs in those systems. To enable data collection, however, there are several configuration tasks that you must perform prior to collecting data. These steps are necessary to enable remote execution of commands on the HMC, without being prompted for a password. The configuration tasks are:
130
1 Enable Remote Command Execution on the HMC 2 Configure the AIX environment either manually, or by using the Perform
Configuration file
3 Verify that the HMC was added to the list of known hosts 4 Verify the settings in the Perform SSH Config file
You can perform these tasks manually, or by executing a script provided by BMC Performance Assurance. These tasks are described in the following sections.
1 In the Navigation area, click the HMC Maintenance icon. 2 In the Contents area, double-click the System Configuration icon. 3 In the Contents area, click Enable / Disable Remote Command Execution. 4 Select the Enable ssh check box. 5 Click OK.
For more details, please refer to the System Configuration chapter in the IBM manual HMC for pSeries Installation and Operation Guide.
NOTE
HMC version 4.2.1 or later is a system requirement for BMC Performance Assurance to collect partition configuration information.
131
Use this file to: simulate how BMC Performance Assurance collects the Physical Partition Configuration data from the HMC. This data could then be used to determine if SSH is configured properly for the Perform collector to gather the Physical Partition Configuration data. create the key files in $BEST1_HOME/local/setup/.ssh create an account on the HMC and copy of the public key over to that account This script is designed to be run with the default settings, which will create the public and private key pair, connect to the HMC, create a user role on the HMC called HMCviewer, copy the public key to the HMC and verify that we can successfully collect data from the HMC. To run the script, perform the following steps:
However, you can override these default settings, by executing the script with any of the flags described in the following table:
132
Table 4
Flag -H HMC
-b BEST1_HOME -d keyfile_directory
-D
-h -p password
-r -s -t keytype -u HMC_USER
Removes an account on HMC. This option requires the hscroot password. Disables the creation of the user role on the HMC (that is, use preexisting role). Specifies the type of keypair that is generated. Options are rsa or dsa. Specifies the user name that the Perform collector uses to connect to the HMC. The default is hmcviewer.
NOTE
When you run the script, you are prompted to provide a password if you did not use the -p option. This password is the new password for the new hmcuser being created.
133
Install SSH on the partition (if it is not already installed) Generate the RSA or DSA keypair (the public and private keys) for the partition Enable remote command execution (if not already enabled) on the HMC Create a user with Viewer privileges on the HMC Copy the public key to the HMC account on HMC
The following IBM publications provide more information on using SSH with AIX:
s
Managing AIX Server Farms (IBM Redbook) Effective System Management Using the IBM Hardware Management Console for pSeries (IBM Redbook) Hardware Management Console for pSeries Installation and Operations Guide AIX 5L Support for Micro-Partitioning and Simultaneous Multi-threading (IBM White Paper)
This command returns the SSH version level, and other information. If SSH is not installed, the software to install the SSH is located on the updated AIX Bonus Pack, for AIX 5L Version 5.1 and later. For further information about setting up SSH on AIX, refer to Chapter 4: Secure network connection on AIX in Managing AIX Server Farms. You can also visit the following IBM website for more information on SSH on AIX:
http://oss.software.ibm.com/developerworks/projects/SSHi
134
Verify that the HMC was added to the list of known hosts
Manually generate the RSA or DSA keypair (the public and private keys)
Use one of the following methods to generate the public/private keypair: Use the ssh-keygen utility to generate the keypair. The method for generating keys may be different between various Secure Shell (SSH) implementations. You must be logged on as the user that installed BMC Performance Assurance to create the keypair.
Manually copy the Public Key to the HMC User Account on the HMC
The public portion of the RSA or DSA keypair must be copied to the key file in the users home directory on the HMC. On the AIX client system, use the secure copy command (scp) to move the authorized_keys2 file to a temporary file on the HMC. Log in to the remote server using the ssh command, and then concatenate the transferred user public key file to the $HOME/.ssh/authorized_keys file. For complete instructions, refer to the chapter on Secure network connection on AIX in the IBM Redbook Managing AIX Server Farms.
Verify that the HMC was added to the list of known hosts
Verify that the HMC name has been added to the list of known hosts in the known_hosts file located in the $BEST1_HOME/local/setup/.ssh directory.
135
In the Config file located in $BEST1_HOME/local/setup/.ssh, verify that the settings for IdentityFile and UserKnownHostsFile are correct for your environment. The following list shows the options and default settings in the Config file:
s s s s s s s
One set of data is collected within the AIX partition. It provides a view of the individual AIX operating system (partitioned or stand alone), which contains resource utilization information. The other set of data is collected from the HMC. It provides a global view of all partitions on the physical system, based on the system configuration data.
You can collect data from AIX partitioned systems as you would other computers. These methods are:
s
for ad-hoc data collection, use the best1collect command line executable for automated data collection, create a Manager run using the BMC Performance Assurance console on either UNIX or Microsoft Windows.
These data collection methods are fully described in the BMC Performance Assurance Implementation Guide for UNIX, or by referring to the online Help.
136
HMC_MANAGED_SYSTEM Specifies the name of the partition for which data is to be collected. SSH Specifies the directory where SSH is installed. By default, BMC Performance Assurance searches for SSH in the /usr/bin directory. Use this option if you have SSH installed in a different directory in your environment (for example, in /usr/local/bin, or /opt/SSH/bin).
137
The following sections describe how to display metrics in Investigate, interpret the data using Analyze, and how to display Visualizer and Perceiver graphs.
138
Description For AIX partitioned systems, this metric indicates the partition type. Possible values for AIX are: LPAR, DLPAR, or SPLPAR. Other values are reported for other vendors partitioned systems from other OS vendors, or for standalone servers. For AIX 5.3 systems, this value indicates whether or not the partition is running in Multi-Threading mode. A value of one (1) if the processor is running in Multi-Threading mode.
1 From the HMC, select Partition Properties. 2 Click the Hardware Tab. 3 Click the Processor/Memory Tab. 4 Ensure that the Allow shared processor pool utilization authority check box is
selected.
139
Metric Entitlement
Description For AIX SPLPAR only, this value defines the portion of physical processor resource that is allocated to the partition, along with the defined number of virtual processors. Entitlement is configured in terms of processing units (PU). One (1.0) PU equals the capacity of a full processor. For Uncapped AIX SPLPARs only, this value specifies the uncapped weight of the partition.
Weight
View partition configurations based on physical system and breakdown by logical systems. View system performance statistics based on physical system and breakdown by logical systems. Create a physical system and build the relationship between logical systems and physical system in Predict model. View and model physical processors properly when SMT is enabled, and view physical CPU utilizations for the SMT enabled partitions.
The following sections describe the reports to investigate when analyzing data from AIX partitioned systems.
140
TIP
When Analyze detects a dynamic configuration change to the partition within the analysis interval, in particular a change to the number of processors, Analyze issues a warning message and uses the maximum number of processors value for the whole interval for modeling and reporting. However, Analyze reports the number of processors as a fraction to Visualizer database. This number reflects the fraction of the interval the that the partition had the maximum number of processors.
Figure 59
Active Processors
141
Description The number of dedicated physical processors available to the partition. For physical system, this field is the sum of all partitions. The system vendor and model. Only reported for the physical system. The size of total physical memory (of the physical system) or the size of allocated memory for logical systems. Reported in Mbytes. The percentage of CPU usage (out of 100) times number of active processors. For shared processor pool, it is calculated based on the Shared Pool Idle Time, instead of accumulation of CPU usage of all SPLPARs in the pool. The number of active processors times 100. The logical system CPU utilization relative to number of physical processors shall be reported as out of either:
s s
Out of
Number of dedicated processors for LPAR or DLPAR, or Number of virtual processors for SPLPAR
Multi-Threading
Indicates the type of CPU multi-threading technology, if it is enabled. For IBM AIX partitions, this value is SMT. SMT (simultaneous multithreading) is an enhancement in the POWER5 processor design to allow for improved overall hardware resource utilization. SMT technology allows two separate instruction streams (threads) to run concurrently on the same physical processor, improving overall throughput.
142
Figure 60
All of the following columns, except the Physical System name, apply to the shared partition, not the physical system. The following table describes each of the columns in the report:
Column Physical System Logical System Partition Type Pool ID Description The name of the physical system (the Host Server). The name of the logical system (for SPLPARs). The partition implementation. The ID of the shared processor pool.
Virtual Processors The number of virtual processors (shared pool processors) configured for the partition. CPU Utilization Entitlement The percentage of CPU usage (out of 100) times the number of virtual processors. Indicates the Entitlement value for the SPLPAR. The Entitlement, along with the defined number of virtual processors, defines the physical processor resource that will be allotted to the partition. Entitlement is configured in terms of processing units (PU). One (1.0) PU equals the capacity of a full processor. For example, a SPLPAR with 1.2 PU of entitlement and 4 virtual processors running in a 3-processor pool. Each virtual processor will get 0.3 PU of processing from the 3 processors in the share pool. The percentage of CPU usage out of its Entitlement. This percentage specifies the uncapped weight of the partition. It is valid only for an uncapped SPLPAR.
Entitlement Utilization
143
Description Indicates whether the partition is running in a capped or uncapped mode. Possible values for this column are either Capped or Uncapped. A capped SPLPAR cannot use more PU than it is entitled to. An uncapped SPLAPR can use more PU if there is spare capacity in the shared pool.
Multi-Threading
Indicates the type of CPU multi-threading technology, if it is enabled. For IBM AIX partitions, this value is SMT. SMT (simultaneous multithreading) is an enhancement in the POWER5 processor design to allow for improved overall hardware resource utilization. SMT technology allows two separate instruction streams (threads) to run concurrently on the same physical processor, improving overall throughput.
1 Click Database=> Select and select the database into which you populated the AIX
partition data.
2 Click Database=> Detail and choose the data for the graph. Ensure that you select
PowerVM from the Virtual Type drop-down list.
3 From the Graphics menu, choose AIX Partition => AIX Partition Hierarchy.
144
Figure 61
Another graph to look at is the AIX Partition % Capacity Used graph. From the Graphics menu, choose AIX Partition => AIX Partition % Capacity Used. Figure 62 AIX Partition % Capacity Used
145
After you select a host, the following tables and charts are available.
Metric Host Partition Info Description Displays information of each of the partitions configured on the selected host, including: s Partition name s Node name s Partition type s Pool ID s Amount of entitlement s Number of processors Displays % Physical Used Total, % Virtual Used Average, % Entitlement Used Average and % Dedicated Used Average for the selected host.
146
Metric Top 10 Partition Virtual Capacity Used Top 10 Partition Entitlement Capacity Used System Hardware Configuration
Description Displays the Top 10 Partition Virtual Capacity Used for the selected host. Displays the top 10 Partition Entitlement Capacity Used for the selected host. Provides the system configuration information that is available for the IBM Power VM host, including: s System Model s Host Name s System Description s Processor Model s Processor Description s Processor Clock Rate s Number of Configured Processors s Number of threads per core s Number of cores per socket s Rating s Rating Type s Multi threading type s Amount of Configured Memory
For more information on Perceiver views and metrics, refer to the online Help or the BMC Performance Perceiver Getting Started.
147
NOTE
Although CPU Utilization can be compared between UDR (sum of User and System from CPU Statistics metric group) and Analyzed data, CPU non-utilization; for example, CPU Wait and Idle cannot be directly compared between UDR and Analyzed data.
UDR data Metrics affected by SMT Investigate Perceiver (udr datasource) Metric group/metric label
Analyzed Data Analyzer Predictor Visualizer Perceiver (vis datasource) Analyzer/ Browse or chart CPU Util per CPU data function only All
Number of processors Logical processors (i.e. twice physical) System Configuration/Number of Processors System Configuration/Max Number Processors CPU Statistics/number of processors displayed System Configuration/Processors in Pool System Configuration/Entitlement Physical Partition Configuration/Total Processors Physical Partition Configuration/Active Processors CPU Statistics/any metric: half of Analyze/Predict System Statistics/CPU Utilization CPU Statistics/any metric
Physical processors
CPU Utilization Per Physical Processor Visualizer only Predict alerts only All
Total
NOTE
Beginning with the 7.5.00 console, Visualizer will show the correct total frame utilization only if all LPARS have been instrumented with collectors. See What are best practices' for performance reporting/analysis for this particular environment? for a discussion of data requirements.
148
The first perspective allows you to see the entire environment and assess overall capacity consumption. The CPU Use Hierarchy (or Partition CPU Use Hierarchy in 4.2.2) is a convenient graph to use, and you want to select the %Proc view, i.e. the view of CPU consumption across all LPARs, relative to the capacity of the entire system. Figure 64 Selected CPU Use Data - overall physical system
In this case, the graphic indicates that a maximum of about 25% of the overall capacity is in use. Note the SMT designation in the upper left hand corner - this identifies this as a system with SMT enabled, but as discussed earlier, the number of processors being used here is the actual number of physical processors present, not the number of logical CPUs which are reported on in other tools. The number of physical processors can be viewed in a couple of places, but again, the CPU Use Hierarchy graphic has this information, as shown in Figure 65 on page 150.
149
Figure 65
The 1600 %nPProc indicates that 16 physical processors are configured throughout this day. But how are the individual LPARs doing? The overall physical system view shows relative consumption of the CPU by each, but it doesn't relate it to their allocations of CPU capacity. Thus, the second view is needed for this, which is how much is each using with respect to its allocated capacity. Again, the CPU Use Hierarchy graphic can be used to see how much of each LPAR's total virtual processing (%LProc) capacity is in use, as shown in Figure 66 on page 151.
150
Figure 66
Even the view shown in Figure 66 is not complete when considering SPLPARs, which are further constrained by the specified entitlement, and whether or not the LPAR is capped (i.e. absolutely limited by the entitlement) or is uncapped (i.e. free to exceed its entitlement if resources are available). This relationship can be viewed on a per LPAR basis by simultaneously graphing the %Entlmnt and the %Total CPU usage. Alternatively, multiple LPARs can be viewed simultaneously using a custom Visualizer .vgd file (available from Customer Support at ftp://ftp.bmc.com/pub/visualizer/vgd/) , showing what % of the entitlement is being utilized. For uncapped LPARs, this value can exceed 100% when demand exceeds the entitlement and resources are available, as shown in Figure 67 on page 152.
151
Figure 67
An integrated view of the LPAR is also available, simultaneously showing all three allocations. Figure 68 on page 153 shows what portion of the allocated capacity is in use.
s s s
Physical processors (total for the system) Virtual processors (for that LPAR) Entitlement processors (for that SPLPAR)
152
Figure 68
Another useful view is across all LPARs within a physical system. The current definition for this graphic shows total physical processor capacity utilized, and average virtual and entitlement utilization.
153
Figure 69
Also, using this graphic over time, either for individual LPARs or for the physical system as a whole, gives the analyst a complete overview of what's happening and what the long-term trends are. Of course you can drill down as appropriate to find out what the underlying cause is for a significant change in utilization. Also, there are supplemental graphs available for identifying individual LPARs that need further study - Top 10 Entitlement Used and Top 10 Virtual Used. See Figure 70 on page 155 and Figure 71 on page 156.
154
Figure 70
155
Figure 71
The %Physical_Used (from the AIX Partition CPU Usage Hierarchy graphic from the custom vgd) or %Proc (from the CPU Use Hierarchy/Partition CPU Use Hierarchy graphic) are useful when you need to present to less technical management,. Only when the absolute physical consumption reaches the danger zone does management need to spend money on additional hardware - problems with Entitlements and Virtual processor allocations require technical expertise to change the existing allocation scheme. Figure 72 on page 157 depicts an example showing the overall utilization of each physical frame, %Physical_Used (using the AIX Partition CPU Usage Hierarchy graphic from the custom vgd)
156
Figure 72
Another useful perspective for environments making substantial use of the Shared Pool, is from the pool perspective . Views are available for the pool as a whole, as well as by SPLPAR.
157
Figure 73
A "drill down" on an SPLPAR now shows all four perspectives simultaneously. Figure 74 Drill down on an SPLPAR
158
When preparing to do capacity planning for this environment, viewing partition use expressed as SPECints, particularly across physical systems, is the perfect way to see exactly how much is being consumed and what size physical system will be needed to support any particular combination of LPARs. Figure 75 SPECints used
This view, by SPECints, is the basis for Predict modeling, which is discussed in a later section. This view is also the basis for Virtualization Planning or Visualizer sizing techniques, which can be used to:
s s
establish the best selection of baseline modeling interval(s) or perform high-level system sizing for either initial consolidations or follow-on consolidations to existing partitioned systems.
159
160
Figure 76
This can be explained when the CPU utilization of the individual processes is explored - WSH generates the majority of the CPU load, and there is exactly 1 WSH process. See Figure 77 on page 162
161
When I look at "per logical processor" usage, it's very uneven. Why?
Figure 77
Different from traditional UNIX dispatching is virtual processor folding and the AIX Partition Load Manager (PLM) (pages 49 and 338-339, 393 in http://www.redbooks.ibm.com/redbooks/pdfs/sg247940.pdf). This feature dynamically adjusts how many of the virtual processors configured to the SPLPAR are used for actual work - if utilization is low, the number of virtual processors is decreased by 1 (per second) and if utilization is high, the number of virtual processors available is increased by 1. So under utilized LPARs will actually use many fewer of the virtual processors, and this could also produce uneven distribution of processor use shown in the example in Figure 77.
When I look at "per logical processor" usage, it's very uneven. Why?
When viewing un-aggregated logical processor usage when SMT is enabled (see How does SMT affect the number of CPUs? on page 147), the relative activity within each pair of logical processors is usually very uneven, since the "second" processor only shows activity when SMT actually occurred, which is not all the time. The coloration of the hierarchy chart clearly shows relationships between the logical processors, e.g. 0 with 1, 2 with 3, and so on.
162 BMC Performance Assurance Virtual Systems Implementation Guide
When I look at "per logical processor" usage, it's very uneven. Why?
Figure 78
Processor Hierarchy
Particularly, see how when activity is relatively high on 0 for pahrmd04 (the red and yellow cells), there is also activity on 1, but of lesser amounts (shades of green). But when activity is lower on 0 (green) there is little or no activity on 1. There are similar relationships between 2/3, 4/5, and 6/7. Figure 79 on page 164 shows the relationship of CPU activity between 0 and 1 graphically.
163
Figure 79
For a good discussion of how SMT works, go to http://www03.ibm.com/servers/eserver/iseries/perfmgmt/pdf/SMT.pdf. The document at that link explains why the data in Figure 79 looks like it does.
Reassign processors among partitions. Move partitions from one hosting system to another system. Move a stand-alone computer to a partition. Change a partition to a stand-alone computer.
The Predict Physical System Summary report provides information on the configuration of the hardware partition, and specific information on each computer in the partition. An example of the report is shown in Figure 80.
164
verify or correct the information in the model identify any resources that might be exceeding your service level objectives
TIP
Evaluating a logical system is similar to evaluating a standalone system, in regard to guidelines and thresholds. Workloads are created on the logical system just as on a standalone system.
For general information on calibrating models and creating valid baselines, refer to the Predict online Help.
The Physical System total line shows the total physical processors expressed as a percentage (% Utilization out of), For example, a value of 1600% translates to 16 physical processors, and the total CPU Utilization across all partitions is relative to a maximum of 1600%. The % Utilization out of value for each SPLPAR shows the number of virtual processors expressed as a percentage. For each DLPAR it shows the number of dedicated physical processors expressed as a percentage. Additionally, each SPLPAR shows Entitlement as the number of physical processors, and % Entitlement as the portion of the entitlement being utilized. Physical CPU processors are assigned to logical systems or share pools as a dedicated resource. Logical systems with dedicated processors own all of that CPU capacity; no other logical system or share pools can use those resources. Processors assigned to a share pool can be shared between SPLPARs running inside the pool.
165
Memory is always dedicated to the logical system regardless if it is LPAR or SPLPAR. The total physical system CPU utilization will be the sum of all the logical system with dedicated processors plus all of its share pool CPUs. Predict Physical System Summary report for an AIX Partition
Figure 80
The following table describes the columns in this report. You may need to scroll to the right to see some of the report columns.
Column Physical System Logical System Description The name of the Host Server in the model. The name of the logical volume in the model. The name has to be unique for a given computer. Two logical volumes on different computers can have the same name. The operating system running on the computer. The specific partition implementation. For AIX, the possible values are LPAR, DLPAR, SPLPAR. Other values are reported for other vendors partitioning systems or standalone servers, such as DSD (Oracle), nPAR, vPAR, nPAR/vPAR (HP). The type of CPU, usually identified in terms close to those the vendor would use. The name of the vendor (example IBM, HP) for the CPU model. A model name is not complete without the vendor name. Predict uses this field to decide what algorithms to use for modeling the system.
166
Description For a physical system, this value includes the Unaccounted CPU utilization of the shared pools, in addition to the utilization of all of the partitions. For logical systems, this number is the percentage out of the total number of online processors assigned to the logical system. In the case of AIX SPLPAR, this number is the percentage out of the virtual processors assigned to the logical systems.
% Utilization out of
For all logical system except SPLAR it is the number of online processors times 100. For an SPLAR, it is the number of virtual processors times 100. For a physical system, it is the sum of the online processors of all the logical partitions times 100. The percentage of the Entitlement that was used by this logical system. The maximum allocation of CPU from the shared pool that can be allocated to this partition. Entitlement is configured in terms of processing units (PU). One (1.0) PU equals the capacity of a full processor. For an SPLPAR, there is Capped or Uncapped entitlement. An uncapped system can use more processing units if there is spare capacity in the pool, meaning that it can use more than it is entitled to. Possible values are Yes (Capped) or No (Uncapped). The average number of transactions that are receiving service as well as those that are waiting to receive service. The transaction throughput in transactions per hour. The transactions counted in the throughput are those which ran on this computer, whether they are called by local or remote users or transactions.
% Entitlement Entitlement
Cap
For more information on the Physical System Summary report, refer to the Predict online Help.
167
The following section investigates adding a SPLPAR to an existing partitioned system, and the potential performance benefits.
One of the partitions has dedicated processors, and two have shared processors grouped into multiple share groups called pools. The LeadTracker application, which the sales personnel use to keep tabs on potential deals, is not used consistently during the week, but is heavily used at the beginning of each week. Perhaps adding a SPLPAR to the partition would be a good solution to alleviate the unique performance needs of this application. The following example illustrates the process for exploring the effect of adding an additional SPLPAR to the group.
1 Collect data on each of the three AIX partitions. 2 Identify good baselines for modeling (those periods with consistent usage). 3 Characterize the workloads in Analyzepaying particular attention to those
workloads which comprise overhead. This process relates system processes to the actual business applications or services. For more information on workloads, baselines, and models, see the online Help or the BMC Performance Assurance for Microsoft Windows Implementation Guide for Microsoft Windows.
4 Open the model in Predict and evaluate it. 5 Review the CPU utilization for each of the major application workloads in the
Predict Physical System Summary report (Figure 80 on page 166).
168
6 Create a new partition on the physical system by completing the following steps: A Right click the physical system, and select New Logical System. The New Logical
System wizard is displayed.
B Click Next on the Welcome page, and then follow the instructions on the wizard
to create the new logical partition. As shown below, add the name, description, and partition type (in this example, SPLPAR) on the Logical System Name page. Click Next.
C On the Shared Pool page, select whether you want to use an existing share pool,
or create a new one. For this example, we will use the existing share pool. Click Next.
169
Additional scenarios
D On the Logical System Configuration page, specify the configuration settings for
the new SPLPAR. Specify the number of virtual processors that will be available to the partition. Refer to the online Help for more information on the Autocalibration option. For an SPLPAR, you can select Capped or Uncapped entitlement. An uncapped node can use more processing units if there is spare capacity in the pool, meaning that it can use more than it is entitled to. Entitlement is used to specify the amount of CPU capacity that can be used. It is specified in terms of processing units (PU). One PU equals the capacity of a full processor. The sum of entitlements for all logical nodes within a pool cannot exceed the number of online processors for the pool. Click Next.
E On the Utilization Objectives page, set the guidelines and thresholds for
acceptable performance for the virtual processor, and optionally the paging rates. Click Finish. The new SPLPAR is now displayed in the tree, as shown below.
F Assign work to the new SPLPAR by right-clicking the desired workload (in this
example, the LeadTracker workload) and selecting Properties. Click the Users tab and complete the Users per computer section to include the SPLPAR_LT computer and to route activity to the new SPLPAR.
7 Evaluate the result, tuning as necessary. Review the CPU utilization for each of the
major application workloads in the Predict Physical System Summary report (Figure 80 on page 166) and compare the results to the initial version of the report, prior to creating the new SPLPAR.
Additional scenarios
With data collected and analyzed from the AIX partitioned systems, you can also perform other What if...? scenarios, such as:
s
What would happen to performance if I reassign processors among partitions? What if I move an existing workload from an existing partition to another partition?
170
What if I move a partition from one hosting system to another system? What if I move the workloads from a stand-alone computer to a partition?
Use UDRviewer (or printudr) on the Physical_System_Partition_Configuration metric group to obtain the complete configuration. There may be alternate sources of this information available, too.
171
Figure 82 shows sample UDRviewer output for Physical_System_Partition_Configuration (selected metrics). Figure 82 Sample UDRviewer output
In this case, the comparison reveals that several SPLPARs are missing, all using Pool 0, which means that modeling and reporting for this configuration would be compromised (unless the LPARs sabsor01, dabsor01, and pavio07a have nothing actually running in them).
172
Another method for making the same observation is to compare the CPU Utilization shown for the Shared Pool, 161.66, with the total of the SPLPARS, 135.04 (128.52 + 4.93 + 1.59). Since the two values don't match, there is at least one additional SPLPAR generating significant CPU activity. Alternatively, if you have a Predict model, you can view the amount of CPU utilization which is not represented by a partition which was explicitly collected. See How does Predict represent the shared pool (for SPLPARs)? on page 176 for an example.
2 Confirm that the CPU model is properly identified for both the physical system
(Analyze Physical System report) and the logical systems (Analyze Hardware Summary report) Figure 83 Analyze Physical System and Hardware Summary reports
A CPU model with "SMT" in its label is almost always the appropriate type of IBM Power 5 CPU as AIX defaults to using SMT whenever possible. A User Translation Table mapping is usually required as the default Translation Table mapping is conservative, and indicates non-SMT:
Name="IBM,9117-570",VendorID=IBM,Rate=1900,Alias=p5-570@1900,FreqTolerance=1
173
In the example reports in Figure 83, there is no published rating on spec for a p5-570SMT@1650, so both a User Hardware Table entry and User Translation Table entry are required. Contact Customer Support for assistance with CPUs which are not published by spec.org, and thus are not in the default Hardware Table.
The Physical System total line shows the total physical processors expressed as a % (% Utilization out of), e.g. 1600% translates to 16 physical processors, and the CPU Utilization there is the total across all partitions, relative to a maximum of 1600%. The % Utilization out of value for each SPLPAR shows the number of virtual processors expressed as a %, and for each DLPAR it shows the number of dedicated physical processors expressed as a %. Additionally, each SPLPAR shows its Entitlement as the number of physical processors, and what portion of the entitlement is being utilized, % Entitlement.
174
Also, you can still use all of the traditional interactive Predict reports (for workloads, relative response time, response time breakdown, etc.) to view performance results for either a baseline or what-if scenario.
As with "traditional" computers, you can edit in different objectives on a per LPAR basis as needed.
175
176
Figure 86
Resource Allocation
In this example, the physical system has 16 processors, and all 16 are assigned to the shared pool (for example, there are no DLPARs on this physical system). The virtual processors assigned to the SPLPARs can total to many more than the physical processors assigned to the shared pool - in this example there are 23 virtual processors compared with the 16 available in the pool. Since there was not collected data available for all active SPLPARs, the missing LPARs are represented as an aggregate Unaccounted CPU Utilization, which in this example is 31%. When a model with a shared pool is evaluated, all possible constraints to CPU performance are evaluated for each SPLPAR, i.e. the number of virtual processors, the entitlement processors, capping, and the number of processors in the pool. Calculated response times will reflect the waiting caused by competition for CPU resources at every level (labeled CPU Wait Time) and overall effects can be seen in the CPU Queue Length (for LPARs on the Physical System Summary report, and for workloads on the Workload Statistics by Computer report). If any of the CPU constraints impose an absolute limit on workload processing, Predict messages will be issued for that SPLPAR, and for the workload(s) on that SPLPAR:
W-CUTSPLRPL: CPU on SPLPAR pahrmd04 is saturated due to its running pool 0 as a constraint and the model is evaluated with a cutback on throughput.
177
W-CUTSPLRPL: CPU on SPLPAR pahrmd05 is saturated due to its running pool 0 as a constraint and the model is evaluated with a cutback on throughput. W-CUTSPLRPL: CPU on SPLPAR pahrmd08 is saturated due to its running pool 0 as a constraint and the model is evaluated with a cutback on throughput. WARNING-WKLSAT Throughput for transaction PD-Perform-Agent@PD-PerformAgent@pahrmd04 running on node pahrmd04 is less than specified arrival rate.
Since these sample messages indicate CPU saturation for every SPLPAR in the pool, this shows that the pool configuration needs to be increased, or an SPLPAR needs to be removed. When a single SPLPAR is messaged, this indicates a constraint in either the virtual processor or the entitlement configuration for that SPLPAR (and the Physical System Summary report will show which needs to be increased). The pool configuration and constraints are not expressly represented in Visualizer.
W-UTLABVGUD: CPU utilization of 65.55 on node IBM,0210A8A3D exceeds the guideline of 60.00.
178
Similar to "traditional" computers, you can edit in different objectives on a per Physical System basis, as needed.
To view the necessary configuration information (Windows Console): 1 Right-click the physical system => Properties. 2 Click Resource Allocation.
Figure 88 Viewing configuration information
179
In this example you can see that there are 16 processors configured on the physical system, and that 14 (2 + 2 + 10) processors are already assigned. At this point you could add a DLPAR which used up to 2 physical processors. If you needed more than that, first reconfigure the number of Total processors before adding a DLPAR which needs more than 2 processors.
Only 14 (12 plus 2) out of 16 processors are committed and the physical system memory is now 20480 (originally it was 24576 MB). If a DLPAR or SPLPAR is added, those memory requirements are automatically added onto the previous physical memory size for the physical system.
180
To remove an SPLPAR from the Shared Pool and dedicate a fixed number of processors to it instead (UNIX console): 1 Use the Nodes window and select the SPLPAR. 2 Edit => Convert => Logical to Regular. 3 Edit => Convert => Regular to Logical, selecting the existing Physical system, and
specifying DLPAR for the Partition type. See How do I add a new DLPAR to an existing physical system? on page 179 for additional steps which may be required in order to accommodate the resources required for the new DLPAR.
181
NOTE
The tasks described in the following sections are for the Windows version of the BMC Performance Assurance console. Similar support is available for the UNIX console. For instructions on how to complete these tasks using the UNIX console, see BMC Performance Assurance Implementation Guide for UNIX.
1 In the Investigate snap-in, right-click Policies and select New Policy. This starts the
New Policy Wizard.
2 On the Welcome page, enter a name for the policy and accept the default location
for the console data repository. The default location is C:\Program Files\BMC Software\Patrol3\BEST1\NTC\Repository. Click Next. This starts the New Computer wizard.
3 Make sure discovered from network is selected in the drop-down list, and then click
the Discover/Add button.
4 On the Discover Computers window, specify an IP address range and click Start
Discovery.
182
The discovery process could take a while depending on the number of computers in your network. After it is complete, you will have a list of all computers within the specified IP address range. You can sort by column headings. For example, if you only want to collect data from computers with the 7.5.10 Agent installed, click the Agent Version column head sorts the discovered computers by Agent version. You can then easily select computers with the 7.5.10 Agent for collection. Alternatively, you can select non-7.5.10 computers and delete them from the list.
5 Click the Agent Version column to sort by Agent type. 6 Highlight the computer you are interested in and click the Add button. To select
more than on computer, hold down the Shift key when making your selections.
7 Click OK. 8 One the Computers to Add to Policy window, click the check-boxes to select the
individual computers from which to collect data.
User Defined directs Analyze to aggregate workload usage based on the default or
user defined workloads, regardless of whether or not the data comes from a system configured as a workload partition. This option is the default and applies to data collected from any supported OS configuration. If you are collecting data from zones, however, keep in mind that this option will aggregate the workload utilization at the global zone level, not at the individual zone level.
s
WPAR directs Analyze to aggregate the workload usage based on the WPAR zone
the workload was executing in, regardless of the default or user defined workloads. This option is used to analyze data collected from a workload partition, with zones configured. Use this option if you plan on viewing the results in Analyze, Visualizer, or Perceiver, and want the workload usage grouped by zone. If you want the default view of workloads, then proceed to Task 3: Set up the Automator script on page 185.
Chapter 4 Implementing Performance Assurance on an AIX System with Hardware Partitions 183
If you are interested in viewing the workload utilization aggregated by WPAR in Visualizer or Perceiver the Manager script file that automates the data collection and data processing tasks requires a workload characterization that builds workloads by either the zone or project option. This is necessary so that you can properly view the performance data in Visualizer or Perceiver. To create a custom workload, complete the following steps:
1 Right-click the Analyze snap-in, then choose New Workload Characterization. 2 When the welcome page of the Analyze Workload Characterization Wizard opens,
click Next.
4 In the Analyze Properties page select the appropriate check boxes. 5 Click Finish. This launches the Computer Wizard. Use this wizard to define the
computers that are to be included in the Analyze workload characterization file. These computers will be represented in the Visualizer and Perceiver graphs that you will later view.
7 From the Build workloads by drop-down list, select WPAR. 8 Click OK to save the setting.
Remember the file name and the location, as you will need this information as you build the Manager script.
184
TIP
Run your script against a small or duplicate database first, previewing and testing it before running it against the production database.
For more information, refer to the Visualizer online Help or the Visualizer Implementation Guide.
1 From the Automator File menu, choose one of the default scriptname.b1a files to
open the Select a Database from the Catalog dialog box. New Daily Script (daily.b1a) New Weekly Script (weekly.b1a) New Monthly Script (monthly.b1a)
2 Select one of the databases from the Database selection list. When you click OK,
Visualizer creates a script and names it with capital letters. For example, if you select demo.b1a, after you click OK, the newly created script is named DEMO.B1A.
3 Review the events in the script and modify as necessary. To change an event,
highlight the event line in the script. Then do one of the following: Choose Edit => Modify. From the keyboard, press Alt+Enter.
4 Choose Run => Preview Mode. 5 Choose Run => Script Now. Click Yes when you see the following prompt.
Preview mode is enabled C the database will not be altered in any way. Are you sure you want to preview the script: scriptname now?
6 Review the log and correct any errors in the script. 7 From Automator, choose File => Save As. The script is now available to be included
in the Manager script.
185
1 Right-click the Manager snap-in, right-click Scripts, and choose New Script from the
menu. This starts the New Script Wizard. Read the Welcome page and click Next to proceed to the Manager Script File Name page.
2 Enter a name for your script. Use the default folders provided, unless you have set
up alternative locations for script files and reports. Click Next.
3 On the Operation window, select Collect New Data. Leave the default selection of
System and applications data.
4 Click Add to browse to the policy that has the computers you identified in Task 1:
Set up a policy that includes the target workload partitions on page 182. On the Select Policy File window, select the policy file and click Open.
5 On the Operation window, click Next. The Analyze and Predict window is
displayed. The Analyze and Predict window lets you decide how you will use the data that you collect. You can generate reports for Analyze, Predict, and Applications (if you selected that option on the previous window). You can also generate Visualizer files and create workload characterization files.
186
7 On the Select Analyze file dialog, click the custom workload you created in Task
2: Create a custom workload in Analyze on page 183, and then click Open.
8 On the Analyze and Predict window, click Next. 9 On the Collect or Analyze Interval window, use the up and down arrows in the
From and To fields to establish the collection interval.
10 Select the time zone, and then click Next. You will need to change the time zone
setting if data is being collected from computers in time zones that differ from the time zone of the console computer.
11 On the Automator Script Option window, click Browse. Use the Automator Script
Option page to specify the Automator script to be executed during the Manager run. Automator is a Visualizer tool that runs user-built scripts to populate a Visualizer database. The Automator script you specify will use Visualizer files you generated with Predict and Analyze to populate the Visualizer database. (See the Visualizer Implementation Guide for information about using Automator scripts.)
12 On the Select Automator File dialog, browse to the Automator script you created in
Task 3: Set up the Automator script on page 185, and then click Open.
13 On the Automator Script Options window, click Next. 14 On the Schedule a run of the script window, select the Schedule the script to run
checkbox. Click the up and down arrows to set the time to run the script. Set the recurrence options for the script in the Run script section.
187
188
Metric CPU User Time CPU System Time CPU Utilization Run Q Run Occ Swap Space Utilization
Description Measure of application related CPU processor load activity. Measure of system-related CPU activity. Percent of measurement interval that processor spends in system state. Normalized to a maximum of 100%. Measure of system responsiveness. Times runQue has been updated in system. RunOcc is updated every second whenever runQue has been updated. Portion of swap space utilized by system. Computed by following formula. Utilization = 100* (Total Virtual Memory Free Virtual Memory) / Total Virtual Memory The ratio of used virtual memory to total virtual memory. Average CPU load as measured every 1 min Average CPU load as measured every 5 min Average CPU load as measured every 15 min Total number of process context switches. Total number of all system fork and forkv calls. Measure of I/O related to paging. Number of pages. Measure of I/O related to paging. Number of pages. Pages read from swap space. Pages written to swap space. Pages scanned in memory. Reported in pages. The ratio of used virtual memory to total virtual memory. Represents the size of the buffer cache in kilobytes. In traditional UNIX systems, the kernel's buffer cache is the mechanism through which most explicit disk I/O passes. In newer versions of UNIX system, the Virtual Memory Manager primarily handles the explicit I/O. The average run queue length, defined as ratio of Run Q and Run Occ. The average swap queue length. Amount of used memory in system. (Default: Kbytes) Represents the size of the buffer cache in kilobytes. In traditional UNIX systems, the kernel's buffer cache is the mechanism through which most explicit disk I/O passes. In newer versions of UNIX system, the Virtual Memory Manager primarily handles the explicit I/O. Total number of pages written to and read from the swap space. Transfer of data between system buffers and disk or other block devices. Transfer of data between system buffers and disk or other devices. Logical reads to buffer cache in system.
Swap Space Ratio 1 Min Average CPU Load 5 Min Average CPU Load 15 Min Average CPU Load Process Context Switches Num System fork calls Page Ins Page Outs Pg Ins Pging Pg Outs Pging Pages Scanned Swap Space Ratio Buffer Cache Size
Avg. Run Q Len Avg. Swap Q Len Used Memory Buffer Cache Size
Total Pg Pging Buffer Block Reads Buffer Block Writes Logical Buffer Reads
189
Description Physical reads to disk or other devices in system. These are physical reads that represent raw I/O. Physical writes to disk or other devices in system. These are physical writes that represent raw I/O. The total number of message operations reported by the system. (Default: Primitivs/sec). The total number of semaphore operations reported by the system. (Default: Primitivs/sec).
For descriptions of these metrics, see BMC Performance Assurance System Metrics for Microsoft Windows and UNIX, Volume One.
190
When you select a host, you get the following tables and charts.
Metric IBM Host WPAR Info Description Displays information of each of the partitions configured on the selected host, including: s WPAR name s Host name s Logical System name s Number of host processors s Number of virtual processors Displays % Physical Used Total, % Virtual Used Average, % Entitlement Used Average and % Dedicated Used Average for the selected host. Displays WPAR CPU capacity used for all the WPARs configured on the selected host. Provides the system configuration information that is available for the WPAR host, including: s System Model s Host Name s System Description s Processor Model s Processor Description s Processor Clock Rate s Number of Configured Processors s Number of threads per core s Number of cores per socket s Rating s Rating Type s Multi threading type s Amount of Configured Memory
For more information on Perceiver views and metrics, refer to the online Help or the BMC Performance Perceiver Getting Started guide.
191
During the migration of a partition, the Perform collector continues running, as do all other processes. The collector detects the change of the host and records it to the Hardware Serial Number metric in the System Configuration metric group. All other configuration and statistics data is collected continuously. While Analyze processes the interval with partition migration, it reports separate partitions to the Visualizer database, one for each host. Perceiver is able to chart the system statistics metrics continuously for partitions that have been migrated from one host to another, and also displays the change of host in the Physical Host System Hardware Configuration table. The Physical CPU Capacity Used by Partition view illustrates the intervals that the partition has been moved by it comparing with its peers. In the case of different configurations (particularly the number of processors) between the source and destination hosts, the physical CPU utilization chart shows the difference of the normalized values, as shown in Figure 91. Figure 91 Evaluating performance of LPARs running WPARs
192
General FAQs
NOTE
WPARs do not have to run within an LPAR. BMC Performance Assurance supports both WPARs running in LPAR and WPARs running in a standalone system.
General FAQs
This section emphasizes upon various FAQs related to the AIX environment.
What are best practices for performance reporting and analysis for the AIX environment?
Instrument all LPARs with a collector (7.3.00 or later).
This ensures proper identification of SMT as well as access to the specific metrics required to properly report on LPAR resource consumption. Any un-instrumented DLPARs will compromise complete frame reporting. If any SPLPARs are not instrumented but one or more instrumented SPLPARs have Shared Pool Authority set, there is representation of the total CPU usage of all "missing" SPLPARs. This is not as detailed as the representation possible with full data collection. Compare Figure 92 on page 194 with Figure 72 on page 157.
193
What are best practices for performance reporting and analysis for the AIX environment?
Figure 92
Ensure that at least one LPAR per physical system is collecting the HMC configuration data. LPARs should be processed together in one Manager domain or policy. If you choose to split LPARs from the same physical system across multiple domains/policies:
s
The HMC configuration requires that a least one LPAR per physical system per domain/policy is collecting the HMC configuration. Thus, some customers have chosen to configure HMC collection for all LPARs in order to support arbitrary analysis groupings. (b)All LPARs from a frame must be populated to the same Visualizer database.
You can see exactly what has been accomplished in Visualizer: every properly instrumented LPAR will appear under the appropriate physical system name, with a designation of DLPAR or SPLPAR (CPU Use Hierarchy or optional AIX Partition CPU Usage Hierarchy graphic).
194
What are best practices for performance reporting and analysis for the AIX environment?
Figure 93
What are best practices for performance modeling for this particular environment?
Keep these important principles in mind:
s
When a Predict model is needed, all LPARs (from a particular physical system) must be identified within one Analyze. And as previously recommended for reporting, collectors should be deployed on all important LPARs. There is representation of the overall CPU usage of any "missing" SPLPARs, but this is not as detailed as the representation possible with full data collection (see also 7 above). The memory configuration for the physical system will reflect the sum of all LPARs which were jointly Analyzed. As described in the modeling section above, competition for the CPU is explicitly modeled and performance results reported. Memory, I/O, and network performance effects are limited to those within each LPAR
195
What are best practices for performance reporting and analysis for the AIX environment?
196
Chapter
This chapter presents the following topics: Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Support for hardware partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Support for HP Integrity VM systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Installation considerations for HP Integrity VM systems . . . . . . . . . . . . . . . . . . . . . . Installation considerations for HP-UX partitions . . . . . . . . . . . . . . . . . . . . . . . . . . Installation considerations for HP Integrity VM systems . . . . . . . . . . . . . . . . . . . Managing the performance of HP-UX partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Data collection considerations for HP hardware partitions . . . . . . . . . . . . . . . . . Displaying data from HP hardware partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . Modeling HP-UX virtual systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Overview of HP Integrity VM systems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . HP IVM Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Data collection considerations for HP IVM systems . . . . . . . . . . . . . . . . . . . . . . . Displaying data from HP IVM partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Viewing HP IVM metrics in Investigate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Viewing HP IVM data in Visualizer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Viewing HP IVM data in Perceiver. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Viewing HP IVM data in Analyze . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Workload and Predict model characterization in Analyze . . . . . . . . . . . . . . . . . . Analyze and the Visualizer Database Schema . . . . . . . . . . . . . . . . . . . . . . . . . . . . Virtualization Planning support for HP IVM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . HP IVM Virtualization Study for Deploying New Guests . . . . . . . . . . . . . . . . . . Reporting capacity usage for HP IVM systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Reporting CPU for a HP IVM system. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Reporting CPU allocation for a HP IVM system . . . . . . . . . . . . . . . . . . . . . . . . . . Modeling HP IVM systems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Implementation of best practices for HP IVM modeling and reporting . . . . . . . Evaluate the Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199 199 200 200 200 200 201 201 201 207 211 212 213 213 214 215 217 222 222 224 225 225 227 228 228 229 229 231
Chapter 5
197
Review the Physical System Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232 Perform What if...? investigations of HP IVM partitioned systems . . . . . . . . . 234 Modeling process summary of HP IVM systems . . . . . . . . . . . . . . . . . . . . . . . . . . 235
198
Overview
Overview
To identify and accurately model the activity on partition-capable HP hardware (both PA-RISC and Itanium), you must be able to collect data from both physical and virtual partitions, which HP refers to as nPartitions (nPars) and vPartitions (vPars), and software partitions, such as Integrity VM. In this environment, hardware resources such as processors, memory and disks can be dynamically reallocated to other partitions on an as needed basis from one partition to another. BMC Performance Assurance enables you to accurately model the entire activity of these types of partitioned servers.
NOTE
Analyzing and modeling data collected from the virtual system environments described in this book use the version 7.5.00 BMC Performance Assurance console on Microsoft Windows. Similar support is available on UNIX consoles.
s s
Collect configuration data about all of the partitions that enables Analyze and Predict correlate the activity of all of the partitions to the Host Server, for capacity planning purposes. View the overall configuration of the server (provided in the Physical Partition Configuration metric group), as well as some additional system information about the partition (provided in the System Configuration metric group). Generate performance reports on the partition configurations, based on the physical server, with breakdowns by logical systems (partitions). Build and evaluate the performance model for partitioning systems, including the relationship between logical systems and physical system. Perform What-if... scenarios on physical and logical systems Draw performance graphs for partitioned and partitioning systems, using Visualizer.
Chapter 5
199
Installation considerations
Installation considerations for HP-UX partitions
Note the following requirements when installing BMC Performance Assurance in a HP-UX environment:
s
Run the installation program according to the instructions in BMC Performance Assurance Getting Started guide. To obtain detailed information about applications or workloads running in the individual partitions, you must install the Perform Agent and Perform collector on each partition.
Check for the existence of the /opt/hpvm/bin/hpvmstatus executable: This is one of the CLIs provided by HP to manage VM guests. Review the output from the /opt/hpvm/bin/hpvminfo utility to identify the host.
HP-UX 11.23 and 11.31 for Itanium based systems support HP IVMs.
200
You should run the Perform collector on any partition that is running the HP-UX operating system, so that Analyze can assemble complete picture of the server and all of the partitions on it. Unlike the other partition platforms (Oracle Solaris and AIX) where an external node controls the server, the server can be controlled and queried locally. The collection of Physical Partition Configuration data is enabled by default. An option in the Collect.cfg file is available should you want to disable the collection of this metric group. Data collection is supported on partitions running HP-UX (11.11, 11.23 and 11.31).
You can collect data from HP-UX partitioned systems as you would other computers. These methods are:
s
for ad-hoc data collection, use the best1collect command line executable for automated data collection, create a Manager run using the BMC Performance Assurance console on either UNIX or Microsoft Windows.
These data collection methods are fully described in the BMC Performance Assurance Implementation Guide for UNIX, or by referring to the online Help.
Chapter 5
The following sections describe how to display and interpret the data using Investigate, Analyze, Visualizer, and Perceiver.
There are also a number of metrics that can specifically apply to HP-UX partitions. The following table describes each of these metrics:
Metric Partition Type Description The specific partition implementation. For HP, the possible values are nPar, vPar, nPar/vPar. Other values are reported for other vendors partitioning systems or standalone servers, such as DSD (Oracle), LPAR, DLPAR, SPLPAR (AIX). The name of the partition, which can be different from the System Name.
202
Metric Logical System Name Physical System Name Physical System Model Name Total Processors Active Processors
Description The unique identifier for the partition. The unique identifier for the server (host). This name matches the Hardware Serial Number in the system configuration within the partition. The model of the server. The number of physical processors available to the partition. The number of physical processors currently used by the partition (for nPar) or the number of physical processors that are assigned to the partition (for a vPar). In a vPar, the processors assigned to the vPar cannot be turned offline. The size of the memory allocated to the partition.
Total Memory
There are also a number of metrics in the Physical Partition Configuration metric group that support HP-UX partitions. The following table describes each of these metrics:
Metric Parent Name Partition Type Unassigned Processors Active Memory Unassigned Memory Description The system complex name for a nPar. This field is blank for a vPar or a nPar/vPar combination, as the complex name is not available for vPars. The type of partition implemented. Possible values are nPar, vPar, or nPar/vPar. The number of physical processors that are not assigned to any partition The size of the memory currently used by the partition. The size of the memory that is not allocated to any partition.
Chapter 5
203
204
1 Click Database=> Select and select the database into which you populated the HP
partition data.
2 Click Database=> Detail and choose the data for the graph. Ensure that you select
HPNPAR from the Virtual Type drop-down list.
3 From the Graphics menu, choose HP-UX Partition => HP NPAR VPAR Partitions
Hierarchy.
Figure 95
Chapter 5
205
Figure 96
When you select a host, you get the following tables and charts.
Metric HP-UX Partition Info Description Displays information of each of the partitions configured on the selected host, including: s Logical System name s Node name s Partition type s Pool ID s Number of dedicated processors
% Physical Used Total and % Displays the total physical CPU capacity used and dedicated Dedicated Used Average used average for all the partitions configured on the selected host.
206
Description Displays the top 10 Partition CPU Capacity Used for the selected host. Provides the system configuration information that is available for the WPAR host, including: s System Model s Host Name s System Description s Processor Model s Processor Description s Processor Clock Rate s Number of Configured Processors s Number of threads per core s Number of cores per socket s Rating s Rating Type s Multi threading type s Amount of Configured Memory
For more information on Perceiver views and metrics, refer to the online Help or the BMC Performance Perceiver Getting Started guide.
Reassign processors among partitions. Move partitions from one hosting system to another system. Move a stand-alone computer to a partition. Change a partition to a stand-alone computer.
The Predict Physical System Summary report provides information on the configuration of the hardware partition, and specific information on each computer in the partition. An example of the report is shown in Figure 97.
Chapter 5
207
When you click the Evaluate model icon, Predict evaluates the model, produces reports concerning the performance of the system represented by the model, and displays more information about the model components. Your next step is to examine the components in the model to
s s
verify or correct the information in the model identify any resources that might be exceeding your service level objectives
For more information on calibrating the model and creating a valid baseline, refer to the Predict online Help.
The following table describes the columns in this report. You may need to scroll to the right to see some of the report columns.
Column Physical System Logical System Description The name of the Host Server in the model. The name of the logical volume in the model. The name has to be unique for a given computer. Two logical volumes on different computers can have the same name. The operating system running on the computer. The type of partition implemented. Possible values are: nPar, vPar, or nPar/vPar.
208
Description The type of CPU, usually identified in terms close to those the vendor would use. The name of the vendor (example IBM, HP) for the CPU model. A model name is not complete without the vendor name. Predict uses this field to decide what algorithms to use for modeling the system. The percent of time the CPU is busy. The maximum percentage depends on the number of processors on the computer. For example, if the computer has 1 processor, the CPU utilization is out of 100% If the number of processors is 4, then the utilization is out of 400%. The number of online processors multiplied by 100. N/A for HP-UX partitions. N/A for HP-UX partitions. N/A for HP-UX partitions. The average number of transactions queued for CPU service at any one time. The number includes the transactions that are receiving service as well as those that are waiting to receive service. The transaction throughput in transactions per hour. The transactions counted in the throughput are those which ran on this computer, whether they are called by local or remote users or transactions.
CPU Utilization
Throughput
For more information on the Physical System Summary report, refer to the Predict online Help.
Chapter 5
209
1 Collect data on each of the existing servers. 2 Identify good baselines for modeling (those periods with consistent usage). 3 Characterize the workloads in Analyze determine which workloads will be
moved, which ones do not need to be moved, and which ones now would be shared in a consolidation, particularly the email application. For more information on workloads, baselines, and models, see the online Help or the BMC Performance Assurance for Microsoft Windows Implementation Guide for Microsoft Windows.
4 In Predict, open the model file, and evaluate the model. 5 Review the CPU utilization for each of the major application workloads in the
Predict Physical System Summary report (Figure 97 on page 208).
6 To move the partition to another system, expand the Computers tree, as shown
below.
7 Right click the computer and select Move Logical System. On the Move Logical
System dialog box, select the physical system to which you want to move the logical partition, and if appropriate, the nPar. Click OK.
210
8 Evaluate the result, tuning as necessary, comparing the results with the previous
Predict Physical System Summary report. You can also explore other What if...? scenarios, such as:
s
What would happen to performance if I move an existing workload from one vPar to another? What would the impact be from this workload consolidation? Create an additional vPar directly on a physical system. Move a standalone computer to a partition.
Support for low to high-end HP Integrity servers: HP Integrity rx1600, rx2600, rx4640, rx7620, rx8620, Integrity Superdome and other future HP Integrity servers. Is an extension of HPs existing partitioning continuum that combines the benefits of HP-UX Virtual Partitions (vPars) and HP Process Resource Manager (PRM). Is a soft partitioning and virtualization technology that provides operating systems isolation. Provides sub-CPU allocation granularity and shared I/O. A partition can hold up to 1/20th of a CPU. The IVMs can be installed on a standalone Integrity server or under an nPar that is running HP-UX. Different virtual machines can run guest operating systems with different versions or patch levels, and they support Linux, Windows and HP-UX.
Chapter 5
211
HP IVM Architecture
HP IVM Architecture
HP Integrity Virtual Machine (HP IVM) is an extension of HPs existing partitioning continuum that combines the benefits of HP-UX Virtual Partitions (vPars) and HP Process Resource Manager (PRM). The HP IVM product is a soft partitioning, and virtualization technology that provides operating systems isolation, with sub-CPU allocation granularity and shared I/O. A partition can have up to 1/20th of a CPU. The IVMs can be installed on a standalone Integrity server, or under an nPar that is running HP-UX. The hypervisor supporting virtualization for IVMs is based on a Type-2 technology, which is similar to the Microsoft Virtual Server, user-mode Linux, and the older VMWARE GSX. These hypervisors are software applications running within an OS. It is a separate software layer, with the guest OS running at a third level above the hardware. The virtual machine can run HP-UX, Linux, Windows, and OpenVMS, however the guest OS support varies based on the version of IVM software. Figure 99 below gives a high level architectural build-up of HP IVM. Figure 98 HP IVM Architecture
HP IVM is built on Type-2 Virtualization technology similar to Microsoft Virtual Server, and older VMware GSX. The Host OS is the first level, the Hypervisor is on the second level, and is a separate software layer running on the Host OS; and finally on top there is the Guest OS that runs at the third level above the hardware.
212
BPA data collector should be installed on the host that is running HP IVM. All the configuration and statistical groups that are collected on a regular HP-UX system are collected for an IVM host system. In addition, the collector collects configuration metrics about all the virtual machines that are running on the host. Collectors can be installed on the IVM virtual machines as well. The data collected from the virtual machines will be populated and available in the database tables. No attempt to correlate metrics collected from inside the virtual machine to that collected from the host OS. For IVM running within HP nPar systems, collector should be installed on the partition hosting the Integrity virtual machines.
The following sections describe how to display and interpret the data using the components listed above.
Chapter 5
213
Maximum percentage (%) of processor resources to which the IVM is entitled UUID (Unique Identifier) of the IVM
Process ID (PID) for the IVM The number of processors not assigned to any IVM Number of KBytres memory not assigned to any IVM Total number of installed processors on the IVM Host Total number of KBytes memory on the IVM Host
The investigate support is the same as for the other virtualized and non virtualized platforms. Figure 99 shows a drill-down of IVM configuration metric group.
214
Figure 99
For more information on any of the metrics, refer to Chapter 2 of the BMC Performance Assurance System Metrics for Microsoft Windows, UNIX, and Linux, Volume One.
Chapter 5
215
216
Chapter 5
217
HP IVM Overview
View Name Virtual Machine Overview Chart Label Physical CPU Capacity Used Chart Type Histogram Description Normalized by the total number of processors in the system Normalized by the number of virtual processors configured for IVM Normalized by the total memory configured Normalized by the total number of processors in the system Normalized by the number of virtual processors configured for IVM Normalized by the total memory configured
View Category
Histogram
Memory Utilization
Histogram
Line
Line
Memory Utilization
Line
218
Table 7
HP IVM Overview
View Category
Histogram
Memory Utilization
Histogram
Line
Line
Memory Utilization
Line
Chapter 5
219
220
For more information on Perceiver views and metrics, refer to the online help, or the BMC Performance Perceiver Getting Started guide.
Chapter 5
221
Shows the total number of processors accessed per nPar or per nPar/vPar (100% = 1 processor). Uses the information available in the IVM configuration metric group to connect the host to the virtual machines. Uses the PID information in the IVM configuration group to retrieve the process record corresponding to the virtual machine. The resource usage reported for the process is used to compute the resource usage for each of the virtual machines. At this time, only CPU and memory numbers are available at each VM level.
222
Chapter 5
223
By default, an Analyze report shows one transaction per partition. (Figure 105) Figure 105 Analyze - Default option
224
In Perceiver, select the Virtualization Planning tab, and click, on Begin Virtualization Study. Select a group of computers to include in the study and click on Deploy New Guests'. Fill in the number for each type of guest to deploy. (Figure 106)
Chapter 5
225
Select Target Host you want the guests to be deployed to, and then click on the Deploy button. (Figure 107)
On selecting the target host, the results of the deployment will be displayed as Deployment Summary. (Figure 108)
226
Total physical processors Processors assigned to each nPar Each standalone nPar with a provision for adequate processors Each vPar within an nPar with provisions for adequate virtual processors and nPar processors.
Chapter 5
227
The total percentage of physical system capacity used, and Percentage of physical system capacity used by partition.
The total number of processors accessed per nPar or per nPar/vPar (100% = 1 processor), and Dynamic shifting of processors between vPars within an nPar causing processor allocation to vary by an interval.
228
It must always be confirmed that the baseline analysis contains all the important partitions, and that they have been properly instrumented. One must instrument all important partitions with Collector v7.3.00 (or later). This ensures access to the new metrics required to properly report on partition resource consumption.
Chapter 5
229
It must be ensured at all times that at least one partition per nPar has a collector (required to obtain complete partition configuration data). Partitions may be processed in separate Manager domains/policies, but must be populated to the same Visualizer.
NOTE
Missing partitions will compromise having complete results!
Each instrumented partition appears under the appropriate physical system name, designated as NPAR or NPARVPAR, as shown in Figure 111 below:
Here, the physical system name, with the partitions properly recognized, are indented one level below.
230
It needs to be confirmed that the CPU model is properly identified for both the physical system (Analyze - Physical System report) and the logical system (Analyze - Hardware Summary report).
General Best Practices, Consolidation Sizing techniques for HP partitions, processes and tools for identifying virtualization candidates are the same as shown for AIX in earlier sections. The process and tools for sizing techniques are also the same. Modeling techniques employed for HP partitions are similar to AIX. It may also be noted that Physical Systems can contain partitions, and a Computer represents one partition. Also, each nPar can be standalone, or can contain multiple vPars as shown in Figure 112 below. Figure 112 Modeling in HP partitions
Chapter 5
231
When you click the Evaluate Model icon, Predict evaluates the IVM model, produces reports concerning the performance of the system represented by the model, and displays more information about the model components. Your next step is to examine the components in the model to:
s s
verify or correct the information in the model identify any resources that might be exceeding your service level objectives
For more information on calibrating the model and creating a valid baseline, refer to the Predict Online Help.
232
Table 8
Column
CPU Utilization
Throughput
For more information on the Physical System Summary report, refer to the Predict Online Help.
Chapter 5
233
Revise CPU configuration of the partitioned system Use the Physical System, and Configuration tabs. Revise configuration of individual nPars and vPars Use the Physical System, and Resource Allocation buttons. Move work onto the physical system. This can be performed as follows: Select the Node Click on Edit - Convert - Regular to Logical (UNIX GUI) or Convert to Logical System (Windows GUI), for each computer to be a partition within a model Import desired Workload(s) from systems into this model Revise CPU utilization objectives. This can be done as follows: Click on Edit - Node (UNIX GUI) or, go to the Properties (Windows GUI), and use the Objectives tab System growth (use Scenario Planner).
For more information on modeling and What if...? analysis, refer to the Predict online Help or the BMC Performance Assurance for Microsoft Windows Implementation Guide for Microsoft Windows.
234
Build a baseline model for the current partitioned system. Set Baseline to ON. Evaluate the model and observe messages/reports for warnings that a CPU objective has been exceeded. Follow implementation Best Practices, such as: Use the Support website Knowledge articles. Open a case with Customer Support for customized assistance with implementation. Allow extra time for the first implementation for a particular type of Virtualization. Apply tools specifically designed to support HP IVM Virtualization This generally requires Collector v7.3, and Console v7.3. Use CDB for a complete view of CPU Usage (historical and future), and derive measurements from physical HP IVM systems which generally include: Three graphs per physical system, giving a complete reporting solution More detailed graphs, used for specific analysis projects % Capacity Used for the physical system Modeling the results for HP IVM virtual systems, which include: viewing physical systems and individual partitions viewing workload performance results within the HP IVM partitions Predictor Scenario Planner results can be shown, as and when required. When using Predictor with HP IVM: Use the appropriate modeling construct for each type of Virtualization (e.g. Physical System) Use the corresponding specific modeling report for each type of Virtualization (e.g. Physical System Summary)
Chapter 5
235
236
Chapter
This chapter presents the following topics: Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Support for Solaris Containers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Support for Solaris Dynamic System Domains (DSDs) . . . . . . . . . . . . . . . . . . . . . Support for Solaris Logical Domains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Installation considerations for Solaris. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . For installing in Solaris 10 environments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . For installing in Solaris DSD environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . For installing in Solaris LDom environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Managing performance of Solaris Containers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Collecting and processing data from Solaris Container systems . . . . . . . . . . . . . Displaying data in Visualizer for Solaris Container systems . . . . . . . . . . . . . . . . Displaying data in Perceiver for Solaris Container systems . . . . . . . . . . . . . . . . . Finding out more using Investigate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Managing performance of Oracle partitioned (DSD) systems . . . . . . . . . . . . . . . . . . Displaying Data in Analyze from Oracle Partitioned Systems . . . . . . . . . . . . . . Modeling Data from a Oracle Partitioned Environment . . . . . . . . . . . . . . . . . . . . Managing performance of Oracle Logical Domains . . . . . . . . . . . . . . . . . . . . . . . . . . . Displaying data in Investigate from Oracle LDoms. . . . . . . . . . . . . . . . . . . . . . . . Displaying Solaris LDom CPU Use in Perceiver . . . . . . . . . . . . . . . . . . . . . . . . . . Modeling data from a Oracle LDom environment. . . . . . . . . . . . . . . . . . . . . . . . . Managing performance with Chip Multi-Threading . . . . . . . . . . . . . . . . . . . . . . . Modeling an UltraSPARC processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238 238 239 239 240 240 240 241 241 241 249 251 253 257 257 257 259 259 260 262 265 266
Chapter 6
237
Overview
Overview
BMC Performance Assurance supports two types of virtual systems on Oracle Solaris systems:
s s s
Solaris Containers Solaris Dynamic System Domains (DSDs) Solaris Logical Domains (LDoms)
s s s
additional metric data and treatment of non-global zone as a node, with constraints CPU, Paging IO and Total IO metrics at the non-global Zone level Visualizer version 4.2.05 with latest patch for reporting of Container and zone data process data available at the global zone level and top 30 processes within each non-global zone (in Visualizer) Container views in BMC Performance Perceiver 7.5.00 and higher
specific zone reports in BMC Performance Analyzer for Servers or BMC Performance Predictor for Servers if you run Solaris Containers in a DSD environment, the Containers are reported as if they are nodes or partitions, with no information on the global zone environment.
238
Identifying a system as a LDom or a control LDom. If the collector detects that it is running on the control LDom, it collects the Physical Partition Configuration metric group, which describes the configuration of the LDoms running on the server. The collector also collects existing Solaris core level statistics and other performance and configuration data from each LDom. Investigate support for the LDom metric groups. Data Analysis that reads and processes the key metrics, identifies and provides the relationships between objects to the database and the model. Support workload characterization of data collected within each LDom. Presentation of key metrics in BMC Performance Perceiver Datacenter Overview charts and detailed metrics in Solaris LDom specific charts. Modeling of Ldom systems based on threads instead of cores. Visualizer charts that present LDom specific views. Collection, analysis, modeling and presentation of zones running within LDoms.
Chapter 6
239
BMC Performance Assurance supports Chip Multi-Threading (CMT) technology on Oracle UltraSPARC T1 andT2 processors, where which ave multiple cores and multiple threads. Each Oracle SPARC system may have 1, 2 or 4 chips (physical processors), each UtraSPARC processor has 4, 6, or 8 cores, and each core can process 4 (T1) or 8 (T2) threads. A single Oracle SPARC system may have as many as 256 individual processing threads, and the collector identifies each thread as an individual processor, providing utilization information and metrics.
You must install the Perform Agent in the global zone. You cannot install the Perform Agent inside a Solaris non-global zone. Visualizer graph support requires version 4.2.05 with latest patch. BMC Performance Perceiver support requires version 7.5.00 and higher.
240
s s
Install the Perform Agent on the control LDom, so that the collector can identify systems as LDoms and collect configuration data about the LDoms. The collector reports configuration and statistical data for only the resources assigned to the LDom. BMC Software recommends that you use LDom Management software version 1.1 or higher, to help correlate the activity of the LDoms back to the servers they reside on. Visualizer graph support requires version 4.2.05 with latest patch. BMC Performance Perceiver support requires version 7.5.00.
The following tasks describe the recommended process for collecting and processing data from Solaris 10 systems configured as zones.
Task Create a policy that includes the global zone. Create a custom workload in Analyze Set up an Automator script. Set up and schedule the Manager script. Verify the data collection and processing. Refer to page 242 page 243 page 245 page 246 page 248
NOTE
The tasks described in the following sections are for the Windows version of the BMC Performance Assurance console. Similar support is available for the UNIX console. For instructions on how to complete these tasks using the UNIX console, see BMC Performance Assurance Implementation Guide for UNIX.
1 In the Investigate snap-in, right-click Policies and select New Policy. This starts the
New Policy Wizard.
2 On the Welcome page, enter a name for the policy and accept the default location
for the console data repository. The default location is C:\Program Files\BMC Software\Patrol3\BEST1\NTC\Repository. Click Next. This starts the New Computer wizard.
3 Make sure discovered from network is selected in the drop-down list, and then click
the Discover/Add button.
4 On the Discover Computers window, specify an IP address range and click Start
Discovery.
The discovery process could take a while depending on the number of computers in your network. After it is complete, you will have a list of all computers within the specified IP address range. You can sort by column headings. For example, if you only want to collect data from computers with the 7.5.00 Agent installed, click the Agent Version column head sorts the discovered computers by Agent version. You can then easily select computers with the 7.5.00 Agent for collection. Alternatively, you can select non-7.5.00 computers and delete them from the list.
6 Highlight the computer you are interested in and click the Add button. To select
more than on computer, hold down the Shift key when making your selections.
7 Click OK. 8 One the Computers to Add to Policy window, click the check-boxes to select the
individual computers from which to collect data.
User Defined directs Analyze to aggregate workload usage based on the default or
user defined workloads, regardless of whether or not the data comes from a system configured as a Solaris 10 Container environment (zones or a project). This option is the default and applies to data collected from any supported OS configuration. If you are collecting data from zones, however, keep in mind that this option will aggregate the workload utilization at the Container level, not at the individual zone level.
s
Zone directs Analyze to aggregate the workload usage based on the zone the workload was executing in, regardless of the default or user defined workloads. This option applies only to data collected from a Solaris 10 system that has been software partitioned using the Solaris 10 Containers technology, with zones configured. Use this option if you plan on viewing the results in Analyze, Visualizer, or Perceiver, and want the workload usage grouped by zone. Project directs Analyze to aggregate the workload usage based on the settings from the project configuration file, regardless of user defined workloads. One of the configuration options in Solaris 10 is to create a project as a way of managing resources for processes running in a zone. This option applies only to data collected from a Solaris 10 system that has been software partitioned using the Solaris 10 Containers technology, with projects configured. Use this option if you plan on viewing the results in Analyze, Visualizer, or Perceiver, and want the workload usage grouped by project.
Chapter 6
243
If you want the default view of workloads, then proceed to Task 3: Set up the Automator script on page 245. If you are interested in viewing the workload utilization aggregated by zone in Visualizer or Perceiver the Manager script file that automates the data collection and data processing tasks requires a workload characterization that builds workloads by either the zone or project option. This is necessary so that you can properly view the performance data in Visualizer or Perceiver. To create a custom workload, complete the following steps:
1 Right-click the Analyze snap-in, then choose New Workload Characterization. 2 When the welcome page of the Analyze Workload Characterization Wizard opens,
click Next.
4 In the Analyze Properties page select the appropriate check boxes. 5 Click Finish. This launches the Computer Wizard. Use this wizard to define the
computers that are to be included in the Analyze workload characterization file. These computers will be represented in the Visualizer and Perceiver graphs that you will later view.
7 From the Build workloads by drop-down list, select Zone or Project, depending on
how you want workload usage to be grouped in Visualizer.
Remember the file name and the location, as you will need this information as you build the Manager script.
TIP
Run your script against a small or duplicate database first, previewing and testing it before running it against the production database.
For more information, refer to the Visualizer online Help or the Visualizer Implementation Guide.
1 From the Automator File menu, choose one of the default scriptname.b1a files to
open the Select a Database from the Catalog dialog box. New Daily Script (daily.b1a) New Weekly Script (weekly.b1a) New Monthly Script (monthly.b1a)
2 Select one of the databases from the Database selection list. When you click OK,
Visualizer creates a script and names it with capital letters. For example, if you select demo.b1a, after you click OK, the newly created script is named DEMO.B1A.
3 Review the events in the script and modify as necessary. To change an event,
highlight the event line in the script. Then do one of the following: Choose Edit => Modify. From the keyboard, press Alt+Enter.
4 Choose Run => Preview Mode. 5 Choose Run => Script Now. Click Yes when you see the following prompt.
Preview mode is enabled C the database will not be altered in any way. Are you sure you want to preview the script: scriptname now?
7 From Automator, choose File => Save As. The script is now available to be included
in the Manager script.
1 Right-click the Manager snap-in, right-click Scripts, and choose New Script from the
menu. This starts the New Script Wizard. Read the Welcome page and click Next to proceed to the Manager Script File Name page.
2 Enter a name for your script. Use the default folders provided, unless you have set
up alternative locations for script files and reports. Click Next.
3 On the Operation window, select Collect New Data. Leave the default selection of
System and applications data.
4 Click Add to browse to the policy that has the computers you identified in Task 1:
Set up a policy that includes the target Solaris systems on page 242. On the Select Policy File window, select the policy file and click Open.
5 On the Operation window, click Next. The Analyze and Predict window is
displayed. The Analyze and Predict window lets you decide how you will use the data that you collect. You can generate reports for Analyze, Predict, and Applications (if you selected that option on the previous window). You can also generate Visualizer files and create workload characterization files.
246
7 On the Select Analyze file dialog, click the custom workload you created in Task
2: Create a custom workload in Analyze on page 243, and then click Open.
8 On the Analyze and Predict window, click Next. 9 On the Collect or Analyze Interval window, use the up and down arrows in the
From and To fields to establish the collection interval.
10 Select the time zone, and then click Next. You will need to change the time zone
setting if data is being collected from computers in time zones that differ from the time zone of the console computer.
11 On the Automator Script Option window, click Browse. Use the Automator Script
Option page to specify the Automator script to be executed during the Manager run. Automator is a Visualizer tool that runs user-built scripts to populate a Visualizer database. The Automator script you specify will use Visualizer files you generated with Predict and Analyze to populate the Visualizer database. (See the Visualizer Implementation Guide for information about using Automator scripts.)
Chapter 6
247
12 On the Select Automator File dialog, browse to the Automator script you created in
Task 3: Set up the Automator script on page 245, and then click Open.
13 On the Automator Script Options window, click Next. 14 On the Schedule a run of the script window, select the Schedule the script to run
checkbox. Click the up and down arrows to set the time to run the script. Set the recurrence options for the script in the Run script section.
15 Click Finish to save the script and set the schedule. Task 5: Verify the data collection
Using the Manager component has the added advantage of allowing you to view collection status with the Status Report feature. To access the Status Report feature:
1 In the BMC Performance Assurance console scope pane, open the Manager snap-in. 2 Click Status Report.
The Perform - Data Collection Status opens in the results pane. The available scripts are in the left portion of the report. To view the status of data collection, click on a script. You can also click Schedule in the scope pane to display the scripts in the results pane. Then right-click on the script for which you want to view reports and select View Status Report.
248
See the online help, available from the help button on the top right portion of the Status Report, or refer to the appendix on Troubleshooting Tools in the BMC Performance Assurance Getting Started for more information about using these reports.
1 Click Database=> Select and select the database into which you populated the
Solaris Container data.
2 Click Database=> Detail and choose the data for the graph. Ensure that you select
ZONE from the Virtual Type drop-down list.
3 From the Graphics menu, choose Solaris Partition => CPU Use Hierarchy.
Chapter 6
249
The following table provides some guidance for interpreting the key metrics available on this graph.
Metric %Total Max Notes The percentage of CPU utilization out of the maximum available CPUs. The maximum amount of CPU available, including all physical CPUs on the system. For example, if there are two physical CPUs on the system the Max value is 200. Shows CPU utilization as SPECInts used and total SPECint Rating. The amount used could exceed the maximum rating since the maximum rating is based on the number of physical processors, and the SPECInts used is based on the number of logical processors. Shows the % CPU utilization per physical processor. In this example this is 135.72% divided by 200%, or 67.86%. Shows the number of physical processors associated with the selected node. Shows the % CPU utilization per logical processor. Shows the sum of all the logical processors of the zones. This metric is not applicable in this environment.
250
Notes This metric is not applicable in this environment. This metric is not applicable in this environment. This metric is not applicable in this environment. This metric is not applicable in this environment.
Chapter 6
251
A standalone system configured with multiple non-global zones and the nonglobal zones can be seen in the CPU Profile view under Solaris Container Summary. In this case, Perceiver views the global-zone system as a regular UNIX computer and makes all metrics and views available for this system. A single computer configured with many dynamic system domains (DSDs). Each DSD can contain a number of non-global zones which can be seen under the Solaris Domain Summary By Resource view category. In this case, Perceiver views each DSD as a separate logical partition.
The CPU Profile view shows information on a host and its zones. If you select a host, you get the following tables and charts. Figure 115 The Perceiver Solaris Container Summary
252
Physical CPU Capacity Used Virtual CPU Capacity Used System Hardware Configuration
To view the DSD, refer to the Solaris Partition Overview. For more information on Perceiver views and metrics, refer to the online Help or the BMC Performance Perceiver Getting Started guide.
Chapter 6
253
NOTE
The tasks described in the following sections are for the UNIX version of the BMC Performance Assurance console. Similar support is available for the Windows console. For instructions on how to complete these tasks using the UNIX console, see BMC Performance Assurance for Microsoft Windows Implementation Guide for Microsoft Windows.
For more information on any of the metrics or metric groups, refer to Chapter 2 of the BMC Performance Assurance System Metrics for Microsoft Windows, UNIX, and Linux, Volume One. The metrics associated with this metric group are
s s s s s s
Zone ID - unique zone identifier Zone Name - name assigned to the Solaris zone Unique ID a counter of how many times it has been rebooted Pool ID resource pool ID with which the zone is associated Init PID process identifier (PID) of initialization process CPU Shares - sum of shares of all active projects is used in calculating the portion of CPU resources to be assigned to projects Root path to root file system
Pool Configuration
The Pool Configuration drill down provides pool information for configured pools on a Solaris computer.
254
The metrics associated with this metric group, collected when pools are enabled, are
s s s s s s s s s s s
Pool Id - unique resource pool identifier Pool Name - resource pool name Pool Status (0 = inactive, 1 = active) Pool Default - is this the default pool (0 = no, 1 = yes) Pset Id - processor set identifier Pset Name - name assigned to the processor set Pset Size number of processors in the processor set Pset Min minimum processors in the processor set Pset Max maximum number of processors in the processor set Pset Default is this the default processor set (0 = no, 1 = yes) Pset Cpus all cpus assigned to the processor set (semicolon separated list)
Chapter 6
255
The Investigate filter feature lets you view the process statistics for a particular zone. Process Statistics for the remaining global and non-global zones on the computer are not displayed in the drill down. In Figure 119 the drill down would display process statistics for the global zone, which has a Zone ID equal to 0. Figure 119 Investigate filter
256
When a System Controller is part of a data collection and is included in the list for processing, Analyze detects if there are any missing partitions from the current processing and displays N/A in the columns for those systems. This information helps you understand if there are any additional partitions that should be included, helping you form a complete understanding of the partitioned environment. For more information on the Physical System report, refer to the Analyze Online help.
Chapter 6
257
With Predict, you can perform any of the following tasks as a part of your What-if... analysis of a partitioned system:
s s s s
Reassign processors among partitions. Move partitions from one hosting system to another system. Move a stand-alone computer to a partition. Change a partition to a stand-alone computer.
The Predict Physical System Summary report provides information on the configuration of the hardware partition, and specific information on each computer in the partition. An example of the report is shown below. Figure 122 Predict Physical System Summary report for a Oracle Partition
For more information on the Physical System Summary report, refer to the Predict online Help. Visualizer reports identify if a machine is running in a partition. The physical computer name is displayed on the CPU diagram. The CPU utilization of the partitions are displayed to present the total physical CPU consumption.
258
1 Click Database=> Select and select the database into which you populated the
Oracle LDom data.
2 Click Database=> Detail and choose the data for the graph. Ensure that you select
LDOM from the Virtual Type drop-down list.
3 From the Graphics menu, choose Solaris Partition => Solaris LDOM Hierarchy.
Chapter 6
259
260
If you select a host, you get the following tables and charts.
Metric LDom Host Partition Info Description Displays information of each of the LDom(s) configured on the selected host, including: s Partition name s Node name s Number of dedicated threads s Number of dedicated processors Displays the % Physical Used Total and % Domain Used Average of all LDom(s) for the selected host.
% Capacity Used
Chapter 6
261
Description Displays the top 10 Domain CPU Capacity Used of LDom(s) for the selected host. Provides the system configuration information that is available for the LDom host, including: s System Model s Host Name s System Description s Processor Model s Processor Description s Processor Clock Rate s Number of Configured Processors s Number of threads per core s Number of cores per socket s Rating s Rating Type s Multi threading type s Amount of Configured Memory
For more information on Perceiver views and metrics, refer to the online Help or the BMC Performance Perceiver Getting Started guide.
At the host level, a thread-utilization option is displayed in the LDom Properties window at the physical system level. For a newly created Oracle Host with LDom, this field is set to enabled when browsing CPU Model filtered by UltraSparc T1, T2 or T2+. For existing nodes created from Analyze, this field will be set by Analyze and displayed correspondingly.
262
At the logical system level, Logical Domain is the partition type for LDom
The Thread-Per-Core field obtains its value from the CPU Model at the host level. This field appears only on LDom logical systems.
Chapter 6
263
The Total threads field appears in the LDom Properties dialog, which equals the total number of processors times the number of threads per core.
264
Chapter 6
265
1 Before creating the new model, you must add the processors description into the
User hardware Table.
2 In the BMC Performance Assurance console, open Predict. 3 Right-click the processor from the Computers tree hierarchy and select Properties.
266
The processors Properties dialog appears. Figure 131 Predict Properties dialog for a CMT processor
4 Click Browse to select the processor model and CPU clock speed. 5 In the Online processors field, verify that the number of Software threads is equal to
or higher than the processors hardware threads. You can verify the number of software threads in Visualizers Commands Hierarchy graph, NumProc metric.
NOTE
If the processor receives fewer software threads than the available number of hardware threads, than the model will predict better performance than the physical processor will actually achieve.
Chapter 6
267
268
Chapter
7
269 270 270 271 271 271 279 281 281 285 287 289 289 292 294
This chapter presents the following topics: Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Support for Xen nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Benefits of using BMC Performance Assurance in this environment . . . . . . . . . Installation and configuration considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Collecting and processing data from Xen systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . How do I collect and process the data? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Viewing results in Analyze. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Viewing results in Perceiver. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Xen Partition Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Xen Host Summary by Resource. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Xen Partition Summary by Resource . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Modeling Xen servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Working with Xen systems in Predict . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . View the Predict report for Xen. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Virtualization Planning support for Xen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Overview
Xen virtualization consists of a Node, which is the host machine and that host machine runs 1 or more domains. Domain-0 (DOM-0) represents the host system. Each guest OS is called a DOM-U where the 'U' is a number. For example a machine that has 4 guests would have DOM-0 through DOM-5. The numbering scheme for the domains is not always linear. For example you could have 4 guests, one could be DOM-2, the next one could be DOM-4, then the next two could be DOM-5 and DOM6. Each time a domain is shut down and restarted it is potentially given a new domain number.
269
Proxy collection
270
Performance data collected from within the domain operating system may be used to display the relative resource usage of individual processes. All domains share system resources, so obtaining accurate measurements of total actual resource usage is necessary to provide a complete picture of system usage.
Installation and configuration considerations Collecting and processing data from Xen systems
You can automatically collect the data from the Xen systems you are interested in, processes the data, and store the data in a Visualizer database. The best practice for implementing BMC Performance Assurance in this type of environment is to create a Manager script to automate the process. The following sections provide the basic instructions for setting up a Manager script creating a custom workload, and using Automator to populate the Visualizer database.
NOTE
Proxy collection is not supported for Xen data.
271
NOTE
The tasks described in the following sections are for the Windows version of the BMC Performance Assurance console. Similar support is available for the UNIX console. For instructions on how to complete these tasks using the UNIX console, see BMC Performance Assurance Implementation Guide for UNIX.
1 In the Investigate snap-in, right-click Policies and select New Policy. This starts the
New Policy Wizard.
2 On the Welcome page, enter a name for the policy and accept the default location
for the console data repository. The default location is C:\Program Files\BMC Software\Patrol3\BEST1\NTC\Repository. Click Next. This starts the New Computer wizard.
3 Make sure discovered from network is selected in the drop-down list, and then click
the Discover/Add button.
4 On the Discover Computers window, specify an IP address range and click Start
Discovery.
The discovery process could take a while depending on the number of computers in your network. After it is complete, you will have a list of all computers within the specified IP address range. You can sort by column headings. For example, if you only want to collect data from computers with the 7.5.10 Agent installed, click the Agent Version column head sorts the discovered computers by Agent version. You can then easily select computers with the 7.5.10 Agent for collection. Alternatively, you can select non-7.5.10 computers and delete them from the list.
5 Click the Agent Version column to sort by Agent type. 6 Highlight the computer you are interested in and click the Add button. To select
more than on computer, hold down the Shift key when making your selections.
7 Click OK. 8 One the Computers to Add to Policy window, click the check-boxes to select the
individual computers from which to collect data.
272
Remember the file name and the location, as you will need this information as you build the Manager script.
User Defined directs Analyze to aggregate workload usage based on the default or user defined workloads, regardless of whether or not the data comes from a system configured as a Xen domain. This option is the default and applies to data collected from any supported OS configuration. If you are collecting data from nodes, however, keep in mind that this option will aggregate the workload utilization at the node level, not at the individual domain level. Partition directs Analyze to aggregate the workload usage based on the zone the
workload was executing in, regardless of the default or user defined workloads. This option applies only to data collected from a Solaris 10 system that has been software partitioned using the Solaris 10 Containers technology, with zones configured.
s
Partition directs Analyze to aggregate the workload usage based on the node the
workload was executing in, regardless of the default or user defined workloads. This option applies to data collected from a Xen (Citrix Xen or Open Source Xen) system that has been software partitioned using the Xen technology, with nodes and domains configured.
s
Project directs Analyze to aggregate the workload usage based on the project setting from the FSS Config file, regardless of user defined workloads. One of the configuration options in Solaris 10 is to create a project as a way of managing resources for processes running in a zone. This option applies only to data collected from a Solaris 10 system that has been software partitioned using the Solaris 10 Containers technology, with zones and projects configured.
If you want the default view of workloads, then proceed to Task 3: Set up the Automator script on page 275. If you are interested in viewing the workload utilization aggregated by partition in Visualizer or Perceiver the Manager script file that automates the data collection and data processing tasks requires a workload characterization that builds workloads by either the partition or project option. This is necessary so that you can properly view the performance data in Visualizer or Perceiver.
273
1 Right-click the Analyze snap-in, then choose New Workload Characterization. 2 When the welcome page of the Analyze Workload Characterization Wizard opens,
click Next.
4 In the Analyze Properties page select the appropriate check boxes. 5 Click Finish. This launches the Computer Wizard. Use this wizard to define the
computers that are to be included in the Analyze workload characterization file. These computers will be represented in the Visualizer and Perceiver graphs that you will later view.
7 From the Build workloads by drop-down list, select Partition or Project, depending
on how you want workload usage to be grouped in Visualizer.
274
TIP
Run your script against a small or duplicate database first, previewing and testing it before running it against the production database.
For more information, refer to the Visualizer online Help or the Visualizer Implementation Guide.
1 From the Automator File menu, choose one of the default scriptname.b1a files to
open the Select a Database from the Catalog dialog box. New Daily Script (daily.b1a) New Weekly Script (weekly.b1a) New Monthly Script (monthly.b1a)
2 Select one of the databases from the Database selection list. When you click OK,
Visualizer creates a script and names it with capital letters. For example, if you select demo.b1a, after you click OK, the newly created script is named DEMO.B1A.
3 Review the events in the script and modify as necessary. To change an event,
highlight the event line in the script. Then do one of the following: Choose Edit => Modify. From the keyboard, press Alt+Enter.
4 Choose Run => Preview Mode. 5 Choose Run => Script Now. Click Yes when you see the following prompt.
Preview mode is enabled C the database will not be altered in any way. Are you sure you want to preview the script: scriptname now?
6 Review the log and correct any errors in the script. 7 From Automator, choose File => Save As. The script is now available to be included
in the Manager script.
275
1 Right-click the Manager snap-in, right-click Scripts, and choose New Script from the
menu. This starts the New Script Wizard. Read the Welcome page and click Next to proceed to the Manager Script File Name page.
2 Enter a name for your script. Use the default folders provided, unless you have set
up alternative locations for script files and reports. Click Next.
3 On the Operation window, select Collect New Data. Leave the default selection of
System and applications data.
4 Click Add to browse to the policy that has the computers you identified in Task 1:
Set up a policy that includes the target Xen systems on page 272. On the Select Policy File window, select the policy file and click Open.
5 On the Operation window, click Next. The Analyze and Predict window is
displayed. The Analyze and Predict window lets you decide how you will use the data that you collect. You can generate reports for Analyze, Predict, and Applications (if you selected that option on the previous window). You can also generate Visualizer files and create workload characterization files.
276
7 On the Select Analyze file dialog, click the custom workload you created in Task
2: Create a custom workload in Analyze on page 273, and then click Open.
8 On the Analyze and Predict window, click Next. 9 On the Collect or Analyze Interval window, use the up and down arrows in the
From and To fields to establish the collection interval.
10 Select the time zone, and then click Next. You will need to change the time zone
setting if data is being collected from computers in time zones that differ from the time zone of the console computer.
11 On the Automator Script Option window, click Browse. Use the Automator Script
Option page to specify the Automator script to be executed during the Manager run. Automator is a Visualizer tool that runs user-built scripts to populate a Visualizer database. The Automator script you specify will use Visualizer files you generated with Predict and Analyze to populate the Visualizer database. (See the Visualizer Implementation Guide for information about using Automator scripts.)
277
12 On the Select Automator File dialog, browse to the Automator script you created in
Task 3: Set up the Automator script on page 275, and then click Open.
13 On the Automator Script Options window, click Next. 14 On the Schedule a run of the script window, select the Schedule the script to run
checkbox. Click the up and down arrows to set the time to run the script. Set the recurrence options for the script in the Run script section.
15 Click Finish to save the script and set the schedule. Task 5: Verify the data collection
Using the Manager component has the added advantage of allowing you to view collection status with the Status Report feature. To access the Status Report feature:
1 In the BMC Performance Assurance console scope pane, open the Manager snap-in. 2 Click Status Report.
The Perform - Data Collection Status opens in the results pane. The available scripts are in the left portion of the report. To view the status of data collection, click on a script. You can also click Schedule in the scope pane to display the scripts in the results pane. Then right-click on the script for which you want to view reports and select View Status Report.
278
See the online help, available from the help button on the top right portion of the Status Report, or refer to the appendix on Troubleshooting Tools in the BMC Performance Assurance Getting Started for more information about using these reports.
Creating a performance model for use by Predict Viewing charts of performance data Creating performance reports Creating input files for Visualizer
When you run Analyze, it uses the workload characterization to evaluate collected performance data, and create a performance model, reports, and, optionally, Visualizer input files. To generate the Xen Summary report, perform the following steps:
1 Right-click Analyze in the Tree view, and choose Open Workload Characterization. 2 Choose the appropriate workload characterization (.an) file for the Xen system. 3 Review the displayed interval or set another interval. 4 Run Analyze by clicking on the green arrow in the menu bar, or right-click on the
Workload Characterization in the Tree view and choose Start Analyze.
5 Expand the Reports item and then expand the Xen selection.
279
The following table describes each of the columns in the report. You may need to scroll to the right to see some of the report columns.
Column Xen Host Description The name of the Xen node that provides all of the management functions for the Xen domains. For reporting purposes, this name is used to refer to the physical host machine. This value is used to refer to the Xen domain. Each domain is identified by its display name specified in the Xen configuration file. The operating system type installed as a guest OS on the VM. Each VM can run Microsoft Windows or Linux as guest OSs. The number of physical processors on the Xen server system. This value is shown only for the top-level server information.
Logical Processors The number of logical processors on the Xen server system. This value is shown only for the top-level server information. Virtual Processors The number of CPUs allocated to the Xen domain. This value is shown for each Xen domain. CPU Util The percentage of CPU time used by a Xen domain. The CPU usage shown in this column is a physical value; it represents the total CPU usage of the amount of CPU that has been allocated for the Xen domain. The number of CPU shares assigned to a Xen domain. The guaranteed minimum CPU percentage reserved for this Xen domain.
280
Description The maximum CPU percentage for a Xen domain. The valid range of values is from zero (0) to the number representing the total physical CPU resources. The maximum may be greater than 100 for SMP virtual machines that may use more than one full physical CPU. The amount of RAM (in MB) for the physical machine. This value is shown only for the top-level Xen node service console row. The memory required for single copy of memory shared between Xen domains. The sum of all memory used for running Xen domains. Specifies the type.
Xen Partition Overview s Partition Overview s Partition Metrics Xen Host Summary by Resource s CPU Profile s Memory Profile s IO and Network Usage Xen Partition Summary by Resource s CPU Profile s Memory Profile s IO and Network Usage
281
Partition Overview
Description: This view displays multi-partition charts for the essential metrics (Physical CPU Capacity Used, Virtual CPU Capacity Used, IO Rate, Physical Memory Used, Network Rate) in your datacenter, in line charts by group of root partitions.
This example shows that all computers in the group have been chosen. You can view information for a particular partition by clicking the drop-down list next to the Computer field.
282
The ranking table (partially shown in Figure 134 and shown expanded in Figure 135 on page 283) shows which partitions (domains) are approaching the resource usage limits on their Xen hosts (zones). It also shows where the computers being measured fall in increments of 10 percentage points, based on a blended value of the essential metrics (Physical CPU Capacity Used, Virtual CPU Capacity Used, IO Rate, Memory Utilization, Network Rate). Figure 135 Xen Partition Overview Ranking Table
This example provides information for all computers in the group. However, if you want to view this information for a particular subset, for example the partitions that fall in the 80 - 100 percentile, you can select those computers and click the filter icon to see only those graphs. Figure 136 and Figure 137 show how this might look. Figure 136 Filtering computers in the Ranking Table
283
Physical CPU Capacity Used - Partition Physical CPU Capacity Used Virtual CPU Capacity Used - Partition Virtual CPU Capacity Used Memory Utilization - Partition Memory Usage Network Rate - Partition Network Rate IO Rate - Partition Total IO Rate
284
CPU Profile
CPU Profile shows the following information for the host that is chosen:
s s
s s s
CPU Utilization by Partition - CPU Utilization - per Partition Virtual CPU Capacity Used by Partition - Virtual CPU Capacity Used - per Partition % Virtual CPU Used - Xen % Virtual CPU Used Virtual Processors Configured - Virtual Processors Configured - per Partition Host System Hardware Configuration - Xen Host System Hardware Configuration
Below the CPU Profile graphs (Figure 139 on page 286), the Host System Hardware Configuration table displays information about the host hardware configuration. You can expand this table by clicking the plus sign in the top right portion of the table.
285
Memory Profile
The Memory Profile view shows the memory configuration, memory utilization, and memory swapping rate for the partitions located on the host computer chosen from the Computer drop-down list. Figure 140 Memory Profile
286
CPU Profile
CPU Profile for individual partitions shows:
s
s s s
Physical and Virtual CPU Capacity Used - Partition Physical CPU Capacity Used, Partition Virtual CPU Capacity Used Virtual Processors Configured - Partition Virtual Processors Configured Partition System Identification and Configuration Host Physical and Logical Processors Configured - Xen Host Physical Processors Configured, Xen Host Logical Processors Configured Host System Hardware Configuration - Xen Host System Hardware Configuration
Chapter 7 Implementing Performance Assurance on Xen Systems 287
Below the CPU Profile graphs (Figure 142 on page 288), the Host System Hardware Configuration table displays information about the host hardware configuration. You can expand this table by clicking the plus sign in the top right portion of the table. Figure 142 Xen Partition Summary by Resource - CPU Profile
Memory Profile
The Memory Profile view shows the memory utilization, and memory swapping rate for the partition chosen from the Computer drop-down list. Figure 143 Xen Partition Summary by Resource - Memory Profile
288
289
verify or correct the information in the model identify any resources that might be exceeding your service level objectives. The guideline and threshold apply only to the Xen node.
TIP
A model containing Xen nodes and the associated domains is much like a model containing physical machines. You can change the node or domain configuration settings (such as CPU, memory, CPU shares, and so on), prior to baselining.
For more information on calibrating the model and creating a valid baseline, refer to the Predict online Help.
290
For either the Xen host or the partition itself, you can click the Virtual System Allocation button on the Configuration tab to see how the system resources are being allocated across the partitions, as shown in Figure 147. You can reassign the physical system resources among the virtual machines using this dialog box. Figure 147 Resource allocations for partitions
291
Expand the Reports tree in the scope pane. Expand the Computer tree. Select the Xen Summary report. Predict displays the report in the results pane, as shown in Figure 148.
292
The following table describes the columns in this report. You may need to scroll to the right of the report to see some of these columns.
Column Xen Host Description The name of the Xen host (zone) that provides all of the management functions for the VMs (domains). For reporting purposes, this name is used to refer to the physical Xen Host Server. This value is used to refer to the VM. Each VM is identified by its display name specified in the Xen configuration file. This display name is not guaranteed to be unique on the Xen host. The model number of the hardware. The hardware vendor. For the Xen Host, this is the total number of physical processors. For VMs, this is the number of virtual processors. For the Xen Host, this is the percent of CPU utilization out of online processors * 100. For the VM, this is the percent of CPU utilization out of physical online processors * 100, with a maximum value of the number of virtual processors * 100. For the Xen Host this equals the online processors * 100. The total disk I/O for the Xen Host is the sum of the I/Os for all the VMs running on that host.
VM System
293
Description For the Xen Host this is the total physical memory. For the VMs, this is the total virtual memory. The memory for the Xen Host is not the sum of the virtual memory of the VMs. The percentage of CPU wait time due to all virtual machines (including itself) on the Xen Host Server. The percentage of CPU wait time due to activity from other virtual machines.
294
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
Index
Numerics
1 Min Average CPU Load 189 15 Min Average CPU Load 189 5 Min Average CPU Load 189 from Solaris 10 systems 181, 241, 271 verifying the data collection of Solaris 10 systems 248, 278 verifying the data collection of Virtual Servers 106 verifying the data collection of VMware systems 36 collecting VMware data 24 Configuration requirements in an AIX environment 130 configuration tasks set up VMware proxy data collection 22 configure_ssh configuration file 132 Containers and zones overview 16 Copying the public key 135 CPU Activity by virtual machine 110 CPU System by VM 49 CPU Use Hierarchy graph for Containers 249 CPU User by VM 49 CPU Utilization by VM 49 creating a policy 32, 102, 182, 242, 272 custom workload characterization file 183, 243, 273 customer support 2
A
AIX partitions introduction 12 types of 12 Analyze Build workload options 183, 243, 273 Analyze Physical System Report for AIX Partitions 140 for HP-UX Partitions 204 Analyze reports for VMware 38, 40, 42, 43 Shared Processor Partition Report for SPLPARS 142 Automator script setting up for Solaris containers 185, 245, 275 setting up for Virtual Server 104 setting up for VMware 33 Avg. Run Q Len 189 Avg. Swap Q Len 189
D
Displaying AIX hardware partition CPU usage in Visualizer 144 Displaying data from AIX partitioned systems 138 from HP-UX partitioned systems 201, 213 from Microsoft Virtual Servers 107 from VMs 37 Displaying HPhardware partition CPU usage in Visualizer 205 Displaying Hyper-V CPU usage in Visualizer 111, 259 Displaying VM CPU usage in Visualizer 44 DLPARs, defined 12 Dynamic System Domains, described 15
B
BMC Software, contacting 2 Buffer Block Reads description 189 Buffer Block Writes description 189 Buffer Cache Size 189 Build workload by Project setting 243 User Defined setting 183, 243, 273 Zone setting 183, 243
C
Collecting data from a HP-UX partition 201 from a Microsoft Virtual Server 101 from an AIX partition 136
E
Enabling remote command execution on the HMC 131
Index
295
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
H
Hardware Management Console (HMC), described 13 HP-UX partitions implementing Perform 199 introduction 14
installing Perform Agent 99, 100 introduction 11 Modeling AIX partitioned systems 164 HP-UX partitioned systems 207 VMware systems 61, 289
I
identifying VMware hosts and VMs in Predict 61, 290 implementing Perform in a Microsoft Virtual Server environment 98 in a VMware environment 20 on a Sun DSD partition 239 on Solaris 10 systems with Containers 241 implementing Perform on an HP partition 199 Installation considerations for Solaris 10 systems 130, 240 for Solaris DSDs 240 in a AIX environment 129 in a HP-UX environment 200 in a VMware environment 21 installation considerations for Perform Agent in Microsoft Virtual Server environment 99, 100 Installing SSH on the AIX partition 134 introduction to AIX partitions 12 IO Activity by virtual machine 110
N
Num System fork calls 189
O
overview of Dynamic System Domains 15 overview of HP partitions 14 overview of Microsoft Virtual Server 11 overview of Solaris 10 Containers 16 overview of Solaris partitions 15 overview of VMware 10
P
Page Ins 189 Page Outs 189 Pages Scanned 189 Perceiver views Solaris Container Overview 252 Perform Agent installation considerations for Microsoft Virtual Server 99, 100 installation considerations for VMware 21 Pg Ins Pging 189 Pg Outs Pging 189 Physical Reads description 190 Physical System Report for AIX Partitions 140 for HP-UX Partitions 204 Physical System Summary Report for AIX partitions 165 for HP-UX Partitions 208, 232 Physical Writes description 190 Predict Physical System Summary Report for AIX Partitions 165 for HP-UX Partitions 208, 232 Process Context Switches 189 product support 2 Project option for building workloads 243
L
Logical Buffer Reads description 189 LPARs, defined 12
M
Manager script setting up for Microsoft Virtual Servers 105 setting up for Solaris containers 186, 246, 276 setting up for VMware 31, 34 Manually configuring the AIX environment 134 Manually creating a user on the HMC 135 Manually generating the RSA or DSA keypair 135 Message definition 190 metric groups for AIX partitions 138 for HP-UX partitions 202, 214, 218 for Microsoft Virtual Servers 101 for Solaris 10 zones 254 Microsoft Virtual Machine Additions, how to install 100 Microsoft Virtual Server collecting data from 101 implementing BMC Performance Assurance 98
R
Run Occ 189 Run Q 189
296
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
S
Semaphore definition 190 setting up a Manager script 31, 34, 105, 186, 246, 276 setting up a policy for Microsoft Virtual Server systems 102 for Solaris systems 182, 242, 272 for VMware systems 32 setting up an Automator script 33, 104, 185, 245, 275 Shared Processor Partition Report for SPLPARS 142 Solaris Container Overview 252 SPLPARs, defined 12 Sun partition implementing Perform 239 implementing Perform on a systems using Containers 241 Sun Solaris partitions Containers and zones 16 Dynamic System Domains 15 introduction 15 support, customer 2 Swap Space Ratio (Unix) 189 Swap Space Utilization description 189 System Hardware Configuration 110
VMware implementing BMC Performance Assurance 20 install considerations of Perform Agent 21 introduction 10 VMware ESX Server virtualization layer, defined 10 VMware ESX Server, defined 10 VMware CPU by Host/VM graph 46 VMware Disk Report 42 VMware Memory Report 40 VMware Per Processor Report 43 VMware proxy host requirements 24 setting up for data collection 22 VMware Summary Report 38
W
working with VMware systems in Predict 61, 289
Z
Zone option for building workloads 183, 243 Zones overview 16
T
technical support 2 Total 189 Total Pg Pging 189 Types of AIX partitions 12
U
Used Memory (Unix) 189 User Defined option for building workloads 183, 243, 273
V
verifying the data collection of Solaris 10 systems 248, 278 verifying the data collection of Virtual Server systems 106 verifying the data collection of VMware systems 36 Virtual CPU Capacity Used 49 Virtual Processors Configured by VM 49 Virtual Systems, introduction 9 Visualizer CPU Use Hierarchy graph for Containers 249 displaying graphs for AIX 144 displaying graphs for HP 205 displaying graphs for Hyper-V 111, 259 displaying graphs for VMware 44 VMware CPU by Host/VM graph 46
Index
297
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
298
Notes
*202938*