Optimizing Troubleshooting Guide English

Action Request System 5.
1 Optimizing and Troubleshooting AR System
PART NO: AR-512-OTG-01
Copyright 2003 BMC Software, Inc. All rights reserved. Remedy, the Remedy logo, all other Remedy product or service names, BMC Software, the BMC Software logos, and all other BMC Software product or service names, are registered trademarks or trademarks of BMC Software, Inc. All other trademarks belong to their respective companies. Remedy, a BMC Software company, considers information included in this documentation to be proprietary and confidential. Your use of this information is subject to the terms and conditions of the applicable end user license agreement or nondisclosure agreement for the product and the proprietary and restricted rights notices included in this documentation. Restricted Rights Legend
U.S. Government Restricted Rights to Computer Software. UNPUBLISHED -- RIGHTS RESERVED UNDER THE COPYRIGHT LAWS OF THE UNITED STATES. Use, duplication, or disclosure of any data and computer software by the U.S. Government is subject to restrictions, as applicable, set forth in FAR Section 52.227-14, DFARS 252.227-7013, DFARS 252.227-7014, DFARS 252.227-7015, and DFARS 252.227-7025, as amended from time to time. Contractor/Manufacturer is BMC Software, Inc., 2101 CityWest Blvd., Houston, TX 77042-2827, USA. Any contract notices should be sent to this address.
Contacting Remedy If you need technical support for this product, or would like to request documentation for a product for which you are licensed, contact Remedy Customer Support by email at support@remedy.com. If you have comments or suggestions about this documentation, contact Information Development by email at doc_feedback@remedy.com. This edition applies to version 5.1 of the licensed program.
Remedy, a BMC Software company 2350 Bayshore Parkway Mountain View, CA 94043 Tel 650.903.5200 Fax 650.903.9001
www.remedy.com
Table of Contents
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Overview of This Manual . . . . . . . . . . . . . . . . . . . . . . . . . 8 Action Request System Documents . . . . . . . . . . . . . . . . . . . . 9 Chapter 1 Managing Performance . . . . . . . . . . . . . . . . . . . . . . . . 11 Understanding Performance . . . . . . . . . . . . . . . . . . . . . . 12 Establishing Performance Goals . . . . . . . . . . . . . . . . . . . . . 12 Gathering Baseline Information . . . . . . . . . . . . . . . . . . . . . 12 Using Server Statistics to Gather Baseline Data . . . . . . . . . . . . . 14 Using AR System Log Files to Gather Baseline Data . . . . . . . . . . . 15 Using AR System Analyzer . . . . . . . . . . . . . . . . . . . . . . . 16 Troubleshooting Performance Problems . . . . . . . . . . . . . . . . . 20 Troubleshooting AR System Alert Performance . . . . . . . . . . . . . 22 Troubleshooting Server Processes on Windows . . . . . . . . . . . . . 22 Tuning System Performance . . . . . . . . . . . . . . . . . . . . . . 22 Peak Use Considerations . . . . . . . . . . . . . . . . . . . . . . . 22 Hardware Considerations . . . . . . . . . . . . . . . . . . . . . . 23 Memory Configuration . . . . . . . . . . . . . . . . . . . . . . . 24 Multithreaded Server Configuration . . . . . . . . . . . . . . . . . . 24 System Load Management . . . . . . . . . . . . . . . . . . . . . . 28 User Actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Data Archiving . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Table of Contents! 3
Action Request System 5.1
Database Performance Tuning . . . . . . . . . . . . . . . . . . . . . 29 Maximizing the Memory Allocated to the DBMS . . . . . . . . . . . . 29 Advanced Database Performance Tuning . . . . . . . . . . . . . . . . 30 Optimizing AR System Searches. . . . . . . . . . . . . . . . . . . . . 34 Creating Effective Indexes . . . . . . . . . . . . . . . . . . . . . . 34 Defining Effective Searches . . . . . . . . . . . . . . . . . . . . . . 36 Optimizing AR System Applications and Workflow . . . . . . . . . . . . 39 Designing Active Links . . . . . . . . . . . . . . . . . . . . . . . 39 Designing Filters . . . . . . . . . . . . . . . . . . . . . . . . . . 40 Designing Escalations . . . . . . . . . . . . . . . . . . . . . . . . 40 Limiting Blocking Actions . . . . . . . . . . . . . . . . . . . . . . 41 Designing Set Fields and Push Fields Actions . . . . . . . . . . . . . . 41 Designing $PROCESS$ Actions . . . . . . . . . . . . . . . . . . . . 42 Designing Join Forms . . . . . . . . . . . . . . . . . . . . . . . . 42 Designing Fields . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Designing Form Views. . . . . . . . . . . . . . . . . . . . . . . . 43 Chapter 2 Using Log Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Tracing Guide Activity . . . . . . . . . . . . . . . . . . . . . . . . . 46 Server Error Log . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 Server Trace Modes . . . . . . . . . . . . . . . . . . . . . . . . . . 46 Trace Log Entry Format . . . . . . . . . . . . . . . . . . . . . . . 48 Alert Log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 arforkd Log (UNIX only) . . . . . . . . . . . . . . . . . . . . . . 50 API Log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 Distributed Server Log . . . . . . . . . . . . . . . . . . . . . . . . 51 Escalation Log . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 Filter Log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 Plug-In Server Log . . . . . . . . . . . . . . . . . . . . . . . . . 56 SQL Log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 Thread Log . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 User Log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
4 " Table of Contents
Optimizing and Troubleshooting AR System
Client Trace Modes . . . . . . . . . . . . . . . . . . . . . . . . . . 62 Activating Workflow Logging in AR System Windows User Tool . . . . . . 62 Activating Workflow Logging in a Web Browser . . . . . . . . . . . . . 63 Log Information for Active Links . . . . . . . . . . . . . . . . . . . 63 Log Information for Macros . . . . . . . . . . . . . . . . . . . . . 64 Combined Server and Client Logging . . . . . . . . . . . . . . . . . . 65 Creating the Client Logging Group . . . . . . . . . . . . . . . . . . 66 AR Log Display Tool for Log Files . . . . . . . . . . . . . . . . . . . . 69 Displaying Real-Time Log Information. . . . . . . . . . . . . . . . . 69 Viewing Log Files . . . . . . . . . . . . . . . . . . . . . . . . . . 71 Full Text Search Log . . . . . . . . . . . . . . . . . . . . . . . . . . 71 Mid Tier Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 Other Trace Modes . . . . . . . . . . . . . . . . . . . . . . . . . . 73 Additional Resources . . . . . . . . . . . . . . . . . . . . . . . . . 74 Chapter 3 Debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 Debugging an AR System Application . . . . . . . . . . . . . . . . . . 75 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 Debugging Process . . . . . . . . . . . . . . . . . . . . . . . . . 77 Debugging Tools . . . . . . . . . . . . . . . . . . . . . . . . . . 79 Debugging Example . . . . . . . . . . . . . . . . . . . . . . . . . 80 Chapter 4 Searching Database . . . . . . . . . . . . . . . . . . . . . . . . . . 81 Searching for Objects . . . . . . . . . . . . . . . . . . . . . . . . . 82 Setting Up the Search Database . . . . . . . . . . . . . . . . . . . . 83 Synchronizing the Search Database . . . . . . . . . . . . . . . . . . 84 Searching for Objects on a Server . . . . . . . . . . . . . . . . . . . 86 Saving Searches . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 Deleting Saved Searches . . . . . . . . . . . . . . . . . . . . . . . 91 Saving Results to a Workspace . . . . . . . . . . . . . . . . . . . . 91 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Table of Contents! 5
6 " Table of Contents
Preface
Audience
This guide is written for administrators who are responsible for monitoring and maintaining Action Request System (AR System) once AR System is installed and the basic environment has been established. It is intended to aid new and current administrators of AR System. This guide builds upon the topics and strategies discussed in the Developing AR System Applications: Basic guide as well as the topics and strategies discussed in the Developing AR System Applications: Advanced guide. If you are a current AR System administrator, this guide helps you enhance the ease of use, optimize the performance, and troubleshoot the problems encountered with your AR System environment. If you are a new AR System administrator, this guide helps you create an effective and efficient AR System environment. Before you explore the topics discussed in this guide, ensure that you understand the terms and concepts discussed in the Developing AR System Applications: Basic guide, because that book contains all the required information for setting up and administering a basic AR System environment. Your knowledge of basic administrative AR System tasks is crucial if you are to implement the strategies discussed in this guide.
Preface ! 7
You must know how to use AR System, including AR System Administrator, AR System Windows User Tool, and AR System Import. Refer to the Installing AR System guide and the Developing AR System Applications: Basic guide for additional information.
Overview of This Manual

!
Chapter 1, Managing Performance, provides information and instructions for fine tuning your AR System environment. This chapter helps you organize your AR System to run faster and more efficiently. Chapter 2, Using Log Files, provides troubleshooting information for various log files and trace modes. Chapter 3, Debugging, is a compilation of essays concerning AR System and the platforms on which it runs. Chapter 4, Searching Database, describes setting up and synchronizing the Search database so that you can search for objects on a server.
8 " Preface
Action Request System Documents

Title and Part Number AR System Concepts Guide AR-512-CG-01 Description Audience Format Print and PDF Overview of AR System architecture and Everyone features with in-depth examples; includes information about other AR System products as well as a comprehensive glossary for the entire AR System documentation set.
Developing AR System Applications: Basic AR-512-DABG-01 Developing AR System Applications: Advanced AR-512-DAAG-01 Configuring AR System AR-512-CFG-01 Optimizing and Troubleshooting AR System AR-512-OTG-01
Basic procedures for creating and Administrators Print and PDF modifying an AR System application for tracking data and processes. Advanced procedures for extending and Administrators Print and customizing AR System applications. PDF Server administration topics on Administrators Print and configuring servers and the mid tier, and PDF maintaining AR System. Administrators Print and Server administration topics and PDF technical essays related to monitoring and maintaining AR System for the purpose of optimizing performance and troubleshooting problems. Database administration topics and rules Administrators Print and PDF related to how AR System interacts with specific databases; includes an overview of the data dictionary tables. Server administration and procedures for Administrators Print by implementing a distributed AR System special server environment with the Distributed order and Server Option. PDF Information about AR System data structures, API function calls, and OLE support. Administrators Print by and special Programmers order and PDF
AR System Database Reference Guide AR-512-DRG-01 AR System Distributed Server Option Administrators Guide AR-512-DSOG-01 AR System C API Reference Guide AR-512-CAPI-01 AR System Java API
Information about Java classes, methods, Administrators HTML and variables that integrate with and AR System. Programmers Procedures for installing and licensing AR System. Administrators Print and PDF
Installing AR System AR-512-IG-01
Action Request System Documents ! 9
Title and Part Number AR System Email Engine Guide AR-512-EEG-01
Description Procedures for installing, configuring, and using the AR System Email Engine.
Audience
Format
Administrators Print by special order and PDF Administrators Print by and special Programmers order and PDF Everyone Everyone Print only Print and PDF Help menu
AR System Error Messages Guide AR-512-EMG-01 AR System Master Index AR-512-MI-01 AR System Release Notes AR-512-RN-01 AR System Windows User Tool Help AR System Import Help AR System Administrator Help
List of AR System error messages with expanded descriptions.
Combined index of all books. New features list, compatibility lists, international issues, open and fixed issues. Procedures for using AR System Windows User Tool.
Everyone
Procedures for using AR System Import. Administrators Help menu Procedures for creating and modifying an AR System application for tracking data and processes. Procedures for using AR System Alert. Administrators Help menu Everyone Help menu
AR System Alert Help
Unless otherwise noted, online documentation is available in Adobe Acrobat (PDF) format on AR System product installation CDs and/or on the Customer Support web site.
10 " Preface
CHAPTER
Managing Performance
As an AR System administrator, you have many tools and methods available to monitor and enhance AR System performance. This chapter describes the following topics:
! ! ! ! ! ! ! ! !
Understanding Performance on page 12 Establishing Performance Goals on page 12 Gathering Baseline Information on page 12 Using AR System Analyzer on page 16 Troubleshooting Performance Problems on page 20 Tuning System Performance on page 22 Database Performance Tuning on page 29 Optimizing AR System Searches on page 34 Optimizing AR System Applications and Workflow on page 39
For additional information, the Remedy Education services training course, AR System Performance Tuning and Troubleshooting, provides training in all aspects of performance tuning.
Managing Performance ! 11
Understanding Performance
Performance refers to operating efficiency, or response time. Many factors affect performance, from hardware components to application design. Managing performance, called performance tuning, involves establishing performance goals, gathering baseline data, monitoring system behavior, and troubleshooting specific problems. Individual forms can affect performance as well. For information on analyzing forms, see Analyzing Forms for Performance on page 19.
Establishing Performance Goals

It is important to establish and document performance goals before you begin making adjustments. The documented goals serve as guidelines against which you can measure system performance to determine where you may need to make adjustments. If performance goals reflect ideal conditions, system performance may not match the goals, but goals are a valuable tool for evaluating performance. You may want to gather baseline data first, and establish goals that reflect your system capacity. Additionally, when you must troubleshoot a specific performance problem, you can set mini goals focused on that problem. See Troubleshooting Performance Problems on page 20 for a sample troubleshooting process.
Gathering Baseline Information

Baseline data (normal operating levels) is useful for comparing normal performance to performance when problems occur. With baseline data, you can determine whether your performance tuning efforts are successful, and identify behavior that is unusual for your system. You can design baseline tests that run at regular intervals. If a test fails, you can use this information to determine if there is a problem. You can also manually run baseline tests during troubleshooting procedures, to compare the results of your tuning efforts with your baseline data. See Troubleshooting Performance Problems on page 20 for information and a sample troubleshooting process.
12 "Chapter 1Managing Performance
Before gathering baseline information, create an analysis of your system components. As you tune your system, you can record the changes you make to the configuration, and note the effects of these changes. To create your analysis, answer the following questions:
!
What is your hardware configuration?

! ! ! !
What is the server platform configuration? How much RAM is installed? What is the system processor configuration? What is the disk subsystem configuration? How many PC and UNIX clients are part of your system? Do your client tools run remotely? Is it local or remote? What type and quantity of data spaces and devices exist? How much memory is allocated?
What is your AR System configuration?

! !
What is your database configuration?

! ! !
What non-AR System applications are running on the client or server? When do they run? What diagnostic operations (disk defragmentation, and so on) are running on the server? On the client? Have you configured fast and list queues? If so, what are the Min and Max values for each queue?
After you have created a system analysis, you can determine how efficient the system is through specific measurements and through user perceptions. Users typically measure AR System performance by the response time of individual operations, such as logging in to the server and searching for data in the database. Response time is the number of seconds it takes for these operations to complete; for example, the amount of time it takes for the results to display after a user initiates a search. Users may alert you when they perceive performance problems. Measurements can be obtained through various methods, including gathering server statistics.
Gathering Baseline Information ! 13
Using Server Statistics to Gather Baseline Data

You can use the Server Statistics form (shown in Figure 1-1) to gather statistical information about the performance and operation of your server. This form is automatically installed when you install AR System. The form gathers statistics about the time spent in various types of calls and the number of operations being performed in the system. In addition, it contains information about the number of current users and the number of bad password login attempts.
Figure 1-1: Server Statistics Form
When activated, the AR System server creates an entry that contains the current value of all the server statistics at an interval you specify. The statistics can be gathered at the server level (summarized across all queues) or at a per-queue level. The fields in the Server Statistics form record the same information that the ARGetServerStatistics function records. For a complete list of the options and their descriptions, refer to the ARGetServerStatistics function in the AR System C API Reference Guide. To begin recording server statistics, update the Server Statistics settings in the Server Information window (as described in the Configuring AR System guide). You can also record statistics by updating the Server-Stats-Rec-Mode and Server-Stats-Rec-Interval options of the ar.conf (ar.cfg) file, as described in the Configuring AR System guide. To view the statistics, open the Server Statistics form in Search mode, specify the criteria for the recording you want, and click the Search button.
Using AR System Log Files to Gather Baseline Data

AR System provides a variety of log files as a resource for gathering different types of data. Log files contain information that you can use to evaluate performance. Some log files are generated automatically, while others must be enabled before they begin to log operations. API, SQL, and filter logs are particularly useful for monitoring performance. Running a log file requires a portion of your system resources, thereby affecting performance. Writing log files in Action Request System will affect performance for multi-threaded systems, because only one thread at a time can write to a log. You do not need to have a log file running at all times. If you have some idea as to what actions might be causing performance problems, enable a log file, perform the suspected actions, and then turn off the log file. This generates a short log file of only the actions performed for the logged operation. The Full Text Search (FTS) log quickly becomes very large, using a large amount of disk space. FTS logging also slows down the re-index operation. You should not turn on the FTS log during a re-index operation, unless you are troubleshooting a specific issue. For a description of the different types of log files, see Chapter 2, Using Log Files.
Gathering Baseline Information ! 15
Using AR System Analyzer

When you create a new form, you can analyze it for potential performance problems before users begin using it. Use the Analyze Forms dialog box in AR System Administrator, shown in Figure 1-2, to analyze one or more forms on a server. For instructions, see Analyzing Forms for Performance on page 19.
Figure 1-2: Analyze Forms Dialog Box
The following table describes the performance considerations of each criterion on the dialog box. Each check box returns the field names (if any) that match each performance criterion. These criteria are also discussed in more detail in the section Defining Effective Searches on page 36.
Table 1-1: Analyze Performance Criteria (Sheet 1 of 2)
Criteria
Description of Problem
Solution To reduce storage allocation, minimize the number of fields that are longer than 255 bytes (or 4000 bytes for Oracle).
Character Fields When the maximum field Longer than 255 length exceeds 255 bytes, Characters database storage is allocated in blocks that average from 1K to 2K. These fields are accessed and stored differently than other data and is slower to access. Oracle has a limit of 4000 bytes, but the system check is set to 255 for consistency across databases. Character Fields Searches become more time with QBE = consuming when the Anywhere query-by-example (QBE) search style is set to find text Anywhere within a text string. Performance is reduced when these character fields are used in a search operation. $PROCESS$ in Field Value
To minimize performance impact, change the QBE match setting from Anywhere to Leading or Equal if you do not anticipate embedded substring searches.
When used in a Set Fields Minimize the use of action, $PROCESS$ causes the $PROCESS$ in workflow. server to wait for the process to be completed before the next action executes. Verify that the processes running are necessary and ensure that they are running efficiently.
RUN PROCESS When running many processes Used simultaneously, operating-system performance may be reduced.
Using AR System Analyzer ! 17
Action Request System 5.1 Table 1-1: Analyze Performance Criteria (Sheet 2 of 2)
Criteria RUN MACRO in Active Link with Character Field QBE = Anywhere
Description of Problem When you run macros from active links that search on fields using a QBE match setting of Anywhere, inefficient and slow searches may result. Make sure you build efficient macros, particularly when using them in active links.
Solution Redesign the workflow to eliminate macros by using actions that directly perform the operations you want. If you require compatibility with clients prior to 5.0 or if a macro is still needed, change the QBE match setting from Anywhere to Leading or Equal if you do not anticipate embedded substring searches. Refer to your database documentation to determine character field length limitations. Reduce the size of fields to 255 characters or less if they need not be larger.
Character Fields Longer than 255 Characters and Used in Search
Some databases do not allow character fields containing more than 255 characters to be searched. Filters, escalations, search menus, and active links that use an OLE Automation, Open Window, Push Fields, or Set Fields operation cause character fields to be searched. If the fields searched exceed 255 characters, performance may be reduced. Searches become time consuming when the QBE search style is set to find text anywhere within a text string of a character field, because indexes cannot be used. Performance is also reduced when filters, escalations, and active links initiate actions that cause these fields to be searched using a qualification.
Character Fields with QBE = Anywhere and Used in Search
To minimize performance impact, change the QBE match setting from Anywhere to Leading or Equal for fields that have indexes.
Large Image
Minimize the size (in bytes) of Every image contained in a images in forms. form is stored with the form definition on the server. The system searches for a size (which is hard-coded as 25K) and lists the images that exceed that size.
Analyzing Forms for Performance

1 Log in to AR System Administrator. 2 Select a server to administer. 3 Choose Tools > Analyze to open the Analyze Forms dialog box (Figure 1-2
on page 16).
4 From the Forms list, select the form that you want to analyze for potential
performance problems. You can select one or more forms. Click while holding the Control or Shift key to select multiple forms. To select every form in the Forms list, select the Select All check box.
5 To examine each server performance problem, select the appropriate check
box in the Criteria list. Figure 1-3 on page 21 describes the performance considerations of each criteria.
6 Click Start.
The results of your analysis appear in the Report region of the Analyze Forms dialog box. If AR System discovers a potential problem with a given field on a long form, you can use the Find Field in the Modify Form window to locate the field and open its Field Properties. See the Developing AR System Applications: Basic guide for more information.
Using AR System Analyzer ! 19
Troubleshooting Performance Problems

Even systems that have been highly tuned for best performance occasionally experience specific performance problems. When a problem is discovered, gather as much information as you can about when and where the performance problem occurs and compare your findings to the performance goals that you have established. Figure 1-3 on page 21 illustrates a possible troubleshooting process. The first question might be to ask whether all users are experiencing the same performance problem, or if the problem is isolated to a specific user. You should also determine if users are receiving error messages, and obtain the message number and text. Error messages can help you isolate a problem more quickly. See the Action Request System Error Messages Guide for more information on error messages. You can also refer to your system configuration records for clues as to where the problem might originate. Additional questions to ask include:
! !
Is performance slow only when logging in to the AR System server? Does the problem occur only when submitting an AR System request or also when modifying a request or searching for data in the database? Is there a specific time of day when the server responds slowly or experiences this specific performance problem? What non-AR System applications were running on the client or server when the problem occurred? If time-outs are occurring, what RPC number appears?
The answers to these questions will help you define your troubleshooting strategy. As you make changes to improve performance, document the adjustments you make and why so that you have a record of your progress.
Figure 1-3: Performance Tuning and Troubleshooting Flowchart
Troubleshooting Performance Problems ! 21
Troubleshooting AR System Alert Performance

If your AR System Alert system is running slowly, turn on alert logging to analyze possible connection difficulties and track the timing of the operations. Use the alert log in conjunction with the SQL log to analyze the alert-related activity that occurs when alert events are added to the database. If the sending of alerts appears to be a bottleneck, adjust the maximum number of threads in the alert queue. For more information about alert logging, refer to Alert Log on page 49. For information about queues, refer to the Configuring AR System guide.
Troubleshooting Server Processes on Windows

If a process requires a DLL file that is missing from the default directory, you will receive an error message listing the missing file, and the thread running the process will hang. To solve the problem, replace the missing DLL in the installation directory, and restart the server. To avoid this problem, do not move DLL files after installation. DLLs should remain in the installation directory.
Tuning System Performance

The following sections describe ways you can enhance system performance.
Peak Use Considerations

The following AR System Administrator operations can temporarily affect AR System performance. You should not make any structural changes during production hours, and you should avoid making the following changes during peak production hours:
! !
Changing the lengths of fields Adding or deleting fields

!
Adding, deleting, or changing:

! ! ! ! !
Forms Active links Filters Menus New object groups
Creating indexes
Definition cache updates, table rebuilds, and table scans can occur as a result of making these changes. Additionally, network backups and escalations should run outside peak use times. When you back up data over the network, you leave less bandwidth for the AR System server to communicate with the clients, temporarily slowing system performance. Work with your network administrator to determine network performance solutions.
Hardware Considerations
The key resources that determine computer platform performance are the CPU itself, physical and virtual memory, and disk subsystem configurations. Application programs are typically either CPU-intensive or I/O-intensive (input/output). AR System is I/O-intensive, so memory and disk configurations are critical. Adequate RAM and a high-performance disk subsystem help minimize the amount of disk I/O, resulting in better system performance. AR System performance can be affected by the CPU performance of both the clients and the servers in the system. Individual servers, in particular, can affect overall system response time. If a server runs slowly, even the fastest client can experience performance problems. Before investing in hardware upgrades, make sure that your AR System workflow and database management system (DBMS) configurations are tuned. Bottlenecks in these areas can overload the best computer platforms.
Tuning System Performance ! 23
After tuning your AR System workflow and the DBMS, evaluate the performance of your computer platform and network to verify whether hardware needs to be upgraded. Work with your system administrator to determine hardware performance solutions, such as using a multi-CPU system, which makes more resources available if there is contention for the CPU between threads of the server process or between the server process and other processes.
Memory Configuration
The amount of physical memory on your clients and servers can affect system performance. Lack of physical memory might cause less-efficient virtual memory to be used.
Multithreaded Server Configuration

AR System multithreaded server functionality allows the server to use the underlying DBMS much more efficiently, thereby improving system performance. Without the multithreaded server, all client requests are routed through one connection to the DBMS, which is not scalable. The multithreaded server solves this problem: AR System can support thousands of users and tens of thousands of operations with optimized workflow and a proper multithreaded configuration. Refer to the Configuring AR System guide for a description of the multithreaded architecture, and instructions for configuring the server. Setting the most effective maximum and minimum threads enhances performance. Several tools can help determine how many fast and list threads to use.
AR System Administrator
Use the Server Information dialog box to set the maximum and minimum fast and list threads allowed.
!
Under Min, enter the smallest number of threads you think you will need. The Min number of threads will be started at server startup. Under Max, enter the smallest number possible to adequately handle the load on the server. Once extra threads have been started, they will continue to run, even if the load declines. Too many threads will degrade system performance.
For more information about threads, see the Configuring AR System guide.
API Logging
API logging helps determine the optimal number of fast and list servers for your system. Examine the API log for operations involving different users that occur without any lapse in time between operations. If the end time for one user is the same as the start time for the next user, then the second user operation was queued at the server, waiting for the first operation to complete. If more than 10 percent of the API calls are queueing for a particular server queue type, consider adding another thread of that queue type.
Operating System Tools

Use the utilities provided with your operating system to view the resources used, and ensure that the operating system can adequately support the number of threads you are running. For information, refer to your operating system documentation or online help.
Thread Configuration Guidelines

The optimal number of threads for each queue depends on a number of variables that can change quickly and often. The primary factors you should consider when setting thread counts are:
! ! ! !
Number and speed of CPUs Amount of available RAM Mix of fast and list operations Network bandwidth
A brief overview of these considerations is provided in the following sections. For detailed information about how these guidelines were determined, see the Remedy White Paper, Optimizing AR System Performance with Server Thread Settings, which is located on the Customer Support web site at http://supportweb.remedy.com.
Number and Speed of CPUs
The number of CPUs in the machine (or virtual machine in a partitioned system) on which the AR System server is running has a significant effect on the optimal thread settings. Consider the number of CPUs as an absolute minimum thread count for each queue. An optimal minimum thread count is approximately 1.5 to 2 times the number of server CPUs. The objective is to maintain a ready supply of work for the CPUs without consuming so many system resources that the system is bogged down maintaining them.
Amount of Available RAM
The main consideration for memory is to make sure that there is enough to support all threads. Just as the number of CPUs helps define the minimum thread settings, the quantity of server memory helps determine the maximum thread settings.
Mix of Fast and List Operations
In most cases, it can be difficult to predict the mix of operations on a production server. Using customized API programs, as described in API Logging on page 25, can help you determine if a server tends to handle more fast or list operations. Consider testing your system under load to determine appropriate settings.
Network Bandwidth
Network bandwidth and optimal thread counts do not have a direct effect on each other. However, you should be sure that you have sufficient bandwidth to support the expected level of server activity. Remember to account for traffic to and from the database if the database resides on a different machine. The amount of traffic between the server and database is usually about the same as the traffic between the server and clients. Therefore, if your database is kept on a separate server, it is likely that it will approximately double the network bandwidth requirements.
Common Misconceptions
There are some fairly common misconceptions about thread settings, and these misconceptions can lead to unreasonably high settings that hinder performance. You should avoid the pitfalls of the following misconceptions:
! !
You cannot have too many threads. The number of threads is somehow related to the maximum number of users. There is no difference between total number of users, maximum concurrent users, and concurrency within the system.
Each thread, used or not, requires some operating system and hardware resources, and setting the count too high can create significant system overhead managing those extra idle threads. That system overhead detracts from server performance. In the worst case, if the required thread memory exceeds physical memory, then the system will end up swapping thread memory to disk. Because of the disk-access delay in this situation, the system performance would immediately degrade to unacceptable levels. Setting one or more threads for each potential user is a mistake and should be avoided. The total number of users for an AR System server is of little significance when tuning the system. The largest number of concurrently active users is more significant, but still does not fully determine the number of threads that should be configured. Even when a large number of users perform operations at the same time, each of those user operations may be turned into several server operations. Most of those server operations complete so quickly that maintaining a thread for each user is wasteful. There should be only enough threads to ensure that each CPU has enough operations to perform that it never becomes idle.
System Load Management

The overall system load on your machines affects performance. In general, the more applications you have running, the slower your system runs. Other applications running in parallel with AR System contend for the same resources. These applications can demand hardware resources on the server or client. They also might access the same database as AR System, resulting in slower performance. Major system and network activities during peak work hours might compete with AR System for resources, so be aware of how other applications are being used. Consider moving applications to another server, or dedicating a server to AR System and the DBMS if necessary.
User Actions
The following user actions can affect the performance of your system:
!
Running other applications on their computers, which compete for resources Running large reports during peak work hours or running unqualified or poorly constructed searches, which contend for system resources Users should design reports to return only the data needed. Setting up automatic polling for results lists Users should eliminate automatic polling or lengthen the polling interval.
Data Archiving
Archive unnecessary and old AR System requests. The fewer records in your database, the faster the searches of the database. To archive requests, generate a report in AR Export or XML format, or use a custom API program to create archive files or DSO entries to an archive form. You can then delete the requests from AR System. To retrieve AR Export or XML files, reload them with AR System Import. As an alternative to archiving, you can store older requests in an archive form. This enables users to access the information and prevents the data from slowing work on the production form. Design this archive form with as little workflow as possible. This form should not consume as many resources as its production counterpart.
Database Performance Tuning

The following guidelines help you set up a database that runs efficiently with AR System:
!
Allocate enough memory to the database where you run your AR System. If your database does not have enough memory or data cache, your system might perform poorly. Allocate as much memory as you can spare (see the next section, Maximizing the Memory Allocated to the DBMS). Use AR System Administrator to index fields where searches are performed. Proper indexing is one of the most significant performance improvements you can implement. See Creating Effective Indexes on page 34. If your AR System is sharing a database server with other applications, your AR System might run slowly. You might need to dedicate a database server for the AR System. If you spread your AR System database over multiple devices, you can improve database performance, especially if you have a large AR System database. For more information, see Advanced Database Performance Tuning on page 30. If you run remote database servers, you increase the traffic over your network, resulting in slower system performance. In addition, remote databases consume other operating system resources on the local machines, such as memory. Ensure that your network does not decrease performance if you run a remote database. Tune your database by using the methods suggested by your database vendor. You might modify your database parameters, kernel parameters, and the physical storage device.
Work with your database administrator to determine database performance solutions.
Maximizing the Memory Allocated to the DBMS

Maximizing the amount of memory given to the DBMS allows more data to reside in memory, reducing the total number of physical disk accesses. Reading data from memory is much faster than reading data from the disk, so performance improves when the amount of memory allocated to the DBMS is maximized.
Database Performance Tuning ! 29
When a page of data is read from disk, it is placed in memory where a logical read is performed. Assuming that the page remains in memory, the next time that data is accessed, it is read from memory rather than disk. A logical read from memory is hundreds of times faster than a physical read from disk. The access times of disk drives are in milliseconds, whereas the access times of memory chips are in nanoseconds. Achieving a high ratio of logical reads to physical reads results in better performance. This is called the buffer cache hit ratio. The buffer cache hit ratio should be maintained at 85 percent or higher for optimum performance. Configuring DBMS Memory
1 Determine the total amount of memory available on the machine. 2 Calculate the amount of memory required by the operating system and any
other applications running on that machine.

3 Configure all remaining memory to the DBMS. Do not exceed the total disk
space used by the database files and devices. Consider anticipated growth.
Note: Calculate the amount of memory required by the operating system and any other applications running on that machine before maximizing the amount of memory given to the DBMS. Allocating too much memory to the DBMS may not leave enough for the operating system and other applications. This can cause overall performance to suffer.
Advanced Database Performance Tuning

When you create forms and indexes, your database allocates space for the structures. Generally, the AR System handles everything automatically. However, to control some of the options yourself, the AR System enables you to attach a clause to the SQL statements that create forms and indexes. This is an advanced feature that most AR System administrators do not need. If you are an experienced database administrator, however, you can use Database Administrator (DBA) extensions to manage database space and improve overall system performance. For example, you can improve the speed of AR System response by specifying that separate devices store indexes and form tables.
To extend the SQL statement when forms and indexes are created, you must first create a configuration file. On UNIX, create the ardb.conf file in your configuration directory. On Windows, create the ardb.cfg file in the conf directory of your AR System installation. See the Installing AR System guide for information about the location of this configuration file. For more information about the file structure and the ardb file, refer to the Configuring AR System guide.
Warning: AR System does not verify that the SQL clauses specified in your ardb configuration file are correct or safe. AR System merely attaches the SQL clause to the statement used when a form or index is created. Because you can append any SQL clause to the CREATE statement for a form or index, use this feature wisely.
When you create a form or index, AR System references the ardb configuration file to append clauses to the SQL statement. If it finds no matching information, AR System creates the form or index as it would normally. If it finds matching information, it appends the specified clause to the SQL statement that creates the form or index. When you use AR System Administrator to change the name of a form, the ardb configuration file is edited automatically to match the new name.
CREATE Statement Syntax

When you create a form or index, AR System references the ardb configuration file for clauses to append. If it finds matching information, it appends the specified clause to the SQL statement that creates the form or index. AR System appends the ardb configuration file clause to the CREATE TABLE statement as follows:
CREATE TABLE <table_name> (<column_definitions>) <ardb_clause>
AR System appends the ardb configuration file clause to the CREATE INDEX statement as follows:
CREATE INDEX <index_name> ON <table_name> (<columns>) <ardb_clause>
Example ardb Configuration Files The following sections show how your ardb configuration could be
formatted for the DB2 Universal, Informix, Oracle, Sybase, and Microsoft SQL Server databases. The examples show ardb configuration file information for the HD-Answer form. In some cases, clauses are added for indexes of the Submitter (ID 2), Status (ID 7), and Assigned To (ID 4) fields. For more information about ardb file format, refer to the Configuring AR System guide.
DB2 Example
For a DB2 database, the clauses for the indexes instruct the database to leave 10 percent of each index free for updates and insertions. In the following example, assuming that you have already created longts long tablespace, only the long columns in the tables for the HD-Answer form build in longts:
Form:HD-Answer Clause:IN userspace1 LONG IN longts Index { Id:1 Clause:PCTFREE 10}
Informix Example
If you want to create and store the HD-Answer form on dbspace seg2 of an Informix database, the format for that section of the ardb configuration file is as follows:
Form:HD-Answer Clause:In seg2
Oracle Example
For an Oracle database, the clauses for the indexes instruct the database to leave 70 percent of each index free for updates and insertions. In the following example, the tables for the HD-Answer form build on seg2:
Form:HD-Answer Clause:TABLESPACE seg2 Index { Id:2 Id:7 Clause:PCTFREE 70 } Index { Id:4 Clause:PCTFREE 70 }
Sybase and Microsoft SQL Server Example
For a Sybase or Microsoft SQL Server database, the clauses for the indexes instruct both indexes to ignore duplicate keys. To store the HD-Answer form on seg2, format the ardb configuration file as follows:
Form:HD-Answer Clause:on seg2 Index { Id:2 Id:7 Clause:with ignore_dup_key } Index { Id:4 Clause:with ignore_dup_key }
Optimizing AR System Searches

Searches are the most common AR System transactions against a database, and most AR System operations are designed around searches. Therefore, optimizing search performance should be a primary goal. Examples of AR System operations that involve searches are escalation qualifications, set fields actions, push fields actions, search menus, and table fields. The most efficient searches are often those that use an index, because indexes are designed to reduce the number of physical disk accesses of the DBMS. However, indexes may not be appropriate for all types of searches. To optimize performance, define effective search criteria that will result in the use of an index. The following sections describe methods for creating indexes and defining search criteria for optimum performance.
Creating Effective Indexes

Failing to use indexes effectively is the leading cause of performance problems in AR System applications. The goal of indexing is to minimize search response time by reducing the number of physical disk accesses of the DBMS. To achieve this goal, you must identify and index frequently searched fields in your workflow. Creating an index does not guarantee its use. When a search is performed, the query optimizer does not automatically use an index. The optimizer chooses whether to use an index and which index to use. If an index is not used, the database server must scan every data page in the table. Table scans can take several minutes to process on large tables. If table scans occur frequently, application performance is likely to suffer. Therefore, you must create indexes that the DBMS will recognize and use consistently as the best choice for retrieving the requested information. In other words, you should index the most useful fields on a form, and make sure that the search criteria promotes the use of indexes.
Indexing Specific Fields

Consider using indexes for frequently searched fields, such as those that store names, employee numbers, and categories. Additional fields to consider indexing include the following fields:
!
Fields used in a search qualification For example, a user presses Return in the Last Name field, which starts a set fields action that searches for more information based on that field.
! !
Fields used in search menus Fields used in the qualifications of Set Field, Push Field, and Open Window active links Fields that are frequently searched (log files can help determine these fields) Fields that are selective Selectivity indicates how many entries are returned for a given search. Generally, a search that returns less than 5 percent of the total data in the form is considered to be selective.
Note: The indexing of a currency field has special considerations. Because a currency field is represented by multiple columns in the main data table, multiple columns will be indexed. Standard queries against a currency field will potentially use any of several different columns, depending on the currency type specified. In order to provide comprehensive coverage, indexing a currency field defines an index for the value column, the type column, and for each functional currency column. This can produce significant overhead for the main data table. Therefore, carefully consider indexing a currency field before doing so. For more information about currency fields, see the Developing AR System Applications: Basic guide
You can use SQL logging to help you determine which fields to index.
Limiting the Number of Indexes

You can create multiple indexes for a form; however, too many indexes can slow your system. Recommendation: Limit the number of indexes to five per form. See the Developing AR System Applications: Basic guide for more information about indexing.
Optimizing AR System Searches ! 35
Defining Effective Searches

The following sections describe techniques for defining search criteria that enhance performance.
Using Properly Qualified Searches

Qualify your searches to avoid table scans. Unqualified searches do not use indexes. If there is no search qualification, every request in the underlying table must be read. Avoid unqualified searches in your workflow. For example, an active link that performs a set fields action, based on the contents of a field, should include a qualification that the field must have a value: Run If: 'Short Description' != $NULL$ If Action: Set Fields. Else Action: Then send a message to the user. You can also require users to better qualify their searches by disallowing unqualified searches (see the following procedure) and limiting the number of requests that can be returned by a search. Disallowing Unqualified Searches
1 Select a server in AR System Administrator. 2 Choose File > Server Information. 3 Click the Configuration tab. 4 Ensure that Allow Unqualified Searches is not selected. 5 Click OK to save your changes.
Use a Run If Qualification

For an active link, escalation, or filter that performs a set fields action based on the contents of a field, include a Run If qualification that the specified field cannot be empty (!= NULL).
Avoiding NOT Operators

Indexes are not used for NOT operations (for example, 'Status' != "Closed"). Therefore, if NOT is the only operation used, the entire database table is searched. If NOT is used in a search, use additional criteria in the qualification to ensure that the index is used. See Table 1-2 for an example.
Placing Search Terms in Proper Order

For searches that reference indexed fields, isolate the indexed field name on one side of the operator. If the field name is not isolated, the search does a table scan instead of using the index. For example, suppose the field Create Date is indexed. In this case, the search string 'Create Date' < $TIMESTAMP$ - 60*60*8 performs an index search because the field name is isolated on one side of the less-than (<) operator. However, the search string $TIMESTAMP$ - 'Create Date' > 60*60*8 performs a table scan because the indexed field name is not isolated. See Table 1-2 for more examples. A search that begins with a wildcard, such as %, does not use an index because the index is ordered by the leading characters of the index key values. This means that the entire database must be searched for matching entries. In other words, search for e%ample, not %mple. See the following table for more examples. The following table summarizes these concepts. The examples assume that there is an index on the referenced field.
Table 1-2: Efficient and Inefficient Search Criteria Examples
Efficient Search (Index considered) 'Submitter' LIKE "Jackson%" 'Status' < "Closed" 'Create Date' < $TIMESTAMP$ 60*60*24
Inefficient Search (Index NOT considered) 'Submitter' LIKE "%Jackson%"
'Status' != "Closed" $TIMESTAMP$ 'Create Date' > 60*60*24
Checking Query-by-Example (QBE) Match

QBE searches configured for Anywhere do not limit users, but they put the most strain on the database. Use AR System Administrator to change the QBE Match setting for fields to Equal or Leading.
Optimizing AR System Searches ! 37
The default setting is Anywhere for both the Match preference in AR System Administrator and the QBE Match property of character fields. When a user searches AR System with the Anywhere setting, the search returns requests with field values matching any part of the criteria specified in a field. For example, an Anywhere search for requests that match the word turn returns all the requests containing any part of the word turn, such as right turn, left turn, turned, return, turned left, and user returned. These searches consume system resources. To improve performance, change the default QBE Match preference in AR System Administrator and the QBE Match setting of character fields to Equal or Leading. While more restrictive to your users, these searches are more efficient than Anywhere searches.
Using Full Text Search

You can use the Full Text Search (FTS) option to expedite searches on diary and long character fields. For additional information about full text search, refer to the Developing AR System Applications: Advanced guide.
Designing Search and SQL Menus

To speed system performance, design your menus to search indexed fields. Use equal or leading qualifications for your search menus, and design the menus to return the fewest possible values. When you use search menus, select Refresh On Connect whenever possible. The On Connect setting causes the menu to refresh only when the user opens the menu the first time after opening the form. One situation where this setting is not desirable is when you have a search menu that depends on a field value in a form. In this case, you must set the menu to Refresh On Open. This setting causes the menu to update every time the menu is used.
Warning: Frequent menu retrievals can impact performance. Only select a Refresh of On Open when absolutely necessary.
With SQL menus, which allow any valid SQL statement to be run, use an efficient SQL statement. For example, rather than using select * statements, specify the two columns to be used as the value and label, and use an indexed field to qualify the query if there are many records in the table.
Optimizing AR System Applications and Workflow

Performance is affected by workflow and application design, as well as hardware, memory, and other system components. In general, the following guidelines will help you enhance performance of applications:
! ! ! !
Eliminate unnecessary fields. Assign field permissions only as needed. Minimize the database length of character fields. Minimize the number of views.
After you have designed a form, you can analyze its performance. See Using AR System Analyzer on page 16 for more information. The following sections describe ways you can design efficient workflow and applications.
Designing Active Links

Active links execute locally, so they typically do not impact the server as much as filters can. In many cases, an active link may be a better choice than a filter. The exceptions are set fields actions and push fields actions, which should be performed in filters whenever possible. See Designing Set Fields and Push Fields Actions on page 41 for more information. You should also limit the number of active links for a given form, because too much background activity can slow the workflow. To improve active link performance, simplify the qualification for active links and combine active links that use the same qualification. This is more efficient than designing two active links that are identical other than their Execute On selection. For example, you might want your users to have the option of clicking a button or pressing Return to open a selection menu list. Design both of these Execute On actions in the same active link.
Optimizing AR System Applications and Workflow ! 39
Designing Filters
To improve filter performance, simplify the qualification and combine filters that use the same qualification. Use the filter log to identify when filters are run and the actions the filters take. Consider using workflow logging to test filters from a single user. This will show all activities triggered by the users transactions. Turn off these logs after analysis, because logs occupy disk space and can slow system performance. By analyzing these log files, you might discover that active links would be better suited for some tasks. The one situation where filters are often more appropriate than active links is for set fields and push fields actions. See Designing Set Fields and Push Fields Actions on page 41 for more information.
Run Process-Run Macro Operations

One filter process that consumes large amounts of server memory and CPU processing time is the Run Process-Run Macro operation. If these operations are a part of your primary workflow, take into consideration the resources required to run these processes. Also consider that these processes run in parallel on the server. If several users run these operations simultaneously, system performance slows. Use alternative methods such as Push Fields, custom API programs, or the Filter API.
Designing Escalations
The following guidelines will help you design efficient escalations:
! !
Use the minimum number of escalations required for your workflow. Run escalations with qualifications that make use of indexed fields whenever possible. Streamline your escalations by including all available criteria in the qualification. Unqualified escalations run against every record for the form, and may process some records unnecessarily. Stagger your escalations to run at different times. Escalations that run simultaneously can impact the performance of your AR System, especially during peak hours.

!
Avoid scheduling multiple escalations that are set to run simultaneously. For example, three escalationsone set to run at 15-minute intervals, one at 30-minute intervals, and one at 60-minute intervalswill run simultaneously every 60 minutes. Allow your escalations the time they need to complete before the next escalation activates. An example is an escalation that searches the database for 30,000 requests but is set to execute every minute.
Use the escalation log to identify the times escalations run, how long they take to complete, and the types of actions your escalations perform. Remember that an escalation can modify a request.
Limiting Blocking Actions

You can help maintain system performance by minimizing the impact of blocking operations. A blocking operation is defined as an action performed during filter processing that waits for a DBMS or an external process to return the requested information. Blocking operations are caused by set fields and push fields filter actions and $PROCESS$ actions that retrieve information from a DBMS or an external process. Blocking operations should be avoided whenever possible because they potentially impact all users, and blocking operations typically are not scalable. However, you may need to use blocking actions for some processes. The following two sections describe ways to minimize performance issues when you must use a set fields action or a $PROCESS$ action. For more information, see the Developing AR System Applications: Basic guide.
Designing Set Fields and Push Fields Actions

Consider using filters instead of active links to perform set fields and push fields actions, especially if the active link Execute On condition is Submit or Modify. The advantage is that the server (filter) should perform the set fields action faster than the client (active link). For example, an active link that performs a set fields action on submit pulls information from the server, only to push that information back. Your system performance will improve if you use a filter to perform the set fields action at the server.
Use only efficient searches in these actions, especially if workflow executes the search frequently. Efficient searches define where the system looks for the data (usually using an index). You can also improve performance by designing your actions to retrieve only the columns needed, although this has less impact on performance than a well-defined search.
Note: Set fields actions cannot be performed in a filter if the user must see the data retrieved by the set fields action prior to the submit or modify operation or if client-based workflow is dependent upon the data retrieved by the set fields action.
Set fields and push fields actions that include database searches or external processes are considered blocking actions, so limit their use for best performance.
Designing $PROCESS$ Actions

To improve $PROCESS$ action performance, have one $PROCESS$ action execute one resource-demanding command and return the results to a temporary field. The remaining actions can retrieve and parse data from the temporary field. For example, if you are setting five fields, write this data to a temporary field with the first $PROCESS$ operation, and have the remaining actions retrieve the data from the local field. An even better solution is to redesign the process to take advantage of the Filter API, as described in the Action Request System C API Reference Guide. This technique uses one long-running process, making it more efficient and significantly faster.
Designing Join Forms

Take note of the following issues when you create join forms:
!
If you create multiple layers of join forms, you might see a decline in the speed of database and system performance. Ensure that you only join fields that have been indexed. Joins on non indexed fields can slow system performance.
Designing Fields
Performance decreases when character field size exceeds 255 bytes (4000 bytes for the Oracle database). Also, maintain a minimum number of diary fields, because the greater the number of diary fields in a form, the greater the impact on performance for that form. Most AR System applications can be effectively designed with one or two diary fields.
Designing Form Views

If you are maintaining multiple form views with trim or control fields, do not duplicate screen objects unnecessarily. Whenever possible, share screen objects between views. The more screen objects you create (data fields, control fields, and trim), the larger you make your forms and the longer it takes AR System Windows User Tool to load, display, or switch to another view. Avoid using many toolbar buttons with different bitmaps in multiple views; this also increases the form size. If you need to include an image, try using a JPEG instead of a bitmap. The file size is generally smaller for JPEG files, and the form will take less time to load.
CHAPTER
Using Log Files
Most of the AR System server functions have trace modes. These modes enable you to log the operations that are performed so you can verify whether the programs are performing the operations you expect. This chapter provides troubleshooting information for the following trace modes and log files:
! ! ! ! ! ! ! ! ! !
Tracing Guide Activity on page 46 Server Error Log on page 46 Server Trace Modes on page 46 Client Trace Modes on page 62 Combined Server and Client Logging on page 65 AR Log Display Tool for Log Files on page 69 Full Text Search Log on page 71 Mid Tier Logging on page 72 Other Trace Modes on page 73 Additional Resources on page 74
Using Log Files ! 45
Tracing Guide Activity

You can create a log file of active link activity that will include all of the active links in a guide. This option logs information about active link activity for each operation, including which active links executed and whether active link execution was successful. Active Link logging is activated in AR System Windows User Tool. For information about activating active link logging, see AR System Windows User Tool help.
Server Error Log

The arerror.log file logs all AR System server error messages. Any message that represents a serious or fatal server error is reported in this file, including messages that cannot be returned to a user, or that are not appropriate for return. The logged messages provide information about failed server threads. For example, any time an AR System server shuts down, a message is written to this file. This is helpful for determining when and why an AR System server process (or any other server process) quits. On Windows, this file is stored in the <ar_install_dir>\Arserver\Db directory. On UNIX, this file is stored in the <ar_install_dir>/db directory. Check this file regularly to see if your servers are reporting errors.
Server Trace Modes

The AR System server process is configured with several trace modes that write to log files:
! ! ! ! !
Alert Arforkd API Distributed Server Escalation
! ! ! ! !
Filter Plug-in SQL Thread User
Because the server is the main process for the system, it must be running at all times. You can activate and deactivate the various trace modes at any time while the program is running.
46 "Chapter 2Using Log Files
Once the log is activated, logging starts immediately. By default, each log file is emptied and restarted when you restart the AR System server process or when you reactivate logging after it has been deactivated. When this occurs, the previous log file is renamed with a.bak extension (for example, <ar_install_dir>\Arserver\Db\arsql.log.bak), and a new file is created. After the initial log is created, there are always two versions of the log file (the current one and the previous one). Though the log files have default names and locations, you can change these names and locations through AR System Administrator. You can choose to append to an existing log file. If you choose this option, data will be appended to the same log file each time logging is turned on or off, which includes a system reboot. You can have different trace modes reference the same log file, providing a sequence of AR System events. This merging of traces is useful for tracing a transaction through several modes. It might also prove useful for debugging API programs. For example, an API trace and SQL trace that write to the same file provide a chronological list of the specific SQL events triggered by specific API calls. However, having multiple trace modes reference the same log file can have an adverse effect on performance, as the opportunity for contention among multiple threads is increased.
Note: Do not combine the arforkd log with other logs. The arforkd information would be randomly interspersed in the log, and, if the information is combined, the log would be confusing to read.
Use the Log Files tab of the Server Information dialog box in AR System Administrator to set trace mode options. The Log Files tab of the Server Information dialog box enables you to activate or deactivate the trace modes at any time. This is convenient when tracing an individual operation or group of operations. You can also choose the append option in this tab. For more information about setting log files, refer to the Configuring AR System guide.
Server Trace Modes ! 47
Trace Log Entry Format

Log entries, whose values vary depending on the trace mode, appear in the following format on one line:
<Log-Type > <TID: xxxxxx> <RPC ID: xxxxxxxxxx> <Queue: 10-char-long> <Client-RPC: 6-char-long> <USER: 30-char-long > /* <timestamp> */ [information] Note: Depending on trace mode you are using and the actions being executed, the log file may include some or all of the entries shown in the example.
The following table describes the possible values for the entries in different modes.
Table 2-1: Values for Trace Log Entries
Entry Log-Type TID RPC-ID Queue Client-RPC USER Timestamp [information]
Possible Value Displayed for all types of logs except thread logs; fixed-size, 4-byte string (examples: API, ESCL, FLTR, SQL, USER) The identification number of the thread executing the operation (fixed-size, 6-digit number) The dispatch ID that identifies the API call of each user in the server (fixed-size, 10-digit number) The type of queue that a particular thread lives on: admin, alert, fast, list, Flashboards, escalation, or private The RPC number requested by the client A 30-character string identifying the user name; blank if no user name (for example, a system call) The date and time the operation was performed; displayed to 1/10,000 of a second Entry detail; varies with the type of log
Note: The trace mode logs will sometimes show Client-RPC 999999 (the 6-digit dispatch ID that identifies the API call of a user in the server). This indicates that internal processes are running or are being caused to run by other processes. For example, this may occur when activities that cause a recache are being logged.
By default, log files are located in <ar_install_dir>\Arserver\Db on a Windows server, or <ar_install_dir>/db on a UNIX server. You can change the path or file name of log files in the Server Information dialog box in AR System Administrator.
Alert Log
The alert log traces all alert management and delivery activities. By default, the log file is named aralert.log. Each line of an alert log file begins with an <ALRT> prefix. Each alert transaction includes a time stamp and the user name. If the user name is AR_NOTIFIER, it indicates an internal operation is being performed. The entry details include information about the status of registration attempts, the success or failure of alert message-sending operations, and the occurrences of user deregistration. Following is a sample alert log file:
<ALRT> <TID: 000625> <RPC ID: 0000014400> <Queue: Admin > <Client-RPC: 390600 > <USER: Demo > /* Mon Apr 30 2001 09:42:22.8120 */ Alert Trace Log -ON <ALRT> <TID: 000052> <RPC ID: 0000000016> <Queue: Fast > <Client-RPC: 390620> <USER: Demo >/* Mon Apr 30 2001 09:43:31.9650 */ Received registration request for user Demo at address 127.0.0.1 on port 1675 <ALRT> <TID: 000052> <RPC ID: 0000000016> <Queue: Fast > <Client-RPC: 390620> USER: Demo >/* Mon Apr 30 2001 09:43:31.9810 */ Queued acknowledgement request for address 127.0.0.1 on port 1675 <ALRT> <TID: 000052> <RPC ID: 0000000016> <Queue: Fast > <Client-RPC: 390620> <USER: Demo >/* Mon Apr 30 2001 09:43:31.9810 */ Preliminary registration for user Demo at address 127.0.0.1 on port 1675 completed <ALRT> <TID: 000069> <RPC ID: 0000000000> <Queue: Alert > <Client-RPC: 390601> <USER: AR_NOTIFIER >/* Mon Apr 30 2001 09:43:32.0430 */Socket read call failed (0) <ALRT> <TID: 000069> <RPC ID: 0000000000> <Queue: Alert > <Client-RPC: 390601> <USER: AR_NOTIFIER >/* Mon Apr 30 2001 09:43:32.0430 */ Failure to receive acknowledgement from address 127.0.0.1 on port 1675
<ALRT> <TID: 000069> <RPC ID: 0000000000> <Queue: Alert > <Client-RPC: 390601> <USER: AR_NOTIFIER /* Mon Apr 30 2001 09:43:32.0430 */Registration state for user Demo at address 127.0.0.1 on port 1675 updated to DOWN <ALRT> <TID: 000052> <RPC ID: 0000000000> <Queue: Fast > <Client-RPC: 390620> <USER: AR_NOTIFIER /* Mon Apr 30 2001 09:43:32.0740 */ Cleanup of registration entry occurred for user Demo at address 127.0.0.1 on port 1675
arforkd Log (UNIX only)

On UNIX systems, the arforkd log traces all arforkd activity. By default, the log file is named arforkd.log. The arforkd process does not perform circular logging. If a log for the process already exists, the process can append its output to the file (if the Log-File-Append flag is set to T in the ar.conf file). For more information about the ar.conf file, refer to the Configuring AR System guide. When there is no arforkd log file but arforkd logging is turned on, arforkd logging is printed to the screen. Following is a sample arforkd log file:
+++ Begin log Wed Jun 6 11:19:27 2001 Wed Jun 6 11:19:27 2001 Arforkd: Opening log file in overwrite mode. Wed Jun 6 11:19:27 2001 Awaiting record marker. Wed Jun 6 11:24:15 2001 Received record marker. Wed Jun 6 11:24:15 2001 Spawning child, shell "", command "mkdir arTmp". Wed Jun 6 11:24:15 2001 Spawned "mkdir arTmp" as pid 22795. Wed Jun 6 11:24:15 2001 Awaiting record marker. Wed Jun 6 11:24:15 2001 process id 22795 has exited with status 0x0. Wed Jun 6 11:25:07 2001 Received record marker. --- Close log Wed Jun 6 11:25:07 2001
For information about the arforkd process, refer to the Configuring AR Stem guide.
API Log
The API log traces all API calls run on the server, regardless of their origin. All AR System operations are logged, including the activity of AR System Administrator and applications that you have developed. By default, the log file is named arapi.log. Each line of an API log file begins with an <API > prefix. Each API transaction includes a time stamp and the user name. The entry details include information about the API calls encountered and some information about parameters passed. Following is a sample API log file:
<API > API Trace Log -- ON <API > <TID: 000086> <RPC ID: 0000000627> <Queue: Admin > <Client-RPC: 390600> <USER: Jane Administrator > /* Tue Nov 02 1999 08:54:06.1640 */ -SSI OK <API > <TID: 000086> <RPC ID: 0000000628> <Queue: Admin > <Client-RPC: 390600> <USER: Jane Administrator > /* Tue Nov 02 1999 08:54:22.0700 */ +SSI ARSetServerInfo -- as user Jane Administrator <API > <TID: 000086> <RPC ID: 0000000628> <Queue: Admin > <Client-RPC: 390600> <USER: Jane Administrator > /* Tue Nov 02 1999 08:54:22.0700 */ -SSI OK <API > <TID: 000086> <RPC ID: 0000000629> <Queue: Admin > <Client-RPC: 390600> <USER: Jane Administrator > /* Tue Nov 02 1999 08:54:25.4610 */ +GSI ARGetServerInfo -- as user Jane Administrator
Distributed Server Log

The distributed server log tracks the steps and events of distributed operations processed by the arservdsd process on UNIX and the serverds.exe process on Windows. Logged information includes the operation transfer to the other system, mapping definitions, and status. The Distributed Server Option is available separately from AR System. By default, the log files are named ardist.log and ardist.log.default and are located in <ar_install_dir>\Arserver\Db on a Windows server, or <ar_install_dir>/db on a UNIX server. If you have configured DSO pools you will also have a log for each pool. By default, these log files are named ardist.log.<pool_name>. You can change the path or file name in the Server Information dialog box.
Each line of a distributed server log file begins with a <DIST> prefix and includes the time of each transaction. Following the time stamp is a description of the processed operation. The distributed server log file does not follow the format of other log files. There is no thread ID or RPC listing. For more information about distributed servers, see the AR System Distributed Server Option Administrators Guide. In the following example, a filter is performing the distributed operation of moving data from request WWW000000002576 of the www-TechSupport form on server1 to request 000000000080896 of the ops-Support Call form on server2. The mapping file used is wwwToSupportCall.
<DIST> Distributed Server Trace Log -- ON (Fri Nov 8 1996 16:43:50.5454) <DIST> Initializing access to the AR System server <DIST> Locating mapping and pending schemas <DIST> Mapping schema -- Distributed Mapping <DIST> Pending schema -- Distributed Pending <DIST> Get a list of items to process (Fri Nov 8 1996 16:43:50.7681) <DIST> 0 new item(s) found <DIST> Sleeping for 18:09 minutes <DIST> Get a list of items to process (Tue Nov 02 1999 06:15:07.2260) <DIST> 1 new item(s) found <DIST> Processing item number 0 (Tue Nov 02 1999 06:15:07.4588) <DIST> Pending type -- 1 <DIST> Source Schema -- www-TechSupport <DIST> Source ID -- WWW000000002576 <DIST> Pending Other -<DIST> -m wwwToSupportCall -x <server2> <DIST> Get source structure <DIST> Using NEW cache definition for www-TechSupport (<server1>) <DIST> Have source structure details, get entry <DIST> Have entry details, get mapping <DIST> Filter-specified mapping -- wwwToSupportCall <DIST> Filter-specified to server -- <server2> <DIST> Mapping name -- wwwToSupportCall <DIST> Target schema -- ops-Support Call
<DIST> Target server -- <server2> <DIST> Get target structure <DIST> Using NEW cache definition for ops-Support Call (<server2>) <DIST> Perform mapping <DIST> Transfer successfully completed. New/overwritten entry 000000000080896 <DIST> Mapping completed <DIST> --> To stage: 8 (of 8) Return Code: 0 DONE <DIST> Delete pending item -- 000000000002812 <DIST> Sleeping for 46:45 minutes
Escalation Log
The escalation log traces escalation activity on the server. It identifies the time an escalation starts, whether it passes the qualification, any actions executed, and the time the escalation stops. By default, the log file is named arescl.log. Each line of an escalation log file begins with an <ESCL> prefix and includes the time of each escalation transaction. Escalation logs might have lines that indicate the escalation is going to run in some amount of time. This means the escalation was checked and was found not to require activation at that time. Following is a sample escalation log file:
<ESCL> <TID: 001280> <RPC ID: 0000390685> <Queue: Escalation> <Client-RPC: 999999> <USER: AR_ESCALATOR > set new firetime of 242 seconds <ESCL> <TID: 001280> <RPC ID: 0000000001> <Queue: Escalation> <Client-RPC: 999999> <USER: AR_ESCALATOR > /* Fri Feb 18 2000 10:39:51.3930 */ Start escalation processing -- Operation <ESCL> <TID: 001280> <RPC ID: 0000000001> <Queue: Escalation> <Client-RPC: 999999> <USER: AR_ESCALATOR > Checking AR System Sample Form Escalation (enabled) : ready to fire now <ESCL> <TID: 001280> <RPC ID: 0000000001> <Queue: Escalation> <Client-RPC: 999999> <USER: AR_ESCALATOR > --> Passed -- perform actions <ESCL> <TID: 001280> <RPC ID: 0000000001> <Queue: Escalation> <Client-RPC: 999999> <USER: AR_ESCALATOR > 000000000000001 <ESCL> <TID: 001280> <RPC ID: 0000000001> <Queue: Escalation> <Client-RPC: 999999> <USER: AR_ESCALATOR > 0: Set Fields
<ESCL> <TID: 001280> <RPC ID: 0000000001> <Queue: Escalation> <Client-RPC: 999999> <USER: AR_ESCALATOR > Character Field (536870913) = Changed by Escalation <ESCL> <TID: 001280> <RPC ID: 0000000001> <Queue: Escalation> <Client-RPC: 999999> <USER: AR_ESCALATOR > /* Fri Feb 18 2000 10:39:51.4250 */ Stop escalation processing <ESCL> <TID: 001280> <RPC ID: 0000000001> <Queue: Escalation> <Client-RPC: 999999> <USER: AR_ESCALATOR > set new firetime of 300 seconds
In the example, the AR System Sample Form escalation activated and performed a set fields action. At the end of the escalation process, the escalation alarm was reset to 300 seconds because the next escalation to be run is set to run in 300 seconds. In 300 seconds, that escalation will be run.
Filter Log
The filter log file shows all of the filters that were checked during an operation. If a filter was executed, the log also provides a list of the operations performed. By default, the log file is named arfilter.log. This log is especially useful for checking the interaction of your filters and for verifying whether a filter is performing correctly. The filter log format is similar to the SQL log format, but the information in the filter log is displayed over multiple lines. The log has the following characteristics:
!
Each line starts with a tag that identifies it as being a filter logging line (<FLTR>). A time stamp line is written for every operation that can trigger filters. A line indicates that processing has started and the operation is being performed. As each filter is checked, a message is written that includes the name of the filter, its escalation order, an indication of whether the filter passed the qualification, and any actions taken with the results of those actions. The last line contains a closing time stamp.
! !
Phases 1, 2 and 3 of filter processing appear in the filter log. As the AR System checks filters, all notify and run process actions are deferred to phase 3. When phases 1 and 2 are completed, and the database has been updated, phase 3 begins. In phase 3, all deferred filter actions are run. This information enables you to track which filters are being checked, which filters are running, and the actions that are being performed. For more information about filter processing, refer to the Developing AR System Applications: Basic guide. Following is a sample filter log file:
<FLTR> <TID: 000625> <RPC ID: 0000014210> <Queue: Admin> <Client-RPC: 390600 > <USER: Demo> /* Sun Jun 09 2002 08:38:59.4530 */ Filter Trace Log -- ON <FLTR> <TID: 000625> <RPC ID: 0000014330> <Queue: Admin> <Client-RPC: 390620 > <USER: Demo> /* Sun Jun 09 2002 08:44:12.7030 */ Start filter processing -- Operation - CREATE <FLTR> <TID: 000625> <RPC ID: 0000014330> <Queue: Admin> <Client-RPC: 390620 > <USER: Demo> form1 - <NULL> <FLTR> <TID: 000625> <RPC ID: 0000014330> <Queue: Admin> <Client-RPC: 390620 > <USER: Demo> Checking AR System Sample Filter (500) <FLTR> <TID: 000625> <RPC ID: 0000014330> <Queue: Admin> <Client-RPC: 390620 > <USER: Demo> --> Passed -- perform actions <FLTR> <TID: 000625> <RPC ID: 0000014330> <Queue: Admin> <Client-RPC: 390620 > <USER: Demo> 0: Message <FLTR> <TID: 000625> <RPC ID: 0000014330> <Queue: Admin> <Client-RPC: 390620 > <USER: Demo> This is a sample message <FLTR> <TID: 000625> <RPC ID: 0000014330> <Queue: Admin> <Client-RPC: 390620 > <USER: Demo> 1: Notify <FLTR> <TID: 000625> <RPC ID: 0000014330> <Queue: Admin> <Client-RPC: 390620 > <USER: Demo> <deferred to phase 3> <FLTR> <TID: 000625> <RPC ID: 0000014330> <Queue: Admin> <Client-RPC: 390620 > <USER: Demo> /* Sun Jun 09 2002 08:44:12.7180 */ End of filter processing (phase 1) <FLTR> <TID: 000625> <RPC ID: 0000014330> <Queue: Admin> <Client-RPC: 390620 > <USER: Demo> /* Sun Jun 09 2002 08:44:13.0620 */ Restart of filter processing (phase 3) <FLTR> <TID: 000625> <RPC ID: 0000014330> <Queue: Admin> <Client-RPC: 390620 > <USER: Demo> 1: Notify <FLTR> <TID: 000625> <RPC ID: 0000014330> <Queue: Admin> <Client-RPC: 390620 > <USER: Demo> Priority: 0 Mechanism: Alert To: User <FLTR> <TID: 000625> <RPC ID: 0000014330> <Queue: Admin> <Client-RPC: 390620 > <USER: Demo> This is a sample notification
<FLTR> <TID: 000625> <RPC ID: 0000014330> <Queue: Admin> <Client-RPC: 390620 > <USER: Demo> /* Sun Jun 09 2002 08:44:13.2030 */ Start filter processing -- Operation - CREATE <FLTR> <TID: 000625> <RPC ID: 0000014330> <Queue: Admin> <Client-RPC: 390620 > <USER: Demo> Alert Events - <NULL> <FLTR> <TID: 000625> <RPC ID: 0000014330> <Queue: Admin> <Client-RPC: 390620 > <USER: Demo> /* Sun Jun 09 2002 08:44:13.2030 */ End of filter processing (phase 1) <FLTR> <TID: 000625> <RPC ID: 0000014330> <Queue: Admin> <Client-RPC: 390620 > <USER: Demo> /* Sun Jun 09 2002 08:44:13.3750 */ Restart of filter processing (phase 3) <FLTR> <TID: 000625> <RPC ID: 0000014330> <Queue: Admin> <Client-RPC: 390620 > <USER: Demo> /* Sun Jun 09 2002 08:44:13.3750 */ Stop filter processing <FLTR> <TID: 000625> <RPC ID: 0000014330> <Queue: Admin> <Client-RPC: 390620 > <USER: Demo> /* Sun Jun 09 2002 08:44:13.3750 */ Stop filter processing
The filter in the example log performed two actions, a message and an alert. The alert was deferred to phase 3 filter actions. The log also shows the execution of another filter, Alert Events, which created the alert performed by the original filter. The execution of another filter will also occur on a push fields action. When the log indicates a failed qualification, this does not mean that the filter malfunctioned. This means that the qualification was not met for this operation, so the else action runs if present.
Plug-In Server Log

The plug-in server log tracks the events of plug-ins that the AR System uses. For more information about using plug-ins, refer to the AR System C API Reference Guide. By default, the log file is named arplugin.log. Each line of a plug-in log file begins with an <PLGN> prefix. Each entry includes information such as the time stamp, the API calls encountered, and name of the plug-in. The following example shows a portion of a plug-in log:
<PLGN> <TID: 002944> <RPC ID: 0000000000> <Queue: Dispatcher> <Client-RPC: 000000> /* Fri Apr 13 2001 15:56:41.6300 */ Plug-in Trace Log -- ON <PLGN> <TID: 002944> <RPC ID: 0000000000> <Queue: Dispatcher> <Client-RPC: 000000> /* Fri Apr 13 2001 15:56:41.6460 */ AREA Plug-in Loaded: EXAMPLE.AREA.SIMPLE version 1
<PLGN> <TID: 002944> <RPC ID: 0000000000> <Queue: Dispatcher> <Client-RPC: 000000> /* Fri Apr 13 2001 15:56:41.6460 */ ARPluginInitialization defined <PLGN> <TID: 002944> <RPC ID: 0000000000> <Queue: Dispatcher> <Client-RPC: 000000> /* Fri Apr 13 2001 15:56:41.6460 */ ARPluginTermination defined <PLGN> <TID: 002944> <RPC ID: 0000000000> <Queue: Dispatcher> <Client-RPC: 000000> /* Fri Apr 13 2001 15:56:41.6460 */ ARPluginCreateInstance defined <PLGN> <TID: 002944> <RPC ID: 0000000000> <Queue: Dispatcher> <Client-RPC: 000000> /* Fri Apr 13 2001 15:56:41.6460 */ ARPluginDeleteInstance defined <PLGN> <TID: 002944> <RPC ID: 0000000000> <Queue: Dispatcher> <Client-RPC: 000000> /* Fri Apr 13 2001 15:56:41.6460 */ AREAVerifyLoginCallback defined <PLGN> <TID: 002944> <RPC ID: 0000000000> <Queue: Dispatcher> <Client-RPC: 000000> /* Fri Apr 13 2001 15:56:41.6460 */ AREANeedToSyncCallback defined <PLGN> <TID: 002944> <RPC ID: 0000000000> <Queue: Dispatcher> <Client-RPC: 000000> /* Fri Apr 13 2001 15:56:41.6460 */ AREAFreeCallback defined <PLGN> <TID: 002944> <RPC ID: 0000000000> <Queue: Dispatcher> <Client-RPC: 000000> /* Fri Apr 13 2001 15:56:41.7240 */ ARDBC Plug-in Loaded: RMDY.ARDBC.XML version 1 . . . <PLGN> <TID: 002092> <RPC ID: 0000000001> <Queue: ARDBC > <Client-RPC: 390695> /* Fri Apr 13 2001 15:57:17.0670 */ -GE OK <PLGN> <TID: 002092> <RPC ID: 0000000002> <Queue: ARDBC > <Client-RPC: 390695> /* Fri Apr 13 2001 15:57:23.3010 */ +GE ARDBCGetEntry -- vendor RMDY.ARDBC.XML table SomeTable.xml entry 000000000000001
SQL Log
Note: This mode of debugging can generate a large amount of information quickly. It is not unusual to have several hundred megabytes of information logged in one day.
The SQL log is similar to the filter log, but the SQL log file consists of a series of SQL commands (one per line), each of which contain the information listed in Table 2-1 on page 48. In this case, the log type is (<SQL >), which identifies the entry as an SQL logging line. By default, the log file is named arsql.log.
If you are licensed for full text search, the SQL debug mode also logs FTS searches in this file. Following is a sample SQL log file:
<SQL > <TID: 000213> <RPC ID: 0000000140> <Queue: Fast > <Client-RPC: 390620> <USER: Joe User > /* Tue Nov 02 1999 08:24:49.8200 */ SELECT * FROM H7 WHERE entryId = '000000000000001' <SQL > <TID: 000213> <RPC ID: 0000000140> <Queue: Fast > <Client-RPC: 390620> <USER: Joe User > /* Tue Nov 02 1999 08:24:49.8360 */ COMMIT WORK <SQL > <TID: 000244> <RPC ID: 0000000143> <Queue: Prv:390685> <Client-RPC: 390685> <USER: Chris Cadre> /* Tue Nov 02 1999 08:24:59.7730 */ SELECT password, notifyMech,licType,licTypeFText, licTypeReserv1,email,shortGroup,longGroup,userName FROM user_cache WHERE userName = 'Chris Cadre' <SQL > <TID: 000244> <RPC ID: 0000000143> <Queue: Prv:390685> <Client-RPC: 390685> <USER: Chris Cadre> /* Tue Nov 02 1999 08:24:59.7730 */ COMMIT WORK
This example shows two types of users: a normal user in a normal fast queue, and a private user in a private queue (390685 in our example). If an SQL error or warning is encountered as a result of the command, the command is followed by another line with the same header, but the SQL command is replaced with *** ERROR *** or *** WARNING ***. The text of the error or warning from the database follows. This information associates an operation with the user who performed the operation, and is useful for identifying database problems.
Thread Log
The thread log traces the history of when threads start, as well as how many times a thread has failed and been restarted by Thread Manager. By default, the log file is named arthread.log. Each transaction begins with the <THRD> prefix, followed by a time stamp. Following the time stamp is detailed information about the started thread. The format for the thread log file is as follows:
<THRD>/*ddd mmm dd yyyy hh:mm:ss:nnnn */ Thread ID <tid> (thread number <tno>) on <queueType><TStatus>
The following table explains the possible values for the entries in a thread log file.
Table 2-2: Values for Thread Log Entries
Entry tid tno
Possible Value for Entries in Thread Log FIle Identifier of the server running thread (fixed-size, 6-digit number) A sequential number assigned to the thread. This number helps recognize the thread from Windows Performance Monitor The queue this thread lives on (for example, admin, fast, list, escalation, or private) A value of started, died, or restarted
queueType TStatus
The following are sample thread log entries:

<THRD> /* Thu Jan 13 2000 18:29:46.7320 */ Thread Id 153(thread number 1) on Admin queue started. <THRD> /* Thu Jan 13 2000 18:33:01.0032 */ Thread Id 153(thread number 1) on Admin queue died. <THRD> /* Thu Jan 13 2000 18:59:46.7320 */ Thread Id 253(thread number 1) on Admin queue restarted.
User Log
The user log traces all user activity on the server. It tracks user logins and logouts. It also tracks users who are blocked by lack of free floating tokens and failed login attempts. Tracking failed login attempts helps to detect any unauthorized access to AR System. By default, the log file is named aruser.log. This log enables you to see who is connecting to AR System and when they connect. It indicates floating license use (and any rejected connections due to all floating licenses being in use) to help you plan for your floating license needs.
The file contains a line of information for every connection or disconnection from the AR System server. This enables you to track users accessing the system. A typical line in the file might look like the following:
<USER> <TID: 001084> <RPC ID: 0000003673> <Queue: Admin > <Client-RPC: 390600> <USER: Chris Cadre> /* Wed Mar 15 2000 08:22:59.4650 */ <USER> FIXED GRANT WRITE Chris Cadre
The line starts with a tag that identifies it as a user log entry (<USER>), followed by the thread ID, RPC ID, queue, and Client RPC information. Next, the user is identified, followed by the time stamp. The entry detail starts with the type of license used (READ, FIXED, or FLOAT). After the license type is the operation that was performed. See Table 2-3 on page 61 for each operation and corresponding definition. If the type of license is FLOAT, additional information reports the number of licenses in use at that time and the number that are available. Also included is the name of the user performing the operation. In the following example, Chris Cadre was granted a floating write license and a floating Full Text Search license. Of the five floating write and five floating Full Text Search licenses, none is left.
<USER> <TID: 001084> <RPC ID: 0000003673> <Queue: Admin > <Client-RPC: 390600> <USER: Chris Cadre> /* Wed Mar 15 2000 08:22:59.4650 */ FLOAT GRANT WRITE Chris Cadre (5 of 5 write) <USER> <TID: 001084> <RPC ID: 0000003673> <Queue: Admin > <Client-RPC: 390600> <USER: Chris Cadre> /* Wed Mar 15 2000 08:22:59.4650 */ FLOAT GRANT FULL Chris Cadre (5 of 5 full)
Following is a sample user log file:

<USER> <TID: 001084> <RPC ID: 0000003673> <Queue: Admin > <Client-RPC: 390600> <USER: Alex Administrator> /* Wed Mar 15 2000 08:22:59.4650 */ FIXED GRANT WRITE Alex Administrator <USER> <TID: 001084> <RPC ID: 0000003673> <Queue: Admin > <Client-RPC: 390600> <USER: Alex Administrator> /* Wed Mar 15 2000 08:22:59.4650 */ FIXED GRANT FULL Alex Administrator
<USER> <TID: 001072> <RPC ID: 0000003681> <Queue: Fast > <Client-RPC: 390620> <USER: Chris Cadre> /* Wed Mar 15 2000 08:23:11.5580 */ <USER> READ BAD PASSWORD Chris Cadre <USER> <TID: 001084> <RPC ID: 0000003698> <Queue: Admin > <Client-RPC: 390600> <USER: Chris Cadre > /* Wed Mar 15 2000 08:23:59.0900 */ FIXED GRANT WRITE Chris Cadre
Table 2-3: License Operations
Operation
Definition A connection was established for a user with a write license. A connection was released by exiting the client. A connection was released by a time-out.
GRANT WRITE RELEASE FLUSH
ADMIN RELEASE A connection was released explicitly by an administrator. BAD PASSWORD An attempt was made to log in as this user, but the password
was invalid.
NO WRITE FREE
A user with a floating write license attempted to connect. However, no floating license tokens were available. A connection was established for read access only. to a write connection as a floating write license token became available.
WRITE UPGRADE A connection for a user was upgraded from a read connection
EXPIRED WRITE
A connection for a write license was released by a time-out. Some other connection is maintained, because FLUSH was not specified. A connection was established for a user with a Full Text Search license. A user with a floating full text license attempted to connect. However, no floating license tokens were available. A connection was established for a pending upgrade onlyno Full Text Search capabilities were granted. A connection for a user was upgraded from a pending FTS connection to an FTS-enabled connection, because a floating FTS license token became available. A connection for a Full Text Search license was released by a time-out. Some other connection is maintained, because FLUSH was not specified.
GRANT FULL NO FULL FREE
FULL UPGRADE
EXPIRED FULL
Client Trace Modes

The following sections describe how to activate workflow logging. The sections also discuss what is generated in the combination workflow log during client workflow execution (that is, active links and macros). All information is logged to the file specified in the log file path.
Activating Workflow Logging in AR System Windows User Tool

In the Logging tab of the Options dialog box (Figure 2-1), you can set preferences that enable you to log AR System Windows User Tool activity. Preferences must be set for each user to enable logging to be set on or off for that user.
Figure 2-1: Logging Tab in Options Dialog Box (User Tool)
Activating Workflow Logging in AR System Windows User Tool

1 In AR System Windows User Tool, select Tools > Options.
The Options dialog box appears.

2 Click the Logging tab. 3 Select the types of logging you want to perform:
! !
In the Client region, you can select Active Links and Macro. In the Server region, you can select API, Database, and Filter.
4 Enter a file name in the Log File Path field.
Click the browse button to select a different directory path.

5 Click OK.
Activating Workflow Logging in a Web Browser

To turn active link logging on and off in the browser client, you must use centralized preferences. Activating Active Link Logging for a Web Client
1 Search for the users entry in the AR System Windows User Tool Preferences
form in AR System Windows User Tool or on the web client. For information about the AR System Windows User Tool Preferences form, refer to the Configuring AR System guide.
2 Click the Logging tab. 3 In the Workflow section, select Yes next to Workflow Log Enable. 4 In the Client section, select Yes next to Active Links. 5 Click Save.
The browser client active link log appears in a separate browser window.
Log Information for Active Links

The following information is generated when an active link is executed:
! ! ! ! ! ! ! !
Name Start, end, and elapsed time (to 1/10,000 of a second) Operation that fires the active link Execution order Form applied to Linked to which field (if applicable) Mode (for example, Query, Modify, Submit) Users name
Client Trace Modes ! 63

! !
Qualification and if it passes or fails Actions generated:

! ! !
Action number Type (for example, Set Fields, Set Characteristics, Push Fields) Name, new characteristics, and value of affected fields, if applicable
The format of the active link log is essentially the same in AR System Windows User Tool and on web clients. The following example shows the workflow of a simple active link named TEST (tag name: SETFIELD) that fires when a user presses the Enter key in the Assigned To field, and if the Status field equals Rejected. The action is to set the new value of field Short Description to This is a test.
<ACTL> /* Mon Mar 08 1999 12:56:46.1234 Demo */ <ACTL> Start active link processing -- Operation - On Return <ACTL> For Schema - ABCD <ACTL> Linked to field - Assigned To (4) <ACTL> On screen type - QUERY <ACTL> <TAG_NAME: SETFIELDFORPURCHASE PUSHFIELDFORPURCHASE> Checking TEST (0) <ACTL> <Qualification> Status = Rejected <ACTL> -> Passed -- perform actions <ACTL> 0: Set Fields <ACTL> Short Description (8) = This is a test <ACTL> /* Mon Mar 08 1999 12:56:46.9000 Demo */ <ACTL> Stop active link processing - On Return <ACTL> /* Mon Mar 08 1999 13:57:41.1200 Demo */ <ACTL> Active Link Trace Log OFF
Log Information for Macros

The following information is generated when a macro is run:
! ! ! !
Name Start and end time (to 1/10,000 of a second) Macro parameters (if supplied) and their values Form/Server that the macro runs on

! !
Operation and the value of affected fields Users name
Note: The web client does not support macros, so macro logging is not available on web clients.
The following example shows the workflow of a simple macro named test_macro that contains two parameters: Sub and Desc. The action is to submit a new entry with the two required fields, Submitter and Short Description, set to the values of the parameters Sub and Desc, respectively.
<MACR> /* Wed Jun 26 2002 09:37:25 */ <MACR> Import macro from file -- D:\Program Files\AR System\Home\arcmds\testxmac.arq <MACR> Parameters: <MACR> Desc=Another Ticket <MACR> Sub=Demo <MACR> /* Wed Jun 26 2002 09:37:37 */ <MACR> Import macro -- test_macro <MACR> /* Wed Jun 26 2002 09:37:37 */ <MACR> Starting processing of macro <MACR> Step: 0 <MACR> Change current working schema -- Form01 (presto) <MACR> Step: 1 <MACR> Submit a new entry -- Form01 (presto) <MACR> Short Description (8) = Another Ticket <MACR> Status (7) = New <MACR> Submitter (2) = Demo <MACR> /* Wed Jun 26 2002 09:37:37 */ <MACR> Completed processing of macro
Combined Server and Client Logging

For debugging purposes, you can set up AR System so that the server-side logging information is incorporated into the AR System Windows User Tool workflow log, creating one centralized log file. This log tracks the activity of a single user through the users operations. It allows debugging of that users interaction quickly and efficiently. The default location for the file is <client_install_dir>/workflow.log.
Combined Server and Client Logging ! 65
This feature is activated in AR System Windows User Tool. When combined server and client logging is activated, all API, Filter, and SQL log entries made by the server are logged in a single file. The log file that results is a combined server and client log for the current client user, in which the client activity (such as active links and macros) and the server activity (filters) are listed in the order of their execution. Time zone differences between the server and the client are not corrected. Therefore, time stamps recorded in the combined log may not be accurate from the clients time perspective. The log entries, however, are recorded in the proper sequence.
Note: Client-side logging is configured and performed independently of server-side logging. Either can be active or inactive independently of each other. If both are active, logging occurs in both locations.
Creating the Client Logging Group

To use the combined server and client logging, you must first set up a special group for users who are allowed to generate combined user-server logging. When you create users, you must make them members of this group so that they can turn on server-side logging in AR System Windows User Tool. The default group that can turn on server-side logging is the Administrator group. You can choose a different group from the Client-Side Logging Group under the Log Files tab in the Server Information dialog box in AR System Administrator. Once you have created this group, users who belong to this group can turn on server-side logging in AR System Windows User Tool. For more information about the Log Files tab, refer to the Configuring AR System guide. You can specify what type of information is logged and choose an alternative path and name for this file by using the Logging tab of the Options dialog box in AR System Windows User Tool. Access the dialog box through the AR System Windows User Tool Tools menu. Select any combination of the five log sources (API, Database, Filter, Active Links, and Macro) for capture. You can also specify the path and name of the log file. See Activating Workflow Logging in AR System Windows User Tool on page 62.
It is assumed that logging information will be obtained in the context of one server at a time. However, users may be in environments where they log in to many AR System servers simultaneously. Therefore, for security reasons, users must be members of the group that you specify for each server that they are currently logged in to in order to obtain any server-side logging information. If users are not members of the group for all servers in to which they are currently logged, they cannot turn on server-side logging. The settings for logging in AR System Administrator determine which trace mode is available for viewing on the client-side.
Note: All servers being logged into must be running AR System 4.5 or later. If one or more servers is running an earlier version, then server-side logging is disabled in AR System Windows User Tool.
Enabling Logging for the Logging Group To enable logging for the Logging group or any group, perform the following steps in AR System Administrator:
1 Choose File > Server Information. 2 Click the Log Files tab. 3 From the Client-Side Logging Group menu, choose the group.
Combined Server and Client Logging ! 67
Choosing the group here makes the logging options available in AR System Windows User Tool for users who belong to the selected group. If the group is not specified, the logging options in AR System Windows User Tool are disabled (grayed out).
Figure 2-2: Log Files Tab in Server Information Dialog Box (Administrator Tool)
4 Click OK. 5 To enable logging for the groups you specified, instruct all members of the
Logging group to log out of AR System Windows User Tool and log in again for these changes to take effect.
AR Log Display Tool for Log Files

The combined client and server workflow information that is written to a log file may be viewed using the AR Log Display tool. The AR Log Display tool is available only on Windows. It is installed with AR System Windows User Tool and resides in the <ar_install_dir> folder. This utility (arlogdisplay.exe) provides the following functionality:
!
Displays the logging events generated in AR System Windows User Tool in real time. Opens any log file for basic editing. Provides basic search and replace capabilities. Allows users to save contents to an RTF file.
! ! !
Note: This tool never modifies the actual log file created by AR System Windows User Tool. It only retrieves the information from a log file. When the user saves this information after having modified it, a .txt file is created.
Displaying Real-Time Log Information

You can view real-time log information, which can be useful in learning what workflow is executed as you are running your application.
Displaying Logging Events in Real Time

In the AR Log Display tool, choose File > Real Time Logging or click the Real Time Logging toolbar button to display logging events as they are being generated by AR System Windows User Tool. You can save the log information as an RTF file on your hard drive. You can annotate or edit this document later.
AR Log Display Tool for Log Files ! 69
AR Log Display searches the registry entry for the last user logged in to AR System Windows User Tool. It then retrieves and displays the log information from that users log file. However, in cases where the preference for Auto Login is set to be False in AR System Windows User Tool, or the server imposes a forced login, this entry would be set to 1. In this case, there are two ways to specify which log file to read for displaying log information in real time:
!
Let the Display Tool choose the log file that resides in the default home directory of the Default User (the Default User is created at the time AR System Windows User Tool is installed). On the command line, if you specify a user name, AR Log Display will read from the log file created for the specified user. For example:
arlogdisplay <user_name> The following figure shows the AR System Log Display tool.
Figure 2-3: AR System Log Display Tool
Viewing Log Files

In addition to seeing the log as it is being generated in real-time logging, you can access the log after it is generated to display, search, or edit the log file content. In the AR System Log Display tool, select File > Open, or click the File Open button on the toolbar to display any log file. The file will display the same as a real-time log, as shown in Figure 2-3. With the file open, you can page through the log file content, do simple searches for strings, and annotate and format the log information with tabs. If the file is modified, it is saved to an RTF file, which can be opened later for further modifications.
Full Text Search Log

The full text search (FTS) log tracks all operations performed by the arfts.exe process on a Windows server or arservftd on a UNIX server. This includes all actions on the FTS index, such as the creation, modification, and deletion of requests. This file is useful for ensuring that your FTS index is being built and maintained as you expect. FTS logging must be turned on in the ar.cfg file on a Windows server or ar.conf on a UNIX server. To enable AR System FTS logging and Verity internal logging, enter the following line into the file:
FTS-Debug-mode: 15 Note: The Verity internal log is not governed by the log management options specified for AR System logging. The log can become quite large and is therefore generally not intended for troubleshooting.
To enable solely AR System FTS logging, enter the following line into the file:
FTS-Debug-mode: 243
By default, the AR System FTS log file is named arft.log and the Verity internal log is named arftext.log. Both files are located in <ar_install_dir>\Arserver\Db on a Windows server, or <ar_install_dir>/db on a UNIX server. You cannot change the path or file name.
Full Text Search Log ! 71
The FTS log records activity only for the FTS indexing process. FTS activity during searching is logged as part of the SQL log, as described in SQL Log on page 57.
Note: The FTS log quickly becomes very large, using a large amount of disk space. FTS logging also slows down the re-index operation. Do not turn on the FTS log during a re-index operation.
Following is a sample FTS log file:

/* Thu Jan 13 16:00:17 2000 */ /* Thu Jan 13 16:00:18 2000 */ /* Thu Jan 13 16:00:36 2000 */ entry Id 000000000000006. /* Thu Jan 13 16:00:36 2000 */ schema 25. /* Thu Jan 13 16:00:36 2000 */ /* Thu Jan 13 16:00:36 2000 */ /* Thu Jan 13 16:00:38 2000 */ /* Thu Jan 13 16:00:38 2000 */ FTS Trace Log -- ON FTS: Reloading cache got INDEX_EID for schemaId 25, fieldId 536870915, VDK insert eid 000000000000006 in field 536870915, Flushing VDK index commands. Submitting 1 records to VDK. Done submitting 1 records to VDK. Done flushing VDK index commands.
In the example, the FTS process received a request to index data from a field in an entry. It prepared the data and submitted it. For more information about FTS debugging, refer to the Developing AR System Applications: Advanced guide.
Mid Tier Logging

You can log mid tier activity to help you debug problems with the mid tier. The mid tier log statement includes method calls, code execution, and informative messages for mid tier components. Mid tier log statements are written into armidtier.log file in the mid tier installation directory. Servlet Engine log and Webserver log statements are logged separately in their respective install directories. To set logging properties for the mid tier, use the AR System Configuration Tool, which contains online help to guide you through filling out the fields.
Following are some examples from a mid tier log:

!
Log Statement with Session ID

<WtXUmAWNLW8QgAaLX9KU> <FormAction> <2001-02-28 16:35:36,639> <Reached isDataAvail() method > Log Statement with no Session ID <-> <Reporting> <2001-02-28 16:35:36,655> <In doStartTag, atts[0] = 16 > Log Statement with Session ID and Exception Stack Trace < WtXUmAWNLADFERSDFFSF> <TagHandler> <2001-02-28 16:35:36,671> <IOException in tagHandler()> java.lang.IOException: IOException at com.remedy.arsys.framework.FormTag.tagHandler(FormTag.java, Compiled Code) at java.lang.Exception.<init>(Exception.java, Compiled Code) at com.remedy.arsys.framework.FormTag.AddInitTags(FormTag.java, Compiled Code) at com.remedy.arsys.framework.FormTag.doStartTag(FormTag.java:122) at pagecompile._jsp._Bugs_xjsp._jspService(_Bugs_xjsp.java, Compiled Code) at com.unify.ewave.servletexec.JSP10HttpJspPage.service JSP10HttpJspPage.java) at com.unify.ewave.servletexec.JSP10Servlet.service (JSP10Servlet.java) at javax.servlet.http.HttpServlet.service(HttpServlet.java:865) at com.unify.ewave.servletexec.ServletExec.CallServletService ServletExec.java) at com.unify.ewave.servletexec.ServerHostInfo.processApplRequest (ServerHostInfo.java, Compiled Code) at com.unify.ewave.servletexec.ServletExec.ProcessRequest (ServletExec.java, Compiled Code
Other Trace Modes

The other daemons (such as armaild on AIX) and two access control tools (arcache and arreload) have one trace option that is either on or off for the duration of the run. The log is activated by using the -d (for debug) command-line option. The log output is written to stdout and shows what is or is not working correctly for the system. It enables you to see exactly what the system is doing during the operations being performed.
Other Trace Modes ! 73
Additional Resources
The following additional resources can help you with your AR System server:
!
Server EventsA form that contains recorded events affecting AR Systems data dictionary. For more information about this form, refer to the Developing AR System Applications: Advanced guide. Server StatisticsA form that contains periodically recorded statistics about the server. For information about the Advanced tab where statistics are set, refer to the Developing AR System Applications: Advanced guide. Windows EventViewerAn application that enables you to monitor and log events in your Windows system. Use the following logs for the specified problems:
! !
System log for trouble starting services Application log for trouble with the AR System server
UNIX SyslogA file on UNIX servers that logs all serious server errors encountered. Error logA file that logs all AR System error messages you encounter. It is named arerror.log. For more information about the arerror.log file, see Server Error Log on page 46.
CHAPTER
Debugging
Debugging an AR System Application

Debugging helps developers determine where a malfunction or unexpected behavior occurs in an application and how to correct it. This section discusses how to debug AR System applications. It includes information about debugging tools and gives an example.
Overview
Understanding form-based application design is helpful in developing a debugging strategy. An AR System application is form-centric: almost every operation in an application occurs in the context of a form, and each form is associated with workflow. Each field on a form corresponds to a column in an SQL table. The user interacts with a form.
Debugging ! 75
The following figure provides an example of what a general debugging procedure might look like.
Figure 3-1: Debugging AR System Applications
76 "Chapter 3Debugging
Debugging Process
The following steps can help guide you through debugging an AR System application.
Step 1Where is the Bug Located?

Isolating the source is the process of eliminating all the possible causes of the bug, until the actual cause is found. After you have identified a small number of objects as probable causes, you can usually solve the problem by examining the object definitions.
Identifying Problem Forms
Starting at the form level, determine which forms are involved with the problem.
!
Can you observe the bug directly on a form? For example, the contents of a field are not what you expect, or fields that should be visible are hidden. In this case, you know immediately which form to focus on.
Can you observe only the results of the bug? The application developer must understand the workflow to determine the correct form. For example, a macro may run on one form that pushes data onto, or creates a request in, a different formthe problem is observed in the form that contains the pushed data, not the form with the bug.
Identifying Problem Fields
Once you have isolated the form that contains the bug, isolate any fields that are associated with the problem.
! !
Can you observe the bug directly in a field? Is a macro involved? If so, you should look at the macro definition. Are the field properties involved? For example, the field is not large enough, or the default values are incorrect.
Debugging an AR System Application ! 77
Step 2Is Workflow Involved?

If you did not resolve the bug by reducing the possible sources of the problem to a form and a small number of fields, the problem is probably workflow related. Workflow is the source of the majority of code bugs. Workflow debugging is complex because a workflow object responds to a two-stage trigger of cause and effect.
!
The first trigger stage (the Execute On condition) is an application event. Filters use request transitions such as Modify or Submit while active links use an interface event like Open Window. The second trigger stage is the evaluation of a qualification. If the qualification evaluates to TRUE, then the actions specified in the If branch are executed. If the qualification evaluates to FALSE, then the actions specified in the Else branch are executed.
Is the Problem with Server-Side Workflow?
Does something happen on the server every time you create a new request? If so, the bug probably involves the server side (filters and escalations).
Is the Problem with Client-Side Workflow?
Does something happen on the server every time you select a button on a form? If so, the bug probably involves the client side (active links).
Which Objects Respond to the Event?
After you have decided to focus on either filters, escalations, or active links, determine which objects respond to the eventthe filters or active links that have an Execute On condition that matches the cause. This reduces the number of objects to examine.
Step 3Reproduce the Bug

AR System Windows User Tool is equipped with a logging feature that captures real-time combined (server and client) workflow. You can usually resolve workflow-related bugs, including combination active link/filter bugs, by inspecting this log. For more information about logging, see Chapter 2, Using Log Files
Using the Combined Workflow Log

1 After you have reduced the number of workflow objects to examine, log in to
AR System Windows User Tool.

2 Enable the workflow log on the Logging tab of the Tools > Options dialog
box.
3 Duplicate the problem you are debugging. 4 Examine the log file, which will have captured information about the cause
and effect and execution sequence of filters and active links.
Debugging Tools
Debugging utilities show how objects are related; this is helpful in isolating objects that may be involved with the bug. AR System Administrator includes a feature designed specifically for working with large AR System applications: an object browser. The object browser is ideal for determining the workflow associated with a form; it can be used to find nearly any relationship among AR System objects. In addition, Remedy offers a range of unsupported utilities designed for debugging large applications. See the Applications Gallery at http://www.remedy.com/appgall/util/util.htm. Utilities that you may find useful are:
!
armaced A utility that is used to edit an existing macro; create a new macro; add, modify, delete, and re-arrange macro commands; rename the macro; or edit the macro help text. Although this utility is available, it is recommended that you convert existing macros to workflow actions, which are easier to maintain. arstructThis utility provides a printout of an applications workflow.
Note: The armaced and arstruct tools are not supported by Customer Support.
Debugging an AR System Application ! 79
Debugging Example
Chris Code works for a county maintenance department. Chris built two applications: one designed to track road maintenance and repairs in the district, and one to report bugs in the application. Chris designed a form called Road Defect, which can be displayed in different ways (with different fields appearing on the form), depending on the contents of various fields. The user clicks on an action button to change the view. When Terry Test clicked on the action button, the form did not change as expected, so Terry reported a bug. The bug report gives Chris a place to start: with the Road Defect form. Chris also knows that a specific button is involved. This information has isolated the problem to a specific area. Chris looks at each active link in AR System Administrator to determine which active links are associated with the button. The links are all simple code, without macros, and the code is correct. Chris then uses active link logging to record the link actions while reproducing the problem. The log file shows that one of the active links is not running, so Chris looks at the active link definition in AR System Administrator and discovers that the link is disabled. Chris enables the active link and reproduces the problem. The active link still does not work, so Chris looks at the workflow log again, and sees that the qualification is failing. The Run If qualification shows that the qualification depends on the contents of a display-only field, normally hidden on the form. Using AR System Administrator, Chris makes the field visible and tests the problem again. The contents of the field are not correct. Chris learned that another programmer disabled the display-only field because it appeared not to be affecting the application. Knowing that most display-only fields have values pushed from other forms, Chris inspects the workflow for the display-only field. The value is set by a single active link, so Chris looks at the active link definition and sees that the source of the Set Fields action is a concatenation of a field the user fills in using a menu, and a string constant, and the constant is not correct. Chris corrects the constant, which solves the problem.
CHAPTER
Searching Database
This chapter provides information for setting up and synchronizing the search database. Setting up the search database enables you to see which workflow is related to fields in a form. After you set up the search database, you can search for objects on a server, create advanced searches, and save the results of your searches to a workspace. The procedures related to these topics are provided in the following sections.
Searching Database ! 81
Searching for Objects

AR System Administrator enables you to search for objects on a server. To begin, you must set up and synchronize the search database. After the database is synchronized, you can perform a search. The following Tools menu options enable you to use the search feature:
!
Set Up Search DatabaseCreates the required forms for establishing the Search database. You will use this menu item once for each server because the menu item is active only if the forms are not available on the server. For information about setting up the search database, refer to Setting Up the Search Database on page 83. Object SearchOpens the Search window where you can set a query to search the database. Before using this menu option, you must synchronize your database with the Tool > Sync Search Database menu command. For more information, see Searching for Objects on a Server on page 86. The Object Search menu item is disabled if any of the following statements are true:
!
The search database reference form does not exist. (You can create it by choosing Tools > Set Up Search Database.) The search database was never synchronized for the current server. (You can synchronize the database by choosing Tools > Sync Search Database.) Synchronization is currently running.
Sync Search DatabaseOpens a dialog box that shows when the database was last synchronized and the current synchronization status of the search database. For more information, see Synchronizing the Search Database on page 84.
Note: To be able to search, the AR System server on which you want to search must be version 4.0 or later.
82 "Chapter 4Searching Database
Setting Up the Search Database

To enable AR System Administrator to search for objects, you must set up the search database. When you set it up, the following objects are created, and they are listed in the appropriate object list in the Server Window:
! ! ! ! ! !
object_search_admin in the Forms list object_search_details in the Forms list object_search_ref in the Forms list object_search_adminSetTime in the Filters list object_search_adminFilter#1 in the Filters list object_search_adminFilter#2 in the Filters list
Warning: The AR System Administrator tool uses these forms to perform searches. Do not rename or modify these forms or the filter. If you want to delete them, delete all of them.
You must set up the search database if you want to see which workflow is related to fields in a form. For more information, refer to the Related Workflow tab discussion in the Developing AR System Applications: Basic guide. Setting Up the Search Database
1 Select a server to administer. 2 Choose Tools > Set Up Search Database.
If this option is disabled, the search database is already set up or the server version is prior to 4.x.
Searching for Objects ! 83
Synchronizing the Search Database

The first time you perform a search, you must synchronize the database. Synchronization takes a snapshot of the database. The searching feature uses this snapshot, so if you make any modifications to the database, you will not see new objects or changes. To see any changes, synchronize the database again. For example, if you synchronize a database, and another administrator adds new objects to the server after the synchronization, you will not find the new objects in the search. You must synchronize the database again.
Note: Synchronizing the search database may be a time-consuming process. It may also slow down your machine if you have a large AR System database.
If you or another administrator exits AR System Administrator while the database is being synchronized, you will not be able to perform an object search because the system assumes that the database is still being synchronized. To correct this problem, perform one of the following tasks:
!
Delete the three search forms: object_search_admin, object_search_details, and object_search_ref. Then, set up the database again. Open the object_search_admin form in AR System Windows User Tool. Perform a search and modify the first request by entering 0 in the Run/Not Run field. When the synchronization is running, the Run/Not Run field contains a 1. When you enter 0, the system knows that synchronization is not running.
Synchronizing the Search Database

1 Select a server to administer. 2 Choose Tools > Sync Search Database.
The Sync Search Database dialog box appears.
Figure 4-1: Sync Search Database Dialog Box
3 Click the Update Search Database button. 4 To continue working in AR System Administrator while the database is
synchronizing, click Close. When the database is synchronized, you receive a confirmation message.
5 Click OK.
Searching for Objects on a Server

Searching for Objects on a Server
1 Select a server to administer. 2 Choose Tools > Search Objects.
The Search Objects window appears.
Figure 4-2: Search Objects WindowBasic Tab
If you have previously saved a search, you can use the Search Name field to set up a search; otherwise, go to step 3. For more information, see Saving Searches on page 90.
3 In the Name field, enter the name of the object you want to find.
In this field, you can use AR System wildcards, which are listed in the Developing AR System Applications: Basic guide.
4 In the Type field, select All to find any object, or select a specific type from
the menu list.

5 Optionally, click the Advanced tab, and enter search criteria.
For information about using the Advanced tab, see Creating an Advanced Search on page 88.
6 Click Find Now.
The objects with the name you entered are listed in the Results list. If you or another administrator have made changes to the database since it was synchronized, the objects listed may be different than what is in the current database. For example, objects may have been added or modified. Deleted objects will not appear even if the database is not synchronized.
7 Double-click on an object to open it. 8 Click New Search to clear all fields and begin a new search.
Creating an Advanced Search

When using the Search Objects window, you can create an advanced search for any object by using the Advanced tab (Figure 4-3).
Figure 4-3: Search Objects WindowAdvanced Tab
In the Advanced tab, use the following fields to create a qualification in the Search Criteria field:
Property Operation Value The property you want to use in the search criteria. Each object type has a different set of properties. The operation you want to use in the search criteria. The available operations vary depending on the property you selected. The value for which you want to search. You can type a value, or you can select one from the menu list, if available.
Note: If you select All in the Type field of the Basic tab, these fields are disabled. You must type a command in the Search Criteria field.
Using the Advanced Tab

1 In the Search Objects window, click the Advanced tab.
Under the Advanced tab, the Search Criteria field contains part of the qualification statement. The field obtains the information from the data you selected in the Basic tab.
2 Click the AND, OR, or NOT button, as appropriate, for your qualification. 3 Create additional criteria for your search: a Select a property from the Property menu list. b Select an operation from the Operation menu list. c Enter a value in the Value field.
If you are searching for workflow objects that contain one workflow action type or an Execute On condition type, you can use the equal to (=) or not equal to (!=) operators. For example, you may want to find all of the active links that contain the Call Guide action only. If you are searching for workflow objects that contain more than one action type or Execute On condition type, use the LIKE operator. For example, you want to find all of the active links that contain the Change Field and Commit Changes actions.
d Click Insert.
The qualification is added to the Search Criteria field. Parentheses are added to the qualification where appropriate. You can also type the qualification instead of using the buttons under the Advanced tab. You can only search on one type of object at a time. If you click the Basic tab after you have entered criteria under the Advanced tab, the other types in the Type field will be disabled. To enable the other types, click New Search.
4 Click Find to find the objects that match the criteria.
Advanced Search Examples

The following list shows examples of qualification that can be used in the Search Criteria field of the Advanced Search tab:
!
('Type' = "Schema") AND NOT ('Form Type' = "Dialog") This statement searches for all forms, except forms that are dialogs. ('Type' = "Active Link") AND ('Actions' LIKE "Call Guide") AND (Name LIKE "AR%")
This statement searches for all active links that contain the Call Guide action and begin with AR. If an active link contains the Call Guide action and another action, that active link will not appear in the Results list. To find all active links that contain a Call Guide action and possibly other actions, change the second clause to:
('Actions' LIKE "%Call Guide%")

!
('Type' = "Menu") AND ('Menu Type' = "Character") This statement searches for all character menus.
Saving Searches
You can save searches that you use on a regular basis. Saved searches are listed in the Search Name menu list of the Search Objects window. Saving a Search
1 Define the search criteria as described in Searching for Objects on a Server on
page 86.
2 Type a name for the search in the Name field. 3 Click the Save Search button.
You can also use choose Actions > Save Search. The search is saved, and its name is added to the Search Name menu list. Executing a Saved Search
1 Choose Tools > Search Objects to open the Search Objects dialog box. 2 Select a saved search from the Search Name list. 3 Click Find Now.
Deleting Saved Searches

If you save a search that you no longer use, you can delete it from the Search Name menu list. Deleting a Search
1 Choose Tools > Search Objects to open the Search Objects window. 2 Choose Actions > Delete Search.
The Delete Searches dialog box appears.

3 Select the search you want to delete. 4 Click Delete.
Saving Results to a Workspace

You can save the results of your search to a workspace. This method is useful if you want to create a workspace, but it would be difficult to find each object in the object list. For more information about workspaces, refer to the Developing AR System Applications: Basic guide. Saving Results to a Workspace
1 Define criteria, and search for objects as described in Searching for Objects on
a Server on page 86.

2 Choose Actions > Save Results to Workspace.
The Save Results to Workspace dialog box appears.
Figure 4-4: Save Results to Workspace Dialog Box
3 In the Workspace Name field, type a descriptive name for the workspace. 4 Optionally, in the Description field, type information that will help you
remember what objects are in the workspace you are creating.

5 Click OK.
Index
A
active links debugging 78 designing 39 log information 63 logging 62 admin queue 59 advanced, searches 88, 90 alerts, log file 49 Analyze Forms dialog box 16 Analyzer. See AR System Analyzer analyzing forms 16 API log file 51 applications debugging 75 performance 23, 39 AR System Administrator configuring threads and 25 definition changes 22 AR System Administrator Analyzer Tool. See AR System Analyzer AR System Alert, troubleshooting 22 AR System Analyzer 16 AR System Log Display tool 69 AR System Windows User Tool client-side logging 66 workflow logging 62 arcache utility 73 archiving data 28 ardb.conf (ardb.cfg) file 32 arerror log file 46, 74 arforkd log file 47, 50 arithmetic operators 37 arlogdisplay utility 69 armaced utility 79 armaild (armailex) utility 73 arreload utility 73 arservdsd utility 51 arservftd (arfts) utility 71 arstruct utility 79
C
client-side debugging 78 logging 65 combined log files 65 configuring DBMS memory 30 memory 24 multithreaded servers 2427 CREATE statement 31
D
data, archiving 28 database administrator extensions 3033 memory allocation 29 performance tuning 29 search 82, 83 setting up search 82 synchronizing for search 82, 84
Index ! 93
database management system. See DBMS DB2 database, ardb configuration file 32 DBMS indexes and 34 memory allocation 29 performance tuning 23 debug modes, using log files 46 debugging applications 75 example 80 forms 77 tools for 79 deleting, searches 91 distributed server log file 51
forms analyzing 16 debugging 77 designing 43 object_search_admin 83 object_search_details 83 object_search_ref 83 Server Statistics 14 views 43 full text search log file 71 logging searches 58 search strategies 38
G
groups, client logging 66 guides logging activity 46 trace 46
E
email, armaild (armailex) utility 73 error log file 46 escalations designing 40 log file 53 EventViewer 74 examples advanced search 90 ardb.conf (ardb.cfg) file 32 debugging 80
H
hardware and performance 23 help, menus 46
I
improving performance. See performance tuning indexes DBMS and 34 fields 35 optimizing number of 35 performance tuning 34 Informix database, ardb configuration file 32
F
fields debugging 77 designing 43 indexing 35 performance tuning 43 filters debugging 78 designing 40 log file 54 object_search_adminFilter#1 83 object_search_adminFilter#2 83 object_search_adminSetTime 83 format, log files 48
J
join forms, designing 42
94 " Index
L
log files activating 47 alert 49 analyzing performance and 15 API 25, 51 AR System Log Display tool 69 arerror 74 arerror.log 46 arforkd 47, 50 arlogdisplay utility 69 client-side logging 65 combined 65 display tools 69 distributed server 51 entries 48 escalation 53 filter 54 format 48 full text search 71 mid tier 72 plug-in 56 real-time 69 renaming 47 server errors 46 SQL 57 Syslog 74 thread 58 types 46 user 59 values for entries 48, 59 Windows EventViewer 74 workflow 62 log information active links 63 macros 64
logging enabling for logging group 67 server activity 74 logging activity, guide 46
M
macros log information 64 performance tuning 40 memory DBMS allocation 29 performance tuning 24 menus, help text 46 Microsoft SQL database, ardb configuration file 33 mid tier, logging 72 modes, trace 46 multithreaded servers, performance tuning 2427
N
network, performance tuning 23 NOT operations 36
O
object_search_admin form 83 object_search_adminFilter#1 filter 83 object_search_adminFilter#2 filter 83 object_search_adminSetTime filter 83 object_search_details form 83 object_search_ref form 83 objects, searching 82, 86 Oracle database, ardb configuration file 33
Index ! 95
P
peak use and performance 22 performance tuning active links 39 analyzing forms 16 API logging and 25 applications 28, 39 AR System Alert, troubleshooting 22 archiving data and 28 blocking actions 41 database 29, 3033 DBMS memory 29 escalations 40 establishing goals 12 fields 43 filters 40 form views 43 hardware considerations 23 indexes 34 join forms 42 log files 15 macros 40 memory configuration 24 network 23 process actions 42 QBE match 37 searches and 34, 36 servers 2427 set fields actions 41 SQL menus 38 structural changes and 22 system analysis 12 system load 28 threads 2427 troubleshooting 2022 user actions 28 workflow 39 plug-ins, log file 56 process actions, designing 42
S
saving results in workspace 91 searches 90 search database 82, 83 searching actions deleting 91 executing 90 optimizing 34 qualifying 36 saving 90 saving results to workspace 91 advanced 88, 90 for objects 82, 86 types of searches arithmetic operators 37 full text search 38 NOT operations 36 QBE 37 wildcards 37 using SQL menus 38 server information, log files 47 server statistics 14 server utilities arcache 73 arlogdisplay 69 armaced 79 armaild (armailex) 73 arreload 73 arservdsd 51 arservftd (arfts) 71 arstruct 79 servers debugging 78 error log file 46 logging server activity 74 performance tuning. See performance tuning set fields action, designing 41 setting up, search database 82 SQL clauses, appending 30 CREATE statement 31 log file 57 menus 38
Q
QBE searches, performance tuning 37
R
real-time log files 69
96 " Index
statistics, server 14 Sybase database, ardb configuration file 33 synchronizing search database 82, 84 Syslog 74 system load, performance and 28
T
threads log file 58 performance tuning 2427 trace modes activating 47 guide activity 46 overview 46 troubleshooting performance 2022 tools 79 tuning. See performance tuning
U
users log file 59 performance and 28 utilities arcache 73 arlogdisplay 69 armaced 79 armaild (armailex) 73 arreload 73 arservdsd 51 arservftd (arfts) 71 arstruct 79 debugging 79
V
values, log entries 48, 59
W
web, logging 63, 72 wildcards, searching 37 workflow debugging 78 logging 62 performance tuning 39 workspaces, saving search results to 91
Index ! 97
98 " Index

Optimizing Troubleshooting Guide English

Caricato da

Informazioni sul documento

Descrizione originale:

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Optimizing Troubleshooting Guide English

Caricato da

Copyright:

Formati disponibili

Action Request System 5.

1 Optimizing and Troubleshooting AR System

PART NO: AR-512-OTG-01

Action Request System 5.1

4 " Table of Contents

Optimizing and Troubleshooting AR System

Action Request System 5.1

6 " Table of Contents

Action Request System 5.1

Overview of This Manual

Optimizing and Troubleshooting AR System

Action Request System Documents

Installing AR System AR-512-IG-01

Action Request System Documents ! 9

Action Request System 5.1

Title and Part Number AR System Email Engine Guide AR-512-EEG-01

List of AR System error messages with expanded descriptions.

AR System Alert Help

Action Request System 5.1

Establishing Performance Goals

Gathering Baseline Information

12 "Chapter 1Managing Performance

Optimizing and Troubleshooting AR System

What is your hardware configuration?

What is your AR System configuration?

What is your database configuration?

Gathering Baseline Information ! 13

Action Request System 5.1

Using Server Statistics to Gather Baseline Data

Figure 1-1: Server Statistics Form

14 "Chapter 1Managing Performance

Optimizing and Troubleshooting AR System

Using AR System Log Files to Gather Baseline Data

Gathering Baseline Information ! 15

Action Request System 5.1

Using AR System Analyzer

Figure 1-2: Analyze Forms Dialog Box

16 "Chapter 1Managing Performance

Optimizing and Troubleshooting AR System

Using AR System Analyzer ! 17

Character Fields Longer than 255 Characters and Used in Search

Character Fields with QBE = Anywhere and Used in Search

18 "Chapter 1Managing Performance

Optimizing and Troubleshooting AR System

Analyzing Forms for Performance

Using AR System Analyzer ! 19

Action Request System 5.1

Troubleshooting Performance Problems

20 "Chapter 1Managing Performance

Optimizing and Troubleshooting AR System

Figure 1-3: Performance Tuning and Troubleshooting Flowchart

Troubleshooting Performance Problems ! 21

Action Request System 5.1

Troubleshooting AR System Alert Performance

Troubleshooting Server Processes on Windows

Tuning System Performance

Peak Use Considerations

Changing the lengths of fields Adding or deleting fields

22 "Chapter 1Managing Performance

Optimizing and Troubleshooting AR System

Adding, deleting, or changing:

Forms Active links Filters Menus New object groups

Tuning System Performance ! 23

'Status' != "Closed" $TIMESTAMP$ 'Create Date' > 606024