Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
Guidance for fine-tuning T24 implementations on Windows Server 2008 R2 and on the SQL Server 2008 R2 and SQL Server 2012 database platform
V1.1 Published: July 2012 Produced by Temenos and Igor Pagliai, Microsoft Based on previous work done by: Lonnye Bower, Kun Cheng, Konstantin Dotchkoff, Roger Toren - writing Peter Carlin, Nicholas Dritsas - reviewing
Abstract
TEMENOS T24 (T24) is a complete banking solution designed to meet the challenges faced by financial institutions in todays competitive market. T24 provides a single, real-time view of clients across the entire enterprise, making it possible for banks to maximize returns and keep costs down. Microsoft SQL Server provides the ideal database platform for T24. By choosing the Microsoft platform, T24 customers experience faster funds transfers, higher security-trade volumes, and quicker close-of-business processes; T24 customers can also benefit from open, state-of-the-art technologies to accelerate innovation, greatly increasing the speed and effectiveness with which new products and services are created. Using Windows Server 2008 R2 as the operating system makes it possible for T24 to exceed performance standards in a scalable, reliable environment that offers ease of management. This white paper provides best practices for configuring and running TEMENOS T24 on Windows Server and on the SQL Server database platform. Implementing these best practices can help you avoid or minimize common problems and optimize the performance of your TEMENOS T24 implementation.
This document is provided as-is. Information and views expressed in this document, including URL and other Internet Web site references, may change without notice. You bear the risk of using it.
This document does not provide you with any legal rights to any intellectual property in any Microsoft product. You may copy and use this document for your internal, reference purposes.
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
Table of Contents
OVERVIEW ............................................................................................................................................... 5 1 BEST PRACTICES FOR THE DATABASE SERVER .................................................................................. 7 1.1 SERVER RECOMMENDATIONS ............................................................................................................... 7 1.1.1 Use the Latest Software Versions...................................................................................................... 7 1.1.2 Use a 64-Bit Server ............................................................................................................................ 8 1.1.3 Use a Dedicated Server ..................................................................................................................... 8 1.2 SIZING RECOMMENDATIONS ................................................................................................................ 8 1.3 STORAGE DESIGN RECOMMENDATIONS .................................................................................................. 8 1.3.1 Use SAN and RAID 10 ........................................................................................................................ 8 1.3.2 Set the Partition Offset ..................................................................................................................... 9 1.3.3 Set the File Allocation Unit Size and Stripe Size................................................................................. 9 1.3.4 Dedicate a High Percentage of SAN Cache to Writes ...................................................................... 10 1.3.5 Configure Windows Drives .............................................................................................................. 10 1.4 HARDWARE AND NETWORK GUIDANCE ................................................................................................ 10 1.4.1 Windows Server Power Plan ........................................................................................................... 10 1.4.2 Use a Private Network Segment ..................................................................................................... 11 1.4.3 Enable Receive-Side Scaling ............................................................................................................ 11 1.4.4 Consider NIC Teaming ..................................................................................................................... 12 1.4.5 Use Multiple HBAs and Set the HBA Queue Depth .......................................................................... 12 1.5 DATA, LOG, TEMPDB, AND BACKUP FILE RECOMMENDATIONS .................................................................. 12 1.5.1 Configure Data Files ........................................................................................................................ 12 1.5.2 Configure the Log File ..................................................................................................................... 13 1.5.3 Configure tempdb Files ................................................................................................................... 14 1.6 RECOMMENDATIONS FOR MEMORY SETTINGS ....................................................................................... 15 1.6.1 SQL Server Memory Settings ........................................................................................................... 15 1.6.2 Lock Pages in Memory .................................................................................................................... 16 1.6.3 Instant File Initialization (IFI) .......................................................................................................... 17 1.7 GUIDANCE FOR SQL SERVER CONFIGURATION SETTINGS ......................................................................... 17 1.7.1 Consider Lower Fill Factor ............................................................................................................... 18 1.7.2 Increase the SQL Server Recovery Interval ...................................................................................... 19 1.7.3 Use Trace Flag 834 .......................................................................................................................... 19 1.7.4 Keep Default Setting for Degree of Parallelism ............................................................................... 20 1.7.5 Keep Default Setting for Number of Worker Threads ..................................................................... 20 1.7.6 Encrypt Client Communication if Required...................................................................................... 20 1.7.7 T24 Database Configuration settings.............................................................................................. 20 2 BEST PRACTICES FOR THE APPLICATION SERVER ........................................................................... 22 2.1 SOFTWARE RECOMMENDATIONS ........................................................................................................ 22 2.1.1 Use the Latest Software Versions.................................................................................................... 22 2.2 T24 CONFIGURATION RECOMMENDATIONS .......................................................................................... 22 2.2.1 Configure the T24 Bulk Factor Parameter ....................................................................................... 22
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server 3
2.2.2 2.2.3
Configure the T24 Handle Cache Parameter ................................................................................... 22 Consider Using Hyper-Threading .................................................................................................... 23
BEST PRACTICES FOR PROVIDING HIGH AVAILABILITY ................................................................... 24 3.1 HIGH AVAILABILITY RECOMMENDATIONS FOR THE DATABASE ................................................................... 24 3.2 HIGH AVAILABILITY RECOMMENDATIONS FOR THE STORAGE..................................................................... 25 3.3 T24 RECOVERY................................................................................................................................ 25 3.3.1 Online Recovery .............................................................................................................................. 25 3.3.2 Close of Business (COB) ................................................................................................................... 25
BEST PRACTICES FOR DISASTER RECOVERY .................................................................................... 26 4.1 4.2 4.3 4.4 4.5 SQL SERVER 2012 HIGH-AVAILABILITY AND DISASTER RECOVERY (HADR) ENHANCEMENTS.......................... 26 SAN MIRRORING ............................................................................................................................. 27 SYNCHRONOUS DATABASE MIRRORING ................................................................................................ 27 ASYNCHRONOUS REPLICATION METHODS ............................................................................................. 27 RECOMMENDATIONS FOR CONNECTION BANDWIDTH ............................................................................. 28
5 6
BEST PRACTICES FOR REPORTING .................................................................................................. 29 BEST PRACTICES FOR PERFORMANCE TUNING .............................................................................. 30 6.1 GUIDANCE FOR IMPROVING QUERY PERFORMANCE ................................................................................ 30 6.1.1 Optimize Standard T24 Queries ...................................................................................................... 30 6.1.2 Optimize Customer-Specific T24 Queries ........................................................................................ 31 6.2 RECOMMENDATIONS FOR MONITORING PERFORMANCE OF THE DATABASE TIER .......................................... 36
BEST PRACTICES FOR SYSTEM MAINTENANCE ............................................................................... 37 7.1 GUIDANCE FOR BACKING UP THE DATABASE ......................................................................................... 37 7.1.1 Use Backup Compression ................................................................................................................ 37 7.1.2 Implement a Backup Schedule ........................................................................................................ 37 7.1.3 Back Up System Databases ............................................................................................................. 38 7.2 RECOMMENDATIONS FOR UPDATING STATISTICS .................................................................................... 38 7.3 RECOMMENDATIONS FOR REORGANIZING OR REBUILDING INDEXES ........................................................... 38 7.3.1 Reorganize Indexes ......................................................................................................................... 39 7.3.2 Rebuild Indexes when Necessary .................................................................................................... 40 7.4 RECOMMENDATIONS FOR UPDATE MANAGEMENT ................................................................................. 40 7.4.1 SQL Server 2012 Setup Product Update .......................................................................................... 40 7.4.2 Run Anti-Virus Checks During Non-Peak Hours ............................................................................... 41 7.4.3 Consider Running Windows Updates on a Test Server before Production ...................................... 41
8 9 10 11 12 13
SUMMARY ..................................................................................................................................... 43 APPENDIX 1: CREATE A MAINTENANCE PLAN TO BACK UP THE TEMENOS DATABASE ................... 45 APPENDIX 2: CREATE A MAINTENANCE PLAN TO BACK UP THE SYSTEM DATABASES .................... 51 APPENDIX 3: UPDATE STATISTICS .................................................................................................. 57 APPENDIX 4: REORGANIZE AND REBUILD INDEXES ........................................................................ 61 APPENDIX 5: INSTALLATION CHECK LIST ........................................................................................ 68
4
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
Overview
To help financial institutions face the challenges posed by todays around-the-clock, global marketplace, Temenos Group AG, the leading provider of integrated core banking systems, offers TEMENOS T24 (T24). T24 pairs comprehensive and powerfully flexible business functionality with advanced and scalable architecture. T24 is built on an open architecture, and it uses established standards such as HTTP, XML, and Java 2 Platform, Enterprise Edition (J2EE). The design of T24 supports multiple application servers and provides horizontal scalability with true non-stop resilience. The capabilities of T24 can be enhanced by the choice of an enterprise-ready database platform. Customers running T24 on a Windows Server operating system and Microsoft SQL Server data management software benefit from a wide range of products and tools that can be used to further improve the performance and extend the capabilities of the T24 banking solution. Figure 1 shows a logical model of the interaction of the various services and components that make up a T24 banking implementation. In this white paper, we focus on best practice guidance for the database layer (in green).
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
This white paper, intended for database administrators (DBAs), describes optimizations that you can make to the database tier and describes the database tiers interaction with the application tier to help ensure a successful deployment of T24 on SQL Server. The paper first discusses best practices for configuring the database server and the application servers and provides guidance for building scalability and high availability into the T24 banking solution. The paper then looks at the options for disaster recovery. Best practices for reporting are presented, and best practices for monitoring the performance of the database tier and for system maintenance are discussed. The paper also provides appendices with step-by-step guidance and extensive links for further information. Implementing these best practices can help you optimize the performance of a T24 implementation and can help avoid or minimize common problems.
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
For more detailed information, see Windows Server 2008 R2. Note: It is highly recommended to install Service Pack 1 for Windows Server 2008 R2 along with the recommended non-security related hotfixes for the Cluster component mentioned in the article: Recommended hotfixes and updates for Windows Server 2008 R2 SP1 Failover Clusters (http://support.microsoft.com/kb/2545685)
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server 7
Its also recommended to install all the proposed operating system updates related to security available via Windows Update released from Microsoft immediately after the operating system installation and before installing SQL Server and T24.
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
LUN 2 13 14 15
LUN 4 46 47 48
You should refer to the article Predeployment I/O Best Practices for procedures to test your I/O subsystem performance. Note: For an online transaction processing (OLTP) system the ideal latency values on a welltuned I/O subsystem are: < 5 milliseconds (ms) for log (ideally 1 ms on arrays with cache) < 20 ms for data on OLTP systems (ideally 10 ms or less)
1.3.3 Set the File Allocation Unit Size and Stripe Size
When formatting the partition that will be used for SQL Server data files, you should use a 64kilobyte (KB) allocation unit size for data, logs, and the tempdb database. The stripe size is also important to reach an optimal configuration. This is set in the SAN management software, not through Windows Server. The following two equations can be used to determine if you have an optimal configuration. Each should result in an integer value: Partition_Offset Stripe_Unit_Size Stripe_Unit_Size File_Allocation_Unit_Size For details on managing disk partition sector alignment, see Disk Partition Alignment Best Practices for SQL Server.
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server 9
You can also present storage to SQL Server through mount points, disk volumes that are mounted as folders on other physical disks, without incurring a performance penalty. For further information about storage design for SQL Server, see Storage Top 10 Best Practices.
Performance and test the performance difference with a significant workload: if no difference will be found, Power Plan scheme can be reverted to the default value. For more information on this potential issue and instructions for changing the Power Plan options, see the links below: Degraded overall performance on Windows Server 2008 R2 http://support.microsoft.com/kb/2207548/en-us Hyperthreading, Turbo Boost, SQL Server and You (MSDN) http://msdnrss.thecoderblogs.com/2012/07/hyperthreading-turbo-boost-sql-serverand-you For general tuning information on Windows Server 2008 R2, see the white paper below: Performance Tuning Guidelines for Windows Server 2008 R2 http://www.microsoft.com/whdc/system/sysperf/Perf_tun_srv-R2.mspx
Note: You should use the Windows RSS registry keys to configure these values instead of NIC settings because NIC settings can be overridden by the Windows registry keys.
1.4.5 Use Multiple HBAs and Set the HBA Queue Depth
You should use at least two host bus adapters (HBAs) to provide redundancy and to increase bandwidth between the SAN and SQL Server. The HBA cards should be set as load balanced and configured to provide high availability among them. Connections between the SAN and server are ideally 8 gigabyte (GB)/sec (but not less than 2 GB/sec). The HBA queue depth may need to be increased if you have a small number of LUNs. A queue depth value between 128 (if there are few LUNs) and 32 (if there are many LUNs) should be considered, queues now default to per LUN rather than per target. When the queue depth is set too low, you may get increasing latency and less-than-expected throughput given the bandwidth between host/storage and the number of disks in a particular configuration.
writes by spreading its write operations across the files based on the ratio of free space among the files, so extending all files at once maintains this optimization. You can leave the autogrowth setting on as an insurance policy so that SQL Server does not stop when it runs out of space; however, do not rely on autogrowth to extend the database files as a standard way of operating. While you should not allocate space for the data files in small units, if you allocate in very large units during autogrowth, the application must wait (possibly several minutes) while the space is allocated. Since you cannot control when autogrowth engages, allocate only by the space needed for a few days of operations.
IMPORTANT: In SQL Server 2008 and SQL Server 2008 R2, a bug exists affecting transaction log file growth of 4GB (or multiples of it) as explained in the article: The SQL Server database
transaction log file does not grow by the configured file growth value (http://support.microsoft.com/kb/2633151)
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server 13
SQL Server 2012 is not affected, for SQL Server 2008 R2 Service Pack 1 and Cumulative Update 4 for SP1 must be installed to solve the problem, for SQL Server 2008 there is no fix; if not possible to apply the hotfix, its recommended to avoid 4GB (or multiples) increment.
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
14
NOTE: In SQL Server 2012 the location for changing the startup parameters is changed, be sure to check the right version in the provided link above. For further information, see the MSDN article Optimizing tempdb Performance.
For each single worker thread, the following amount of memory is required for specific CPU architecture: (x86): 512KB; (x64): 2048 KB; (IA64): 4096 KB;
If you need to do calculation over the 32 logical processors, the general formula below can be used:
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
15
32bit CPU <= 4 CPU = 256 > 4 CPU = 256 + ((#CPU - 4) *8) 64bit CPU <= 4 CPU = 512 > 4 CPU = 512 + ((#CPU - 4) * 16) For example, on a x64 bit machine with 32 logical CPUs, the maximum number of thread is internally calculated by SQL Server as (512 + (28*16)) = 960 and since on x64 each thread stack is 2MB, the total amount of memory that should be reserved is (960*2) = 1920 MB. NOTE: In SQL Server 2012, 32bit AWE memory feature is deprecated: The "awe enabled" SQL Server feature is deprecated http://support.microsoft.com/kb/2644592 For more information on how the memory management has been changed in SQL Server 2012, see the KB article and links below: Memory configuration and sizing considerations in SQL Server 2012 http://support.microsoft.com/kb/2663912 SQL Server Memory Manager Changes in Denali http://blogs.msdn.com/b/sqlosteam/archive/2011/01/04/sql-server-memory-managerchanges-in-denali.aspx New SQLOS features in SQL Server 2012 http://blogs.msdn.com/b/sqlosteam/archive/2011/11/10/new-sqlos-features-in-sqlserver-2012.aspx
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
16
This can take a quite a bit of time for a very large file which can become critical in disaster recovery and restore operations; to solve this problem, SQL Server 2005 (and later versions) can leverage Windows (Windows Server 2003 and later) OS Instant File Initialization (IFI) feature to remove the need to zero-out the file when it is allocated making the process almost instantaneous. The operating system just allocates the disk space, but the content of the file is actually what is originally on the disk. Instant File Initialization (IFI) is only available if the SQL Server database engine service account has been granted the Perform Volume Maintenance Tasks (SE_MANAGE_VOLUME_NAME) user right at the OS level on the server and all cluster nodes if a clustered configuration is used. Members of the Windows Administrator group have this right and can grant it to other users by adding them to the Perform Volume Maintenance Tasks security policy. To grant this right, use the following steps: On a pure performance perspective, using this feature is recommended, but there are some securities implications that should be considered and reviewed in the article below (also apply to SQL Server 2012): http://msdn.microsoft.com/en-us/library/ms175935(v=sql.105).aspx If the final customer is comfortable in being able to address this type of risk based on his inhouse security protection mechanisms, configurations and operations, then we recommend to proceed. NOTE: This feature is only available for data files, not transaction log files. For more information on this feature you can visit the link below: http://sqlskills.com/BLOGS/KIMBERLY/post/Instant-Initialization-What-Why-and-How.aspx
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
17
Pad_index On On On On On
Note: This testing has been done with a standardized workload. Every application has different functionality, and if high latch contention is observed, you must identify these candidates for the application-specific workload profile. For more information on the fill-factor option for indexes, see the article Fill Factor.
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
19
Leave value for parameter Auto Shrink to FALSE (default configuration); Leave value for parameter Page Verify to CHECKSUM (default configuration);
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
20
For modification of default value (0) for parameter Target Recovery Time (Seconds), see section 2.7.2 Increase the SQL Server Recovery Interval.
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
21
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
22
App Server Memory < 8 GB >= 8 GB and < 32 GB >= 32 GB and < 64 GB >= 64 GB
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
23
In a cluster configuration with jDLS installed on the database server, you can configure jDLS in resilient mode on both database server nodes and have the jDLS in the same failover group as SQL Server. If the jDLS or SQL Server fails, then the connections are dropped (as well as the locks), and the processes reconnect to the secondary node and take locks via the jDLS of that node.
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
24
For general overview on the high-availability features of SQL Server, see High Availability Always On Technologies and High Availability with SQL Server 2008 R2.
4.1 SQL Server 2012 High-Availability and Disaster Recovery (HADR) enhancements
SQL Server 2012 AlwaysOn Failover Cluster Instances (FCI) and AlwaysOn Availability Groups (AG) provide a comprehensive high availability and disaster recovery solution. Prior to SQL Server 2012, many customers used FCIs to provide local high availability within a data center and database mirroring for disaster recovery to a remote data center. With SQL Server 2012, this design pattern can be replaced with an architecture that uses FCIs for high availability and availability groups for disaster recovery business requirements. Availability groups leverage Windows Server Failover Clustering (WSFC) functionality, enable multiple features not available in database mirroring, with or without shared replicated SAN storage between sites providing a cost effective solution able to guarantee zero data loss. Based on these new SQL Server 2012 HADR functionalities, Microsoft and Temenos built a deployment reference architecture and guidance for implementing a high-availability and disaster-recovery solution for TEMENOS T24 running on the Microsoft Application Platform and leveraging SQL Server 2012 new features; the result of this work can be viewed in the white paper below: A deployment reference architecture and guidance for implementing an HADR solution for TEMENOS T24 running on the Microsoft Application Platform http://blogs.technet.com/b/sql_server_isv/archive/2012/05/18/a-deploymentreference-architecture-and-guidance-for-implementing-an-hadr-solution-for-temenost24-running-on-the-microsoft-application-platform.aspx
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server 26
For more information on SQL Server 2012 AlwaysOn technologies and Availability Group, you can visit the links below: Microsoft SQL Server AlwaysOn Solutions Guide for High Availability and Disaster Recovery http://msdn.microsoft.com/library/hh781257.aspx SQL Server 2012 AlwaysOn High Availability and Disaster Recovery Design Patterns http://sqlcat.com/sqlcat/b/msdnmirror/archive/2011/12/22/sql-server-2012-alwaysonhigh-availability-and-disaster-recovery-design-patterns.aspx
The primary methods of asynchronous replication are asynchronous database mirroring, log shipping, or transactional replication (server-to-server transactional replication or peer-to-peer replication). With these methods, there is a window of a few seconds to a few minutes where the transactions committed on the primary site are not yet committed on the DR site. Note that when using asynchronous database mirroring or log shipping, the log from the primary site cannot be applied to the DR site once the DR site has been used to store new transactions in its new role as the primary database. Transactions that were not shipped to the DR site before switching to the secondary database server are lost. For more information, see Log Shipping Overview, Database Mirroring Overview, and Transactional Replication Overview.
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
28
The reporting database may be configured for simple database recovery because it is not updated by client transactions; this lets you skip the backup process, but it means that if the database becomes corrupted, you need to re-initialize the subscription from the primary backup. Another possible alternative, both for SQL Server 2008 R2 and SQL Server 2012, is Transactional Replication but its not recommended since: It requires modifications to the database schema (triggers, views, tables, stored procedures); There are several strict requirements on the structure of the published tables; The overhead on the primary production T24 database may affect heavily database throughput;
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
29
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
30
You can use the following query to monitor the index usage on a regular basis: SELECT object_name(i.object_id) AS object_name, i.name AS index_name, s.index_id, user_seeks + user_scans + user_lookups AS user_reads, system_seeks + system_scans + system_lookups AS system_reads, user_updates, system_updates FROM sys.dm_db_index_usage_stats s JOIN sys.indexes i ON s.index_id = i.index_id AND s.object_id = i.object_id WHERE s.database_id = db_id() AND i.type <> 0 ORDER BY user_reads DESC
The close-of-business (COB) calculations are generally the most processing-intensive tasks in T24. The optimization of COB queries is critical for reducing the total duration of the COB. To identify slow-running queries (or just queries which can be optimized to help shorten the COB time), you can use the sys.dm_exec_query_stats dynamic management view. The following query selects the top 50 SQL Server statements ordered by the total CPU time (i.e., total amount of CPU time, in microseconds, for all executions of each statement): SELECT TOP 50 SUM(query_stats.total_worker_time) AS "total CPU time", SUM(query_stats.total_worker_time)/SUM(query_stats.executio n_count) AS "avg CPU Time", SUM(query_stats.execution_count) AS "executes", SUM(query_stats.total_logical_reads) AS "total logical reads", SUM(query_stats.total_logical_reads)/SUM(query_stats.execut ion_count) AS "avg logical reads", SUM(query_Stats.total_logical_writes) AS "total logical writes", SUM(query_Stats.total_logical_writes)/SUM(query_stats.execu tion_count) AS "avg logical writes", MIN(query_stats.statement_text) AS "statement text" FROM (SELECT QS.*,
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server 31
SUBSTRING(ST.text, (QS.statement_start_offset/2) + 1, ((CASE statement_end_offset WHEN -1 THEN DATALENGTH(ST.text) ELSE QS.statement_end_offset END - QS.statement_start_offset)/2) + 1) AS statement_text FROM sys.dm_exec_query_stats AS QS CROSS APPLY sys.dm_exec_sql_text(QS.sql_handle) as ST) AS query_stats GROUP BY query_stats.query_hash ORDER BY 1 DESC After completion of the COB, you should look at F_JOB_TIMES for long COB job times. When you have identified the long-running jobs, search the content of the T24 log file &COMO& to match the jQL queries corresponding to the long-running jobs. Compare these jQL queries to the previously identified SQL Server statements to select queries that are good candidates for optimization. Online Processing:
For online processing, identify slow-running operations in the user interface (UI) based on user feedback and try to reproduce them (on a test database system). Use the SQL Server Profiler to capture the corresponding SQL Server queries. After identifying the SQL statements, you should test them and measure the CPU time and I/Os. This can be done using the sys.dm_exec_query_stats dynamic management view and the query described in the previous section. If you identify multiple long-running queries involved in the online operations, sort the queries based on the average CPU time, and consider the number of executions of each query. Optimizing a query which is executed very often is more important than optimizing queries executed only few times. 2. Consider optimizations. After identifying slow-running queries that are good candidates for optimization, consider the following: If there are queries on tables STMT_ENTRY, CATEG_ENTRY, or RE_CONSOLSPEC_ENTRY using a where clause that is different from RECID = <value>, then these are most likely jQL SELECT queries in a custom-developed code. There should generally be no queries on these tables with a search condition using any fields (other than the RECID). These are the biggest T24 tables, containing huge number of records, and there is usually heavy activity on these tables. The application logic and the jQL SELECT queries on these tables should be reconsidered and, if possible, rewritten. For example, instead of using STMT_ENTRY, you can use the table ACCT_ENT_TODAY in many cases. ACCT_ENT_TODAY is much smaller in size; the
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server 32
key is the account number, and the rows in the table contain the key of the entries that have been made into that account during the current business day. If a query has a search condition on a single-value field, then consider scalar promotion (see step 3). An example of this type of query is: SELECT t.RECID,t.XMLRECORD FROM F_HOLD_CONTROL t WHERE t.XMLRECORD.exist(N'/row/c2[.="NEW.LOAN.REPORT"]') = 1 If a query has a search condition on a specific multi-valued field (specific local reference field), then consider scalar promotion (see step 3). For example, the following query uses specifically the eighth value of a multi-value field in the search condition: SELECT t.RECID,t.XMLRECORD FROM FBNK_CARD_ISSUE t WHERE t.XMLRECORD.exist(N'/row/c14[@m="8"][.=12345]') = 1 Consider the number of promoted columns and indexes per table. Indexes improve the performance of select queries, but also increase the time that is required to perform modifications (i.e., inserts, updates, deletes) to the underlying table. Too many indexes may degrade the overall performance. As a general rule, you should avoid creating more than seven indexes on a table. Do not create XML indexes on T24 XMLRECORD fields. The impact on transaction latency is too high, and the benefit in query performance is usually not significant.
3. Use scalar promotion. A single-value field (or even a specific value of a multi-valued field) that is part of the XMLRECORD can be promoted as computed column of the table and be used in relational search conditions. Further, a relational index can be created on the computed column to improve the query performance. The minimum required T24 version for scalar promotion is R10 SP5. The detailed steps to promote a single-value XML field are as follows: 1.) Create a persisted computed column for the specific field. Create a user-defined function that evaluates the value of the field. The return value of the function should be a single scalar value. Using this function, the computed column should be added to the table and persisted.
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
33
Following are two examples of scalar promotion: -- example 1 -- scalar promotion of single valued field CREATE FUNCTION udf_HOLD_CONTROL_C2(@xmlrecord XML) RETURNS nvarchar(35) WITH SCHEMABINDING BEGIN RETURN @xmlrecord.value('(/row/c2/text())[1]', 'nvarchar(35)') END ALTER TABLE F_HOLD_CONTROL ADD C2 AS dbo.udf_HOLD_CONTROL_C2(XMLRECORD) PERSISTED -- example 2 -- scalar promotion of specific multi-valued field CREATE FUNCTION udf_CARD_ISSUE_CUSTOMER(@xmlrecord XML) RETURNS integer WITH SCHEMABINDING BEGIN RETURN @xmlrecord.value('(/row/c14[@m="8"]/text())[1]', 'integer') END ALTER TABLE FBNK_CARD_ISSUE ADD C14_8 AS dbo.udf_CARD_ISSUE_CUSTOMER(XMLRECORD) PERSISTED
2.) Create non-clustered index on the computed column.
After creating the persisted computed column, create an index for this column: -- example 1 CREATE INDEX ix_HOLD_CONTROL_C2 ON F_HOLD_CONTROL(C2) -- example 2 CREATE INDEX ix_CARD_ISSUE_CUSTOMER ON FBNK_CARD_ISSUE(C14_8) Once there is a promoted column for a specific field in the table, T24 automatically translates the queries to use that column relational in the where clause if there is a search condition on that field.
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
34
4. Verify optimizations. Verify that the changes are successful and measure the impact of the optimizations. For scalar promotion (promoted and indexed fields): Verify the query translation. Without scalar promotion, T24 uses a query syntax such as: SELECT t.RECID,t.XMLRECORD FROM FBNK_CARD_ISSUE t WHERE t.XMLRECORD.exist(N'/row/c14[@m="8"][.=12345]') = 1 The execution of this query usually uses a table scan to retrieve the results. After promoting the field c2, T24 should translate the query as: SELECT t.RECID,t.XMLRECORD FROM FBNK_CARD_ISSUE t WHERE c14_8 = 12345 In this case, index lookup on ix_CARD_ISSUE_CUSTOMER is used. Prove that the index is used by reproducing the query and verifying the actual execution plan. You can run the query in SQL Server Management Studio and activate the icon Include Actual Execution Plan on the SQL Editor toolbar. Alternatively, you can use the SET STATISTICS PROFILE ON statement to display execution plan information. Verify the performance of the query has improved. After using the application for a period of time (e.g., couple of hours or days) or after running COB, use the sys.dm_db_index_usage_stats dynamic management view to verify the index usage. Consider the ratio between index reads and index writes, keeping in mind that an index usually improves the performance for read operations but slows down modifications (i.e., inserts, updates, deletes) at the same time. You can use the queries described in section Standard T24 Queries for this purpose. You should monitor the index usage statistics on a regular basis. For jQL queries changed in the application: Execute the jQL query on the jsh> interface, and verify the performance. Additionally, you should measure the CPU time and I/O operations on the SQL Server. You can use the query described in the Close-of-Business section of the document for this purpose.
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server 35
5. Use T24 logical optimization. If you are not able to achieve sufficient performance improvement to specific queries or COB jobs by applying steps 14 above, then consider using concat files, tables used for application-specific purposes in the application layer. In some cases, concat files can also be used as a logical index in T24.
Predicts
Need for expansion of files or storage. Need to backup transaction log more often. Additional processors. Storage configuration too slow. May indicate a large amount of disk fragmentation, slow disks, or disk failures. (Should be 10 ms or less.) Need to spread data across more LUNs. Need for additional memory. Need for increased network capacity or segmentation.
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
37
It is recommended to schedule the full backup to run after the COB processing and after any index maintenance tasks (e.g., index reorganize or index rebuild). If you implement peer-to-peer replication, you must also implement a backup schedule for the target database(s). The distributor database operates in simple recovery mode, so it does not need to be backed up.
To decide which defragmentation method to use, analyze the index to determine the degree of fragmentation. By using the DMF sys.dm_db_index_physical_stats, you can detect fragmentation in a specific index, all indexes on a table or indexed view, all indexes in a database, or all indexes in all databases. Using this function, you have access to the fragmentation levels available in defined columns at any given time. Table 6 shows some of the information returned by the sys.dm_db_index_physical_stats function. Column avg_fragmentation_in_percent fragment_count avg_fragment_size_in_pages page_count Description The percent of logical fragmentation (out-of-order pages in the index). The number of fragments (physically consecutive leaf pages) in the index. Average number of pages in one fragment in an index. The total number of index or data pages.
You can then use Table 7 as a guide to determine the best method to correct the fragmentation. avg_fragmentation_in_percent > 10 percent and < = 80 percent > 80 percent Corrective statement ALTER INDEX REORGANIZE ALTER INDEX REBUILD
Note that very low levels of fragmentation (less than 10%) should not be addressed by either of these commands because the benefit from removing such a small amount of fragmentation is almost always vastly outweighed by the cost of reorganizing or rebuilding the index. The white paper Microsoft SQL Server 2000 Index Defragmentation Best Practices can provide additional guidance. 7.3.1 Reorganize Indexes Reorganize an index when the index is not heavily fragmented. Use the ALTER INDEX statement with the REORGANIZE clause (replaces the DBCC INDEXDEFRAG statement in Microsoft SQL Server 2000). To reorganize a single partition of a partitioned index, use the PARTITION clause of ALTER INDEX.
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
39
The reorganize process uses minimal system resources. Also, reorganizing is automatically performed online. The process does not hold long-term blocking locks; therefore, it does not block running queries or updates.
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
40
For more information on this feature see the links below: New features in SQL Server 2012 setup: Product Updates in SQL Server 2012 Installation http://msdn.microsoft.com/en-us/library/hh231670(v=SQL.110).aspx Product Updates in SQL Server 2012 Installation http://msdn.microsoft.com/en-us/library/hh231670.aspx
implement their own update services, defining a specific strategy and installing the updates on a test server to verify their impact before installing them in production. For more information, see Windows Server Update Services. Security patching is an important subset of updates. You can find information about security patching in Ten Principals of Microsoft Patch Management. Also see the Microsoft Support article Guidelines for choosing antivirus software to run on the computers that are running SQL Server.
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
42
8 Summary
TEMENOS T24, coupled with Microsoft SQL Server database software, provides financial institutions with a complete banking solution. Using the best practices described in this white paper can help optimize the performance of the database tier. The links in the section that follows provide even more information. For the latest benchmarking results on SQL Server 2012, see the link below: TEMENOS T24 and SQL Server 2012 running on Intel Xeon processor-based servers set a new world record in the latest high-water benchmarking tests Following is a list of technical articles that can help you learn more about specific Windows and SQL Server topics: Windows Configuration:
Receive-Side Scaling Introduction to Receive-Side Scaling Receive-Side Scaling Enhancements in Windows Server 2008 Desktop Heap Overview Desktop Heap, part 2
Storage Configuration: Predeployment I/O Best Practices Storage Top 10 Best Practices DiskPart Disk Partition Alignment Best Practices for SQL Server
SQL Server Configuration: Isolation Levels in the Database Engine Optimizing tempdb Performance Configure Server Startup Options Recovery Interval Option Fill Factor How to reduce paging of buffer pool memory in the 64-bit version of SQL Server
High Availability and Disaster Recovery: A deployment reference architecture and guidance for implementing an HADR solution for TEMENOS T24 running on the Microsoft Application Platform Microsoft SQL Server AlwaysOn Solutions Guide for High Availability and Disaster Recovery SQL Server 2012 AlwaysOn High Availability and Disaster Recovery Design Patterns Introducing SQL Server 2012 AlwaysOn
43
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
Overview of AlwaysOn Availability Groups High Availability with SQL Server 2008 Failover Clustering in Windows Server 2008 R2 SQL Server 2008 Failover Clustering Database Mirroring Overview Synchronous Database Mirroring (High-Safety Mode) Database Mirroring Best Practices and Performance Considerations Log Shipping Overview Transactional Replication Overview Best Practices for Replication Administration Replication Security Best Practices Peer-to-Peer Transactional Replication
Database Maintenance Considerations for Backing Up and Restoring System Databases Tuning the Performance of Backup Compression in SQL Server 2008 Best Practices for Recovering a Database to a Specific Recovery Point Statistics Used by the Query Optimizer in Microsoft SQL Server 2008 Reorganizing and Rebuilding Indexes SQL Server 2008 R2 - Application and Multi-Server Management SQL Server 2008 failover cluster rolling patch and service pack process Update Management TechCenter Troubleshooting Performance Problems in SQL Server 2008
See the SQL Server Best Practices portal for technical white papers, the SQL Server Best Practices Toolbox, Top 10 Lists, and other resources.
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
44
It is important to create a maintenance plan to back up the Temenos database. The following step-by-step instructions guide you through the process. 1. Open SQL Server Management Studio, expand the Management node, and then expand the Management Plans node. 2. Right-click Maintenance Plans, click Maintenance Plan Wizard, and then type an appropriate name for this Temenos application database backup plan.
3. Click Separate schedules for each task, because you will later select both full and transaction log backups. Click Next.
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
45
4. On the Select Maintenance Tasks dialog box, select Clean Up History, Back Up Database (Full), Back Up Database (Transaction Log), and Maintenance Cleanup Task. Click Next twice.
5. As you complete the Maintenance Plan Wizard, you should select options based on the needs of your organization. The following figures show examples of options you may want to select. a. On the Define History Cleanup Task dialog box, select the historical data you want to delete and the schedule. Click Next.
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
46
b. On the Define Back Up Database (Full) Task dialog box, click on These databases, and select the T24 user database in the Database(s) box. Click OK.
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
47
c. On the Job Schedule Properties dialog box, select the frequency and duration of full backups. It is recommended to schedule the full backup to be executed, after the COB processing. Click OK.
d. On the Define Back Up Database (Full) Task dialog box, specify information about the full backup. Click Next.
e. On the Job Schedule Properties dialog box, select the frequency and duration of transaction log backups. Click OK.
Figure 9. Options to set the frequency and duration of transaction log backups.
f.
On the Define Back Up Database (Transaction Log) Task dialog box, configure the transaction log backup. Click Next.
g. On the Define Maintenance Cleanup Task dialog box, configure the cleanup tasks. Click Next.
h. On the Select Report Options dialog box, select whether to write the report to text file or send the report through email. Click Next. 6. Review your selections, and then click Finish.
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
50
3. On the Select Plan Properties dialog box, click Single schedule for the entire plan or no schedule, and click Change to select the frequency and duration of transaction log backups. Click Next.
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
51
4. On the Select Maintenance Tasks dialog box, select Clean Up History, Back Up Database (Full), and Maintenance Cleanup Task. Click Next.
5. On the Select Maintenance Task Order dialog box, make sure that the tasks appear in the order shown in Figure 15. Click Next.
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
52
6. As you complete the Maintenance Plan Wizard, you should select options based on the needs of your organization. The following figures show examples of options you may want to select. a. On the Define Back Up Database (Full) Task dialog box, select System databases. Click OK.
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
53
b. Define the full backup task by selecting from the various options. Click Next.
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
54
c. Define the maintenance cleanup task by selecting from the various options. Click Next.
d. Define the history cleanup task by selecting from the various options. Click Next.
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
55
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
56
3. On the Select Plan Properties dialog box, type an appropriate name for this statistics update plan, and click Single schedule for the entire plan or no schedule. Click Change to select the frequency and duration of statistics updates.
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
57
4. Based on your organizations needs, select the frequency and duration of the statistics updates on the Job Schedule Properties Update Temenos Statistics (FullScan) dialog box. Click OK, and then click Next.
5. On the Select Maintenance Tasks dialog box, select only Update Statistics. Click Next.
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
58
6. On the Select Maintenance Task Order dialog box, make no changes. Click Next.
7. On the Define Update Statistics Task dialog box, click on These databases, and select the T24 user database in the Database(s) box. Click OK.
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
59
8. On the Define Update Statistics Task dialog box, select Tables and views in the Object box. Under Scan type, click Full scan. Click Next.
9. On the Select Report Options dialog box, select report options based on your organizations needs. Click Next.
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
60
DECLARE partitions CURSOR FOR SELECT * FROM #work_to_do; -- Open the cursor. OPEN partitions; -- Loop through the partitions. WHILE (1=1) BEGIN; FETCH NEXT FROM partitions INTO @objectid, @indexid, @partitionnum, @frag; IF @@FETCH_STATUS < 0 BREAK; SELECT @objectname = QUOTENAME(o.name), @schemaname = QUOTENAME(s.name) FROM sys.objects AS o JOIN sys.schemas as s ON s.schema_id = o.schema_id WHERE o.object_id = @objectid; SELECT @indexname = QUOTENAME(name) FROM sys.indexes WHERE object_id = @objectid AND index_id = @indexid; SELECT @partitioncount = count (*) FROM sys.partitions WHERE object_id = @objectid AND index_id = @indexid; SET @command = N'ALTER INDEX ' + @indexname + N' ON ' + @schemaname + N'.' + @objectname + N' REORGANIZE'; IF @partitioncount > 1 SET @command = @command + N' PARTITION=' + CAST(@partitionnum AS nvarchar(10)); EXEC (@command); PRINT N'Executed: ' + @command; END; -- Close and deallocate the cursor. CLOSE partitions; DEALLOCATE partitions; -- Drop the temporary table. DROP TABLE #work_to_do; GO -- rebuild indexes -- Ensure that a USE <databasename> statement has been executed first. SET NOCOUNT ON; DECLARE @objectid int; DECLARE @indexid int; DECLARE @partitioncount bigint;
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server 62
@schemaname nvarchar(130); @objectname nvarchar(130); @indexname nvarchar(130); @partitionnum bigint; @partitions bigint; @frag float; @command nvarchar(4000);
-- Conditionally select tables and indexes from the sys.dm_db_index_physical_stats function -- and convert object and index IDs to names. SELECT object_id AS objectid, index_id AS indexid, partition_number AS partitionnum, avg_fragmentation_in_percent AS frag INTO #work_to_do FROM sys.dm_db_index_physical_stats (DB_ID(), NULL, NULL , NULL, 'LIMITED') WHERE avg_fragmentation_in_percent > 80.0 AND index_id > 0; -- Declare the cursor for the list of partitions to be processed. DECLARE partitions CURSOR FOR SELECT * FROM #work_to_do; -- Open the cursor. OPEN partitions; -- Loop through the partitions. WHILE (1=1) BEGIN; FETCH NEXT FROM partitions INTO @objectid, @indexid, @partitionnum, @frag; IF @@FETCH_STATUS < 0 BREAK; SELECT @objectname = QUOTENAME(o.name), @schemaname = QUOTENAME(s.name) FROM sys.objects AS o JOIN sys.schemas as s ON s.schema_id = o.schema_id WHERE o.object_id = @objectid; SELECT @indexname = QUOTENAME(name) FROM sys.indexes WHERE object_id = @objectid AND index_id = @indexid; SELECT @partitioncount = count (*) FROM sys.partitions WHERE object_id = @objectid AND index_id = @indexid; SET @command = N'ALTER INDEX ' + @indexname + N' ON ' + @schemaname + N'.' + @objectname + N' REBUILD'; IF @partitioncount > 1
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
63
SET @command = @command + N' PARTITION=' + CAST(@partitionnum AS nvarchar(10)); EXEC (@command); PRINT N'Executed: ' + @command; END; -- Close and deallocate the cursor. CLOSE partitions; DEALLOCATE partitions; -- Drop the temporary table. DROP TABLE #work_to_do; GO You should generally create a nightly job for index reorganization (using the 1 sample script) and a weekly job for index rebuild (based on the 2 sample script). Ideally, the index maintenance tasks (reorganization or rebuild) should be performed after COB processing and before a full backup is executed. To create a nightly or weekly job to perform the reorganizing or rebuilding, perform the following steps: 1. Right-click Jobs, and click New Job.
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
64
2. Name the job so it can be easily identified as the reorganization or rebuilding index.
3. Click Steps, and then click New. Enter the step name (e.g., Reorg or Rebuild), select the type Transact-SQL script (T-SQL), specify the T24 database name, and copy the script sample into the Command field. To make sure that there are no errors in the T-SQL script, click Parse. Click OK.
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
65
4. Select the Advanced page, change the On Success action to Quit the job reporting success, and click OK.
5. On the New Job Schedule dialog box, schedule the job to run nightly or weekly, depending on the needs of your organization. Click OK.
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
66
6. On the Job Properties Temenos Index Reorg/Rebuild dialog box, configure the notifications for the job based on your organizations needs. (Note that database email must be configured correctly for the notifications to work.) Click OK.
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
67
Best Practice Guidance for Running TEMENOS T24 on Microsoft SQL Server and Windows Server
68