SQL Tips and Best Practices for Meditechs DR

2011 MUSE International Tuesday Education Session May 31st

Presenter: Jamie Gerardo

Todays Agenda

Operational Key Topics

Background Jobs DR Errors Auditing/Validation Report Development Standards Finding Data Writing Efficient Code T-SQL Tips

Development Key Topics

Operational

Background jobs DR Errors Audits / Validation Server Maintenance

Monitor space Backing up and testing backup restore Preventative server maintenance Logging Tasks Following recommendations and guidelines

Meditech Website

MT DR Background Jobs Client Server

http://www.meditech.com/

* Graph from Meditech website

MT DR Background Jobs Magic

* Graph from Meditech website

MT DR Background Jobs 6.0

* Graph from Meditech website

Monitoring/Maintaining Background Jobs

You can create additional bkg jobs (depending on the ring release) Do not reboot server without stopping jobs (completely) Background job can be running although a table is not updating Review Meditech Recommendations and website

Additional Background Jobs

Why create additional background jobs? (Current job is taking too long to update the DR tables.) 1. You can split up larger applications into multiple jobs. 2. You can segment off particular tables that are used for real time reporting. 3. You can split off applications that take longer. * Additional jobs depends on the ring release version

Rebooting the DR Server

Do not reboot the server without stopping the background jobs! What happens if you do?
a. b. c.

Table sequences will get off track Data will be skipped Table ILs may need to be done to get data over to the DR.

You can stop the background jobs by: a. Calling Meditech or b. Stopping the jobs through the DR application

DR Errors

DR Parameters

Activity Index Error Log It is not recommended to report every error Provide as much information as possible

Monitor Error Summary

Prioritize DR Errors Logged See Meditech Website

DR Parameters

Activity Days (Typically defaults to 710 days): Determines how long activity is kept around in an index to be transferred to DR.

Can be set at the table level by Meditech for research/problem issues

Error Log Days (Default is 3 days): This should be at least 710 days.
C/S 6.5 but all Meditech Versions have this option in the DR Parameters (it just looks different)

DR Errors
o Only report DR Errors that reference tables you use or will use in the future. o Only report the latest version of the same or similar error. o Report Skipped activity as soon as possible, this data will eventually be removed from the index.

Key items when reviewing errors

Example not to report:

DATE: 20110414 TIME: 0001 TABLE: LACSAX1 LabSpecAlerts Sequence: SOURCEID: OSC MIS DB: OSC PROGRAM: DrXferBkg[K] APPL DB: SCH.SSM DPM: LAB.C.SPEC TYPE: PM Program missing ELEMENT: EXPECTED: ERROR VALUE: ROW: MESSAGE: SCH.SSM's LabSpecAlerts expander is missing

Example to report:
DATE: 20110221 TIME: 1143 TABLE: VISITCLI AdmVisitClinicalQueries Sequence: 4391 SOURCEID: OSC MIS DB: OSC PROGRAM: DrXferBkg[T:ADMQUERY] APPL DB: ADM.OSC DPM: ADM.PAT TYPE: S Socket ELEMENT: SQL EXPECTED: ERROR VALUE: ROW: 873550 MIO5UD Y 1 MESSAGE: Primary key VisitD is missing

This is typically not a table that will ever be used so dont report the error

DR Error Messages

Typical Messages you can ignore:

Always confirm with your applications specialist. These are some general guidelines.

MESSAGE: Unable to open prefixes to ADM.OSH database [1] MESSAGE: Missing subscript at position 0 for OE.STAT.transcription.stats MESSAGE: Violation of PRIMARY KEY constraint 'mtpk_ep551386'. Cannot insert duplicate key in object 'DMisUserC~. Check Server Error Log for more info SQL Non Fatal Error MESSAGE: Line 1: Incorrect syntax near '{'.. Check Server Error Log for more info SQL NonFatal Error MESSAGE: Unable to find segment from physical [No prefix for segment [Physical.] and nil in @.db] MESSAGE: No pointer for include children of AP.AHS.TmpOpIC table MESSAGE: Syntax error converting datetime from character string.. Check Server Error Log for more info SQL Non Fatal Error <EP550409> MESSAGE: CON DR Server not responding to ECB command - ACK 98 SQL Fatal Error <VISITCLI> DR Server not responding to ECB command - ACK 98

Messages you want to report:

AdmVitalSigns Activity skipped, not in Pending status Primary key CheckID is missing Missing subscript at position 1 for MRI.DRC.insure.order Unknown error [SYS] - Segment A is down, unable to start DrXferBkg[T:ADMQUERY]

MESSAGE: MESSAGE: MESSAGE: MESSAGE:

Logging DR Errors
Include ring release version in description along with table name and indication of the issue. Example Descriptions:

5.61 AdmVisitQueries Primary Key Missing VisitID 5.65 MRI.DRC.insure.order - Missing subscript Pos 1 5.61.1 OeOrderQuery - Expander is missing

Issue: DR #6133704 (Mar 2, 2009) Status: Open Priority: Routine Priority Lists: #3 on DR Priority List Description: 5.61.13 LabSpecimenTests - Column Discrepancies Request Type: Problem Customer Contact: James Durbin (617-555-1212) Issue Notification: james@whatsuphosp.org (All Edits) Module Notifications: ACME,SMH (jmcdonald@acmeware.com) - MEDITECH Edits Set a Priority List on

your tasks to get the quickest resolution

Auditing / Validation

Blue Elm DR Auditor or other tool? Audit tables every 1-2 months Re-Analyze (and within 2 days) Research audits and provide MT recent examples (within the activity Index) Application purge parameters Manual validation Matching NPR / DR Reports Meditech Auditing Tool CS 5.65, Magic 5.64.17(?), 6.0 Is in its infancy with development and testing.

Blue Elm DR Auditor

More DR Auditor Tips Start with auditing tables you use Audit like tables Only a few tables at a time Rebuild Schemas after a new Meditech ring release Recreate audits after an update Delete old audits

DR Auditor A step further

You can query the audits to find specific examples or the most recent examples. You can also confirm that the data is still missing.

Meditech tables and corresponding DRAuditor table names:

Reporting discrepancies to Meditech

Meditech would prefer audits that are not older than a week. Once the data has been purged from the Activity index it makes it difficult to troubleshoot this issue. Typically log 1 table per Task Task descriptions should include the Table Name along with Missing Row or Column Discrepancy Ex: 5.65 AdmVisits Missing rows Attach the HTML Report in your task or provider recent examples from your sql query.

DR Maintenance

Overall Server Maintenance

Monitor disk space Database Backups DBCCs on a monthly basis

Database Maintenance

Integrity checks (using wizard)

Analyze and repair index fragmentation on a monthly basis

Details of Maintenance

Monitor disk space

Database space

How much space is the database using Always have 25% of total disk space as free (livedb E: drive)

Drive space

Index defragmenting

Dont use the database maintenance wizard

Wizard will drop and recreate indexes, which you dont want to do.

Defragging Indexes
DETERMINING FRAGMENTATION AT THE TABLE AND INDEX LEVEL
Logical Scan Fragmentation lower is better, anything above 30-35% would be a candidate to defragment. Avg. Page Density higher is better, ideally 90-95%. The fuller each page is, the denser (less fragmented) the data are.

DEFRAG INDEX IF NECESSARY DBCC INDEXDEFRAG ( DBNAME, TABLENAME, INDEXNAME)

SQL Development

Finding Data

How can you find data in the DR? Database, Stored Procedures and Tables How to make your code run fast and more efficiently Helpful code tips

Organization

Efficiency

T-SQL TIPS

Finding Data

If you know the NPR structure then finding data will be much easier In general you can think of a Detail Segment as a table Tools Meditech Website SysDrTables Shift F9 (C/S) and Shift F8 for Magic DR Application Menu

1. 2. 3.

4.

Meditech DR Home Page

Overview of the data repository

* Table Structure

Customer Responsibilities

Viewing the Table Structure http://www.meditech.com/

Useful Table information

Meditech - Table Structure

1. Shows the equivalent NPR Parent/Child relationships 2. Interactive Primary Keys that displays other tables with foreign keys

SysDr Tables in livedb

---------------------------------------------------------------------------- A general search by DR Field Name -- '%Comment%' is a wildcard search for any field with Comment -- You can modify the name as needed for your search ---------------------------------------------------------------------------

SELECT T.Name, C.* FROM livedb.dbo.SysDrColumns C INNER JOIN livedb.dbo.SysDrTables T ON C.TableID = T.TableID WHERE C.Name like '%Comment%' order by 1
-------------------------------------------- A search by specific NPR field -------------------------------------------

Examples of how to query Sys DR tables

-------------------------------------------------------- A search by Dpm and Detail Segment -------------------------------------------------------

SELECT T.Name, C.* FROM livedb.dbo.SysDrColumns C INNER JOIN livedb.dbo.SysDrTables T ON C.TableID = T.TableID WHERE C.NprElement = 'BAR.PAT.account' order by 1

SELECT T.Name, C.* FROM livedb.dbo.SysDrColumns C INNER JOIN livedb.dbo.SysDrTables T ON C.TableID = T.TableID WHERE C.NprDpm = 'BAR.PAT' AND C.NprSegment = 'main' order by 1

SysDr Result Example

SELECT T.Name, C.* FROM livedb.dbo.SysDrColumns C INNER JOIN livedb.dbo.SysDrTables T ON C.TableID = T.TableID WHERE C.Name like '%Comment%' order by 1

Shows table name, column, data type along with the DPM, NprSegment and NprElement

Shift F9 (C/S) and Shift F8 for Magic

Table and Field Inquiry on the DR Application Menu

I rarely use these tools on the DR Menu They can be helpful but I find the previously covered options to be the best.

Magic 5.61 Screen

Organization - Database

1. Dont save stored procedures and/or tables in livedb. 2. Create a database to keep your stored procedures, views and tables. Make sure Recovery Mode is set to simple You can set up the database files similar to livedb 3. You will want to include the database you create in your backup plan.

What is a stored procedure?

A stored procedure is a saved set of code on the sql server that allows you to run:
EXEC spBarAccountsByAccountType
Rather than SELECT BVFD.AccountType, BV.PostalCode, BV.AccountNumber FROM livedb.dbo.BarVisits BV INNER JOIN livedb.dbo.BarVisitFinancialData BVFD ON BV.SourceID = BVFD.SourceID AND BV.BillingID = BVFD.BillingID

Organize your Stored Procedures

Name your stored procedures so that you can easily locate them. Ex: spErDepartVolumesByHour Ex: spAdmRevisitsTable Re-name stored procs no longer in use. Ex: x_spErDepartVolumesByHour Use Header information to describe a stored procedures use. Only save useable code as a stored procedure. Save test code and research code as a text file or label appropriately.

Stored Procedures
To save a stored procedure you CREATE PROC. This saves the stored procedure on the server/database youve selected

Once created you can Modify and View Dependencies

Stored Procedure Example Header

Saving code as a text file

Items to save as a text file 1. Research queries 2. Testing queries

Default Location this can be changed

Creating Efficiencies

Indexing

Estimated Execution Plan

Joining on Primary Keys Creating Datamarts (tables) Using temp tables vs. sub queries Functions Using With (NoLock)

Table Indexing

Clustered

Every Meditech table has a clustered index, which is the physical order of the table by primary key(s). Never modify or delete There is only 1 per table A non-clustered index creates a separate 'internal' table that stores only the selected key values of the table in order. Each 'record' in this index contains the key value from one record in the table along with a pointer to either the data record itself or to a value in the clustered index. A max of 249 non-clustered indexes per table You will want to be selective with indexes you build

Non-Clustered

Typical Indexes used

BarChargeTransactions

BarCollectionTransactions

ixServiceDateTime ixTransactionProcedureID ixProcedureChargeDept

ixVisitID ixAdmitDateTime ixServiceDateTime ixServiceDateTime ixStatus

ixReceiptDateTime ixInsuranceID
ixAdmitDateTime ixVisitID ixAccountNumber (Field4) ixUnitNumber (Field3)

AdmittingData

BarVisits

AbstractData

AdmVisits

DMisUserStatisticsDetail

Example of Execution Plan

Display Estimated Execution Plan

Creating an Index

Joining on Primary Keys

Joining on the primary keys will make your report run faster. Omitting the primary key will not only slow down your query but many times your output will not be correct. Each application has a unique identifier (primary key) that will allow you to join to other applications.

Primary Key Example

SELECT AV.VisitID, AV.LocationID, AV.[Name], AD.AdmitDateTime FROM livedb.dbo.AdmVisits AV WITH (NOLOCK) Even if there is only one SourceID, you will want to use the Clustered Index for faster processing.

INNER JOIN livedb.dbo.AdmittingData AD WITH (NOLOCK) ON AV.SourceID = AD.SourceID AND AV.VisitID = AD.VisitID

WHERE Status='ADM IN

SQL Design Query Editor

You can use this tool but you still need to join on the primary keys. This tool will not automatically do that for you.

Primary Key Example 2

SELECT AV.VisitID, AV.LocationID, AV.[Name], AD.AdmitDateTime, BV.PrimaryInsuranceID, BVFD.Balance FROM livedb.dbo.AdmVisits AV INNER JOIN livedb.dbo.AdmittingData AD ON AV.SourceID = AD.SourceID AND AV.VisitID = AD.VisitID LEFT JOIN livedb.dbo.BarVisits BV ON AV.SourceID = BV.SourceID AND AV.VisitID = BV.VisitID LEFT JOIN livedb.dbo.BarVisitFinancialData BVFD ON BV.SourceID = BVFD.SourceID AND BV.BillingID = BVFD.BillingID where Status='ADM IN'

VisitID is in a number of tables but youll only want to use it to join to a parent type table BarVisits, AbstractData, Lab Specimens, SchAppointments

Use the applications primary key (unique identifier) within the application tables. Adm - VisitID Bar BillingID Abs AbstractID Oe OrderID Sch - AppointmentID

Creating Datamarts / Tables

When do you create a new table?

Processing time for a report may dictate the need for a table. Combining data from various applications for faster queries. Combining data and creating custom fields for reporting needs. Building Report Models

Datamart Example
Issue: You have a very large query for nursing with lots of data from various assessments and with pharmacy data. This query takes over 10 minutes to run and will take longer if additional filters are added. Solution: 1. Create a table with all the data you need for your reports 2. Set up a SQL Server Agent Job to populate the data on a nightly basis 3. Create a set of store procedures to use for your report output.

Datamart Example - Continued

NursingStoredProcedure
This table gets updated daily. A SQL Server Job is set up to populate on a nightly basis with new data.

Additional Stored procedures are written to use for reporting purposes NursingLocationSummary

Creating Tables

You can write sql code to insert data into a table by first creating the table then inserting the data.
After the first table insert typically you will: TRUNCATE TABLE PatientBirthDates INSERT PatientBirthDates

1. CREATE TABLE PatientBirthDates ( Name varchar(35) not null, AccountNumber varchar(15) not null, DOB datetime null) 2. INSERT PatientBirthDates SELECT Name, AccountNumber, BirthDateTime FROM AdmVisits

You can insert directly into a table as below. The data types and fields will populate based on the data returned.
The INTO only works the first time.

Select * INTO tbNewTable from DMisUsers

Either way of creating the table is fine. Just note that you may need alter data types and write code to add Primary keys. Typically modifications will need to be made when the table is created by an INTO. Youll also need to Truncate Table where necessary. ** Will demonstrate

User Defined Function

What is a User Defined Function?
Functions are subroutines used to encapsulate frequently performed logic. Any code that must perform the logic incorporated in a function can call the function rather than having to repeat all of the function logic.

Built-in functions operate as defined in the Transact-SQL Reference and cannot be modified. The functions can be referenced only in Transact-SQL statements using the syntax defined in the Transact-SQL Reference.

Examples AVG, SUM, COUNT,DATEADD, DATEDIFF,NAME, ETC..

User-defined functions allow you to define your own Transact-SQL functions using the CREATE FUNCTION statement. For more information about these built-in functions

This is what well be doing in class.

FUNCTIONS

Useful Acmeware functions

fxAge
fxProperCase fxConvertGramsToLbs fxMeditechTimeIDToDateTime fxIsNumeric

Function - fx.Age

Function - fxProperCase
Selecting the data: SELECT [Name], dbo.fxProperCase(Name) AS ProperName, ProviderGroupName, dbo.fxProperCase(ProviderGroupName)AS ProperGroupName FROM livedb.dbo.DMisProvider
This takes any value and converts it to upper and lower case. Works great for creating consistencies in your reports.

Function - fxConvertGramsToLbs

Code could possibly need to be tweaked based on the Grams data type.

Function - fxMeditechTimeIDToDateTime

There are various fields throughout Meditech that are in seconds. This function will calculate the date for you.

Function - fxIsNumeric
There will be times where you need to ensure that a field strictly has numeric values. Using the System IsNumeric does not always work.

SQL Tips
Dates Dropping Temp Tables Documenting in code Creating a #Dates and populating Using WITH (NOLOCK)

TSQL Tips - Dates

DateDiff

Calculates the difference between two dates Adds a period of time to a date (or subtracts)

DateAdd

Years, Months, Days, Hours, Minutes or Seconds

SQL Date Default

5/26/11 defaults to 5/26/11 00:00:00

DateDiff

DateDiff calculations have countless uses in your code development.

* Be aware of the time values and rounding specifically with Days

DateDiff Calculations for Elapsed Time

Be aware of using standard sql functions, they may not work the way you expect.

DateAdd Calculations
First Day of Current Month: SELECT DATEADD(MM, DATEDIFF(MM,0,GETDATE()), 0) Explanation: 1. 0 = 19000101 2. The DATEDIFF calculates the number of months since 19000101 3. The DATEADD adds the same number of months back to 19000101 to give you the beginning of the current month

Last Day of Current Month: SELECT DATEADD(SS,-1,DATEADD(MM,DATEDIFF(MM,0,GETDATE())+1,0)) Explanation: 1. DATEDIFF(MM,0,GETDATE())+1 - calculates the number of months from the current date since 19000101 and adds 1 2. DATEADD(MM,DATEDIFF(MM,0,GETDATE())+1,0) - adds the above number of months to 19000101 (this will give you the first day of next month) 3. The last DATEADD substracts 1 second to give you the last day of the current month (ie. 9/30/09 23:59:59 ) First Day of Last Month: SELECT DATEADD(MM, DATEDIFF(MM,0,DATEADD(MM,-1,GETDATE())),0) Explanation: 1. DATEADD(MM,-1,GETDATE()) - Subtracts 1 month from current date 2. DATEDIFF(MM,0,DATEADD(MM,-1,GETDATE())) - calculates the number of months since 19000101 3. The DATEADD adds the calculated number of months back to 19000101 to give you the beginning of the previous month

DateAdd Calculations
Last Day of Last Month: SELECT DATEADD(SS,-1,DATEADD(MM,DATEDIFF(MM,0,GETDATE()),0)) Explanation: 1. DATEADD(MM,DATEDIFF(MM,0,GETDATE()),0) - same code as getting the first day of the current month 2. DATEADD substracts 1 second to give you the last day of previous month

First Day of Current Year: SELECT DATEADD(YY,DATEDIFF(YY,0,GETDATE()),0) Explanation: 1. 0 = 19000101 2. The DATEDIFF calculates the number of years since 19000101 3. The DATEADD adds the same number of years back to 19000101 to give you the beginning of the current year 4. This is the same as the month calculations but instead of mm for month you use the yy for year Last Day of Last Year: SELECT DATEADD(SS,-1,DATEADD(YY,DATEDIFF(YY,0,GETDATE()),0)) Explanation: 1. 0 = 19000101 2. The DATEDIFF calculates the number of years since 19000101 3. The DATEADD adds the same number of years back to 19000101 to give you the beginning of the current year 4. The next DATEADD substracts 1 second to reflect the day before just before midnight.

Examples using DateDiff & DateAdd

Understanding how the data functions work will help you right the appropriate code for your particular needs.

SQL Date Defaults

Because SQL defaults to a time of 00:00:00. We code for that with a DateAdd.

Keep this in mind when creating data range parameters so that you include the full last day of the search

Code for Dropping Temp Tables

When using temp tables enter this before each temp table and it will save you a lot of time and hassle with continuously dropping the table.

IF OBJECT_ID('tempdb.dbo.#TableName')IS NOT NULL DROP TABLE #TableName SELECT Fields INTO #TableName FROM MyTables

You can also enter the code at the end of your stored procedure or query to make sure the temp table has been dropped.

Document in your code!!

Documenting through out your code is a huge help with understanding laterjust what you were thinking at the time. Document on any bit of code that is the least bit out of the ordinary. Not only what by why. Notate in each step of your code what you are doing.

Documenting Example:

Creating a #Dates Temp Table

The #Dates table works great for Summaries that need to include every day regardless of data values.

Using WITH (NOLOCK)

When data in a database is read or modified, the database engine uses special types of controls, called locks, to maintain integrity in the database. Locks basically work by making sure database records involved in a transaction cannot be modified by other transactions until the first transaction has committed, ensuring database consistency. The benefit of using WITH (NOLOCK) is that it allows you to keep the database engine from issuing locks against the tables in your queries; this increases concurrency and performance because the database engine does not have to maintain the shared locks involved.

Thank You!

6/3 10:00 Replacement Session - 6.0 Custom Reports with Data Repository and SSRS 2008, Ian Proffer presenting 6/3 11:00 - 372 Basic Meaningful Use Reporting From Data Repository, Glen DAbate presenting 6/3 1:30 - 334 Inspiring Reporting Options in the World of 6.0, Jamie Gerardo presenting