Sybase Interface Processes Configuration Guide

f I D E S S A
T R A D I N G P L A T F O R M
Making better use of trading technology
fidessa Product Infrastructure
Version 5.3
Sybase Interface Processes

Configuration Guide
Project Code:
Issue: 1.10
Status: Released
Issue Date: 2nd July 2001
Author: royalblue financial plc
File: FPI_5_3_SIP_CFG_1_10.doc
The copyright in this work is vested in royalblue financial plc and

the document is issued in confidence for the purpose only for
which it is supplied. It must not be reproduced in whole or in part
or used for tendering or manufacturing purposes except under an
agreement or with the consent in writing of royalblue and then
only on the condition that this notice is included in any such
reproduction. No information as to the contents or subject matter
of this document or any part thereof arising directly or indirectly
therefrom shall be given orally or in writing or communicated in
any manner whatsoever to any third party being an individual firm
or employee thereof without the prior consent in writing of
royalblue financial plc.
Copyright 2001 royalblue financial plc
fidessa is a registered trademark of

royalblue financial plc
Change History
Issue Date Issue Number Description of changes
26 Jan 1998 1.0 First issue
21 Apr 1998 1.1 F2SGen now requires ipcName to be specified on command line
and the config file output needs the Install hostCfg mechanism.
New update regime for FidToSyb.
Modified config items for FidToSyb:
recoveryRewindMins (new, optional)
dbLoginTimeout (new, optional)
dbStaleTimeout (new, optional)
updPollPeriod (meaning altered)
upqMaxBatch (default changed, meaning altered)
pollPeriod (new, optional)
databaseMap (was missing from previous version of

document)
29 Apr 1998 1.2 Added details on OAToDb
6 Jul 1998 1.3 Documented new FidToSyb configuration option flushAtClose
Documented amendment to new FidToSyb update regime
28 Oct 1998 1.4 Updated for infrastructure v4.2.2:
Documented new Sdd options dbServerName,

dbPassword, and replace.
Explained that the default FidToSyb mapping list is not used

when a parameters block is specified.
17 Nov 1998 1.5 Updated for infrastructure v4.2.3 patch A:
New Sdd option modifiedFieldsOnly documented.
21 Jun 1999 1.6 Updated for Infrastructure v4.3.1
New Sdd option upqMaxBatch documented.

Issue Date Issue Number Description of changes
21 Dec 1999 1.7 Updated for infrastructure v5.0.0
Sdd resync mode, status & nack sprocs, and

sendUpdateLastSourceRef option documented
FidToSyb status & stream status sprocs, data synchronisation

status, file based health checking
OA Stats documented in Appendix A.
Corrected documentation of standbyActive config option
31 Mar 2000 1.8 Updated for infrastructure v5.1.2
Added batchDelaySeconds
22 Jan 2001 1.9 Updated for infrastructure v5.3.0
Added new replay and replayAction command line

arguments for FidToSyb.
Added section on FidToSyb replay mode.
FidToSyb, timestampSproc and primaryKeySproc

items now expressions
nd
2 July 2001 1.10 Updated for infrastructure v5.3.1
Added encryptedDbPassword for Sybase password encryption.

Overview Page 5
Contents
1. Overview ........................................................................................ 7
2. Sdd ................................................................................................. 8
2.1 Design ............................................................................................................... 8

2.2 Operation ........................................................................................................ 10
2.3 Automatic Configuration.................................................................................. 12
2.4 Sdd.cfg Detail.................................................................................................. 14
2.5 Environment variables .................................................................................... 21
2.6 Command line Arguments .............................................................................. 21
2.7 OpenAccess Stats .......................................................................................... 21
3. FidToSyb...................................................................................... 23
3.1 Design ............................................................................................................. 23

3.2 Operation ........................................................................................................ 23
3.3 FidToSyb.cfg Detail......................................................................................... 28
3.7 fidessa to Sybase column mapping ................................................................ 37
4. OAToDb........................................................................................ 39
4.1 Design ............................................................................................................. 39

4.2 Transactions ................................................................................................... 39
4.3 Configuration................................................................................................... 40
4.4 OAToDb.cfg Detail .......................................................................................... 40
fidessa Product Infrastructure Copyright 2001 royalblue Issue: 1.10

Version 5.3 Status: Released
Sybase Interface Processes Configuration Guide File: FPI_5_3_SIP_CFG_1_10.doc
Page 6 Overview
5. Appendix A .................................................................................. 43
5.1 Sdd OpenAccess Statistics............................................................................. 43

5.2 FidToSyb OpenAccess Statistics.................................................................... 44
5.3 OAToDb OpenAccess Statistics ..................................................................... 47
6. Appendix B .................................................................................. 49
6.1 Built in subst ( ) substitutions in FidToSyb ...................................................... 49
Issue: 1.10 Copyright 2001 royalblue fidessa Product Infrastructure

Status: Released Version 5.3
File: FPI_5_3_SIP_CFG_1_10.doc Sybase Interface Processes Configuration Guide
Overview Page 7
1. Overview
fidessa products use Sybase as a repository for standing data and reporting data. Static data
contains information that wouldnt normally change during the course of the day such as lists of
instruments and alternate instrument identifiers or user names and passwords. A fidessa
process called Sdd handles the copying of data from Sybase to fidessa.
Reporting data is auditing or statistical information such as trades, executions, positions etc.
FidToSyb performs the copying of data from fidessa to Sybase.
A third fidessa process, OAToDb provides an OpenAccess transaction interface to Sybase stored
procedures.
The static and reporting data may be held in different or the same Sybase databases. The
Sybase tables, stored procedures and triggers for these are generally provided and mainted by
the fidessa Data Architecture (FDA) release.
The Sybase SQL Server may also be accessed from tcl programs using the Sybtcl tcl library
distributed with fidessa. Further details of this library are outside the scope of this document but
full details may be obtained from documents in the third-party area on the fidessa web server.
Available data flows:
tcl program SybTcl
Open Access OAToSyb Sybase data
FidToSyb
fidessa rtd
Sdd

Page 8 Sdd
2. Sdd
2.1 Design
The purpose of Sdd is to maintain the fidessa data with respect to the Sybase data. To do this, it
performs two distinct tasks - download all and updates. The download all task is performed
automatically when the fidessa system is cold started to load the empty fidessa tables. The
updates task is performed thereafter to make changes to fidessa data as changes are made to
Sybase data.
Remote
Procedure
Calls
Sybase SQL Server Fidessa
Static Sdd OA Txn

Handler rtd
Database
Sybase Open Fidessa

Row Access Database
Data Transactions Writes
.
Each fidessa real-time database table has a Sybase stored procedure (or sproc) associated with
it. These sprocs are used for selecting data to download from Sybase to fidessa and are termed
sddsel sprocs. For example, the ALT_COUNTERPARTY_ID table for FTS, may have a sproc
named FTSsddsel_ATL_COUNTERPARY_ID
These sprocs can be run in two ways. They can be run in with an ALL argument to return all
rows from the table or they can be run with a RECORD argument to return a particular row
identified by the key values in the remaining arguments. For example, to obtain a listing of all
rows from the ALT_COUNTERPARTY_ID table, you might run the following sql statement:
FTSsddsel_ALT_COUNTERPARTY_ID ALL

Sdd Page 9
and to list a particular record, you might try:
FTSsddsel_ALT_COUNTERPARTY_ID RECORD, CTLA, VIEW
To download all records to fidessa, Sdd runs the sddsel sproc for each table with the ALL
argument and raises add or update fidessa transactions.
To maintain the fidessa the iTB_table_update table that contains Sybase changes. The
iTB_table_update table is maintained automatically by triggers as follows:
Tables: Triggers: Insert Table

Name and key field
INSTRUMENT values for each deleted,
modified or inserted
row
USER
iTB_table_update
ALT_COUNTERPARTY_ID
ETC
Note: The table names and sproc names are given as examples only, and do not necessarily reflect those
assigned by FDA.
Each table in Sybase has a trigger associated with it. A trigger is a stored procedure or sproc
that is automatically executed when a certain event occurs. These triggers are set to execute
when a row in a table such as STB_INSTRUMENT is added, modified or deleted. The trigger
writes the fidessa real-time database table name, i.e INSTRUMENT, in this case, together with
enough key values to uniquely identify the row that was added, modified or deleted from the
table, to iTB_table_update. The iTB_table_update table contains an auto-incrementing
SEQUENCE_NUM field that increases by one each time a trigger inserts a row. This all means that
iTB_table_update contains a sequenced list of changes that have been made since
iTB_table_update was created.
There is also an sddsel sproc for the iTB_table_update table but it has slightly different
arguments. The first argument identifies a SEQUENCE_NUMBER and the second specifies the
fidessa product in which Sdd is running. This second argument controls which tables updates
are returned to which product. sddsel_TABLE_UPDATE returns all the rows that have a greater
SEQUENCE_NUMBER than the one specified (if any). Sdd then processes the row data and runs
the sddsel sproc with the RECORD option and the key values from each iTB_table_update
row to get the full table row from Sybase. It then raises a TXN_ADD, TXN_REPLACE, or
TXN_MODIFY transaction for the new or modified data. If the sddsel sproc returns no data then
Sdd assumes the iTB_table_update row was written because a delete was done and so
raises a TXN_DELETE transaction to delete the data from fidessa instead. Every OA transaction
sent to fidessa will contain the SEQUENCE_NUM (encoded in the SOURCE_REF OA field) from the
iTB_table_update column in the corresponding row.

Page 10 Sdd
2.2 Operation
As Sdd starts up, it performs the following steps:
1. Validates command line args, reads in config files and logs into Sybase
2. Waits for fidessa database available
3. Sends an OA TXN_GET_LAST_SOURCE_REF message to fidessa
4. Receives a response and saves it as last_sequence_number
5. If doing download_all, then retrieve the last record number in the iTB_table_update
table, using iSP_last_source_ref sproc, and run the sddsel_<table> ALL sprocs
for each table. Then wait for all transactions to be ACKed by fidessa.
6. Enter a loop that runs sddsel_TABLE_UPDATE sproc with the last_sequence_number

(initially) from step 4. If Sdd restarts then this number can be recovered from fidessa (step 3),
and Sdd can continue from where it stopped.
Sdd operates an internal OA message queue. This Queue grows as OA messages are sent by
Sdd and shrinks as OA messages are ACKed or NACKed by fidessa. If messages are sent while
the OA transaction service is down, the messages are stored internal to Sdd and sent when the
service becomes available again. This means that no transactions are lost if there is a comms
outage.
2.2.1 Download Modes

Sdd can download data either in real time as updates occur, or in batch mode when a full
download is required. The default mode of operation is real time. A full download of all the
configured tables can be executed by starting the SDD with the DOWNLOAD_ALL process mode.
For example:
StartProcess FTS_SDD DOWNLOAD_ALL
See the fidessa system Operations Guide for more on Process Modes. Using this mode will
cause SDD to will create new records in the rtd if they are not already present, and overwrite
existing ones. Then Sdd will carry on and update the fidessa database in real-time.
2.2.2 Download Order

The order in which the tables are downloaded during a full download is configurable. This is
useful if data in one table is dependent on another being present. For example, INSTRUMENT
data should be downloaded before INSTRUMENT_PRICE data. An extract from a sample
configuration file is given below:
Sdd
{
.
.
.
databaseMap
{
.

Sdd Page 11
.
.
downloadOrder
{
COUNTERPARTY,
INSTRUMENT,
INSTRUMENT_PRICE
}
.
.
.
}
2.2.3 Re-sync Mode

New in This mode allows the fidessa Server to be re-synced with the Sybase database. In this mode,
v5.0.0 Sdd will remove all SOURCE_REF table entries, and update the LAST_SOURCE_REF with the last
sequence number in the iTB_table_update table. This mode would typically be used
following a Sybase upgrade or truncation of the iTB_table_update table, when a customer
does not want to perform a DOWNLOAD_ALL operation.
2.2.4 Related Groups

A related group is an optional configuration item. It is used to download data that is related to the
current download record. For example, if an INSTRUMENT record is being downloaded, it may be
useful to download ALT_INSTRUMENT_ID records for the instrument as well. Related group
information is not used during a full download. An extract from a sample configuration file is
given below:
recordMap
{
name INSTRUMENT
.
.
.
relatedGroup
{
name ALT_INSTRUMENT_ID
dbDownloadName fidessa..sddsel_ALT_INSTRUMENT_ID GROUP,
parameters ( INSTRUMENT_ID )
join { INSTRUMENT_ID INSTRUMENT_ID }
}
.
.
.
}
2.2.5 Dependents
A dependent is an optional configuration item. It is used to download data that the current
download record depends on. For example, if an ALT_INSTRUMENT_ID record is being
downloaded, and its parent INSTRUMENT record does not exist in the RTD, an attempt is made
to download the INSTRUMENT record first using the dependant information. If the INSTRUMENT
record cannot be found a warning is displayed, followed by an error when an attempt is made to
store the ALT_INSTRUMENT_ID record in the RTD. An extract from a sample configuration file is
given below:
recordMap

Page 12 Sdd
{
name ALT_INSTRUMENT_ID
.
.
.
dependant
{
name INSTRUMENT
join { INSTRUMENT_ID INSTRUMENT_ID }
}
.
.
.
}
2.3 Automatic Configuration

If you are using the fidessa Data Architecture (FDA) then all sql and Sdd configuration should
already be installed. This section should be ignored.
2.3.1 Preliminaries
The following actions need to be carried out before any configuration can take place.
1. Set up the configuration file path environment variable (normally done by source
SetupPS). This environment variable is used by all of the file generation utilities.
2. Decide which real-time database (RTD) tables should have data downloaded from Sybase.
3. Edit SddGen.cfg to specify the Sybase database name and the tables to be downloaded.
Each table block should also specify the primary key to be used for the table. An example
section of an SddGen.cfg file is shown here:
SddGen
{
general
{
dbName fidessa
}
table
{
tableName INSTRUMENT
keyName INSTRUMENT_BY_ID
}
table
{
tableName COUNTERPARTY
keyName COUNTERPARTY_BY_ID
}
.
.
.
2.3.2 Creating Sybase Tables

1. Run the SqlTabGen executable as follows:
SqlTabGen SddGen.cfg

Sdd Page 13
A separate file will be generated for each table in the local directory. The file names have the
format table_<tablename>.sql. Each file contains Transact-SQL commands to create
the table and any indexes based on the definitions in the data dictionary file Dct.cfg.
2. Each SQL file should then be loaded into the database as an input file to isql. Errors are
reported if names are longer than the maximum allowed (30 characters in Sybase). This
sometimes occurs for indexes. If the indexes are required, then they will have to be created
manually and given a shorter name.
3. There are two files which are not generated automatically as they relate to the
TABLE_UPDATE table, which does not exist in the RTD, and hence is not in the data
dictionary. The files are table_TABLE_UPDATE.sql and sp_sdd_TABLE_UPDATE.sql.
These files need to be edited to specify the database name, and then loaded into the
database using isql.
4. Use isql to verify that the tables/indexes have been created successfully.
2.3.3 Sdd Configuration File

The Sdd configuration file Sdd.cfg consists of a hard-coded section which #includes an auto-
generated file called Sdd.cfg.auto. This section describes how to generate this file and
identifies the sections that need to be edited manually.
1. Generate the Sdd.cfg.auto file by running the following utility:
SddCfgGen
This utility creates configuration file blocks for the tables specified in the SddGen.cfg file.
2. Edit the Sdd.cfg.auto file if any dependents or related groups need to be set up.
3. Edit the Sdd.cfg file, specifying the download order of the tables and the database name
and user.
2.3.4 Creating Stored Procedures And Triggers

The Sdd process requires stored procedures to access the Sybase data and triggers to allow
real time updates to be downloaded. This section explains how to generate these automatically.
1. Run the following utility:
SddSqlGen
This generates SQL script files for the stored procedures and triggers for each of the tables
specified in SddGen.cfg. The file names of the stored procedures and triggers have the
format sp_sdd_<tablename>.sql, and tr_sdd_<tablename>.sql, respectively.
2. Edit any stored procedures which read related group information from the database.
3. Each SQL file should then be loaded into the database as an input file to isql.
4. Use isql to verify that the stored procedures and triggers have been created successfully.

Page 14 Sdd
2.4 Sdd.cfg Detail

The Sdd.cfg file contains configuration items as follows:
Path/item Default Meaning
Sdd/general/
ProcessName Name used to identify process in trace

and operator messages.
Name of the OA control service.
Used for SOURCE_ID in OA TXNs
OaTxnSinkName Name of OpenAccess Transaction

Service
DownloadBatch 10 The maximum number of outstanding

txns to allow. i.e. transactions that have
been sent to oaTxnSinkName that Sdd
has not received any ACK or NACK for.
UpdateBatch 200 The maximum number of rows from

TABLE_UPDATE that are processed each
updPollPeriod (later)
ModifiedFieldsOnly FALSE This sets the process-wide default for the

modifiedFieldsOnly option, which
can be changed on a per-record basis.
See the documentation of the

recordMap block for more details.
New in UseModifyForAdd FALSE This sets the process-wide default for the
v5.0.0 useModifyForAdd option, which can be
changed on a per-record basis.
See the documentation of the

recordMap block for more details.
New in SendUpdateLastSourceRef TRUE From v5.0.0, as Sdd starts, it will send a

v5.0.0 TXN_UPDATE_LAST_SOURCE_REF
transaction to each transaction sink. This
allows transaction sinks based on the v5
OpenAccess transaction framework to
retain a limited number of SOURCE_REF
records.
If the transaction sink is not based on the

v5 transaction framework, it is likely to

Sdd Page 15
NACK this transaction, which will cause a

harmless warning in the Sdd log file. To
disable the sending of this transaction
(and hence inhibit the warning) set this
configuration item to FALSE.
New in BatchDelaySeconds 5 seconds Specifies the time to wait, in seconds

v5.1.2 between invocations of the update
sproc, when outstanding updates exist
within Sybase.
From v4.3.1, Sdd implements intelligent

polling. This means that if outstanding
transactions exist within Sybase,
following the invocation of the update
sproc, it will not wait upqPollPeriod
seconds before attempting to retrieve
them, but will attempt to retrieve them
immediately. In turn this may cause Sdd
to consume unnecessary CPU, and
Sybase server time.
Sdd/sybase/
DbUserName Sybase login name
DbName Sybase database name (optional -

defaults to default database for
dbUserName)
DbServerName $DSQUERY Sybase server name. It is used when and

only when the environment variable
DSQUERY is not defined.
DbPassword $PASSWORD Sybase login password for dbUserName.

It is used when and only when the
environment variable PASSWORD is not
defined.
If an encrypted password is required, this

item should not be specified.
New in EncryptedDbPassword None For added security, the Sybase login

v5.3.1 password may be specified in
(Optional)
configuration in an encrypted format.
The encrypted form of the password is

Page 16 Sdd
generated by using a TCL extension, run

as,
ExecTclProc Fid_Utl encrypt

<passwd>.
If this config item is not specified, then

the DbPassword item is used.
New in Sdd/StatusSprocs/ (Optional)

V5.0.0
New in StatusSproc Sybase stored procedure to invoke on

V5.0.0 DATABASE_AVAILABLE,
DATABASE_CLOSING, and on Sybase
login. (See section entitled Status Stored
Procedures below)
New in NackSproc Sybase stored procedure to invoke on

V5.0.0 receving a TXN_NACK. (See section
entitled Status Stored Procedures below)
Sdd/databaseMap/recordMap/ (one for each download table)
Name rtd table name
Key rtd key name
Replace FALSE If true, TXN_REPLACE will be used

instead of TXN_MODIFY for this table.
ModifiedFieldsOnly Value If true, only primary key fields and those

specified in whose values have changed are
the general specified when a TXN_MODIFY is
block generated. If no field values have been
changed, no transaction will be
generated.
This option can be used to reduce the

number of transactions necessary when
performing a download all, making the
operation faster. It is also used to
guarantee accurate update reporting in
FTS, where all the fields which are
supplied in a modify transaction are
marked as changed in the resulting

Sdd Page 17
update queue entry.
Please see the warning below this table

regarding the use of this option.
New in UseModifyForAdd Value If true, TXN_MODIFY will be used instead

v5.0.0 specified in of TXN_ADD for this table. See section
the general 2.4.2.
block
Sdd/databaseMap/recordMap/downloadAll/ (one only)
DbDownloadName sql to output all rows from Sybase for this

table
Sdd/databaseMap/recordMap/downloadRecord/ (one only)
DbDownloadName sql to output a specific row from Sybase

when the key field values are appended
parameters () list of fields names making up the key

used in dbDownloadName
Sdd/databaseMap/recordMap/fieldMap/
name value pairs name is rtd field name and value is sybase column
name. All fields must be specified.
Sdd/databaseMap/recordMap/update/
DbTableName name of this table as used in the

updTableNameColumn in
TABLE_UPDATE.
dbKeyColumns () The Sybase column names from

TABLE_UPDATE corresponding to the
parameters() list.
Sdd/databaseMap/
downloadOrder () List of download tables names to specify

the order in which tables are downloaded
during download all.
Sdd/updateProc/
UpdateName Sproc name for

sddsel_TABLE_UPDATE
updTableNameColumn Name of column in TABLE_UPDATE

Page 18 Sdd
containing table name data
updSequenceColumn Name of column in TABLE_UPDATE

containing SEQUENCE_NUM data
upqPollPeriod Period between executions of

updateName in seconds
upqMaxBatch 1000 Maximum number of rows to retrieve

from TABLE_UPDATE on each invocation
of updateName.
Typically, many of these parameters are '#include' ed from

ClientSpecific.<host>.cfg. This will depend upon the product (FTS, OMAR, etc) and the
site.
2.4.1 Use of modifiedFieldsOnly

The modifiedFieldsOnly option can be used in all fidessa systems to limit the number of
transactions generated when a download all is repeated, as a transaction will only be sent if at
least one field on a given record has changed its value.
modifiedFieldsOnly is also used in FTS to limit the fields sent to the transaction handler for
each modification. FTS has two classes of modify transactions replace and update. When a
subset of available fields is specified, a replace handler will remove any unspecified fields from
the record being modified; an update handler will merge the new fields into the existing record.
modifiedFieldsOnly is useful with an update class transaction, as it ensures that only fields
whose values have changed are posted to the update queue. However, with a replace class
transaction, only specifying modified fields will result in existing fields which have not changed
being removed.
WARNING: DO NOT SET THIS OPTION FOR FTS TABLES WHICH HAVE A REPLACE CLASS HANDLER FOR THE
MODIFY TRANSACTION. IF IN DOUBT, CHECK TRH.DIRECTORY.CFG.
2.4.2 Use of useModifyForAdd

It is recommended that the useModifyForAdd option be set to TRUE wherever possible. This
results in all insertions to the fidessa RTD to be sent as TXN_MODIFY transactions rather than
TXN_ADD transactions. The reason for doing this is to avoid the scenario where the same
record is updated in the Sybase database in quick successsion, resulting in two TXN_ADD
transactions being raised by Sdd, the second of these will fail. Although the resulting data will
still be correct, the TXN_NACK resulting from the second TXN_ADD can be mis-leading.

Sdd Page 19
In order to use this option in FTS, transactions in the Trh Directory should be configured
with an existance optional clause in the MODIFY transactions, for example,
transaction
{ type MODIFY
class Update
function BOOK
instance
{ record
{ name BOOK
primaryKey
{ join { to BOOK by BOOK_BY_ID
existance optional }
}
}
}
}
Note: In all systems other than FTS, using modifyForAdd should work without the configuration of the
existance optional clause.
2.4.3 Status Stored Procedures

Sdd allows the specification of two status-stored procedures, the Sybase interface to these are
as follows,
statusSproc
proc iSP_sdd_status
(
@system_name DT_SYSTEM_NAME,
@sytem_type DT_SUB_SYSTEM,
@system_version DT_SYSTEM_VERSION,
@status_message DT_FREE_FORMAT
}
It should be noted that the system_name, system_type and system_version parameters

should be specified in the Sdd configuration, with a trailing ,.
status_message will be used to record one of the following events.
status_message Sdd Event
DATABASE_CONNECTED Sdd login to Sybase
DATABASE_AVAILABLE Received DATABASE_AVAILABLE event
DATABASE_CLOSING Received DATABASE_CLOSING event

Page 20 Sdd
DOWNLOADALL_START Download-all has started
DOWNLOADALL_COMPLETE Download-all is complete
An example of a valid statusSproc definition would be,
statusSproc data_33..iSP_sdd_status 'fidessa-FTS-MAIN', 'FTS', '4.2.0',
nackSproc
The Sybase interface to the nackSproc will be as follows,
create proc iSP_sdd_nack

(
@system_name DT_SYSTEM_NAME
@system_type DT_SUB_SYSTEM,
@txn_type DT_TXN_TYPE /* (varchar(16) */
@record DT_SYBASE_OBJECT,
@sequence_num int,
@msg_time datetime,
@error_code int,
@error_text DT_FREE_FORMAT
}

should be specified in the Sdd configuration, with a trailing ,. An example of a valid
nackSproc definition would be,
nackSproc data_32..iSP_sdd_nack 'fidessa-FTS-MAIN', 'FTS', '4.2.0',
The additional parameters are defined as follows,
Parameter Name Description
txn_type The transaction type of the message that caused the NACK, e.g
TXN_ADD, TXN_MODIFY
record The RTD record type of the NACKed message
sequence_num The sequence number of the NACKed message
msg_time The date / time the transaction was originally sent
error_code As returned in the transaction NACK
error_text As returned in the transaction NACK

Sdd Page 21
2.5 Environment variables

Sdd uses environment variables as follows:
Name Default Meaning
PASSWORD The Sybase login password for dbUserName
DSQUERY Default Sybase SQL server name
SYBASE Location of Sybase install.
Note: As of infrastructure v4.2.2, the Sybase login password and default SQL server name are specified in
the Sdd configuration file.
2.6 Command line Arguments

Sdd has command line arguments as follows:
-download_all FALSE Download all Sybase data. This is usually configured as a

command line argument in the appropriate start mode in
ProcessDetails.syscfg. (See Note 1 below)
-exit_after_load FALSE Causes Sdd to exit after download all has completed.
-resync FALSE Causes Sdd to run in re-sync mode, see section 2.2.3.
-c <file.cfg> Sdd.cfg Specifies the config file to be used
Notes:
1. The -download_all flag can be configured to be set if Sdd is started when fidessa is
started cold by using a MODE: setting in ProcessDetails.syscfg as show here:
PROCESS:
NAME=EMS_SDD
CMDLINE=Sdd -c Sdd.cfg
DESCRIPTION=Sybase Data Download
START=YES
MODE:
NAME=DOWNLOAD_ALL
CMDLINE=Sdd -c Sdd.cfg -download_all
MODE:
NAME=RESYNC
CMDLINE=Sdd c Sdd.cfg -resync
2.7 OpenAccess Stats

Sdd implements HEALTH and COMMAND=DUMP attributes and REQ_CLASS requests.

Page 22 Sdd
The REQ_CLASS request contains the following:
Number of transactions outstanding (not yet Acked or Nack)
The first and last row numbers from TABLE_UPDATE for the last poll.
The SOURCE_REF of the last TXN Acked and the last TXN Nacked.
The SOURCE_REF of the last TXN sent.
For each download table:
Numbers of records downloaded during download all.
Totals for number of add, modify and delete transactions made (both during download
all and since).
The OpenAccess Statistics are further detailed in Appendix A.

FidToSyb Page 23
3. FidToSyb
3.1 Design
FidToSyb operates in two stages. When FidToSyb starts, it is in the recovery stage. When
recovery has finished, it enters the update stage. During recovery, FidToSyb processes each
service from the config file in the order specified in the recoveryOrder config item list. For
each service, it gets the last record from the Sybase table and uploads all records after that
record. Then it moves on to the next service in the recovery list. When all services have been
recovered, FidToSyb enters the update stage and starts polling each 'service' for recently
updated records and uploads those records to Sybase.
last last key GetLast

database or timestamp Procedure
reads values Calls
Fidessa Sybase SQL Server
rtd FidToSyb Reporting Reports

Database
Polls for Update /

Updated Insert
Records Sprocs
3.2 Operation
Note that the primary table being uploaded from fidessa must have a sorted key on timestamp.
For example, to add a sorted timestamp key to the TRADE_AUDIT table, you could use the
following configuration in DctClient.cfg.:
customiseRecord
{
name TRADE_AUDIT

Page 24 FidToSyb
key { name TRADE_AUDIT_BY_TS id 50002 sort (TIMESTAMP) }

}
Note that the id used here is just an example. It should be unique on your system.
The record from which recovery is started is determined according to whether the insertsOnly
config item is set for the service. If true FidToSyb expects records to be inserted into the rtd and
never updated and as such, never updates existing records in Sybase, only ever inserts records,
even during recovery. This allows client triggers that expect only inserts to operate correctly. It is
not always possible to use insertsOnly services since it requires a particular kind of key (see
later). If insertsOnly is set to FALSE, then the service may upload duplicate records during
recovery and later, may download updates to sybase records that have been changed in the rtd.
3.2.1 Recovery of insertsOnly services

FidToSyb either runs the primaryKeySproc or generates some SQL that obtains the key
value(s) for the last record uploaded to Sybase by FidToSyb. FidToSyb takes this key value to
obtain the original record from fidessa. It then scans each fidessa record after the original in
TIMESTAMP order, uploading each row to Sybase in turn. For example, if Sybase contains four
records with TIMESTAMPs and keys as shown and fidessa contains six records with timestamps
and keys as shown then only two new records are uploaded.
Sybase table Fidessa table
T/S Key T/S Key

11:41 28 11:41 28
11:44 29 11:44 29
11:47 30 11:47 30
12:00 31 12:00 31
12:03 32
12:06 33
FidToSyb obtains the key value for the last row uploaded to Sybase by FidToSyb (during a
previous run) which is 31. FidToSyb then looks up the record with that key value in fidessa..
Then FidToSyb scans all the records in fidessa that come after that record in TIMESTAMP order
and uploads them to Sybase. In this case, the records with a timestamp of 12:03 and 12:06
would be inserted into the Sybase table. The key used should be a system generated, sorted
key. Any key using the TIMESTAMP would be inappropriate since TIMESTAMP values may be
different across a live/standby pair. Note that updates to records with a key value older than the
last will be ignored. If a suitable key cannot be found then that service in FidToSyb will have to

FidToSyb Page 25
be configured as a non-insertsOnly service that uses the TIMESTAMP key, which always
increments with every modification.
3.2.2 Recovery of non-insertsOnly services

FidToSyb either runs the timestampSproc or generates some SQL that obtains the timestamp
value of the last record uploaded to Sybase by FidToSyb. FidToSyb subtracts
recoveryRewindMins minutes (1 by default) from this value then scans the fidessa table from
the record before the timestamp onwards in TIMESTAMP order, uploading each row to Sybase in
turn. This overlap is introduced to compensate for the time on the primary and standby machines
being different. For example, if Sybase contains 4 rows with timestamps as shown and fidessa
contains 6 rows with timestamps as shown then 2 records are uploaded for a second time
(update) and two new records are uploaded for the first time (insert)

11:41:00 11:41:00
11:58:44 11:58:44
11:59:04
11:59:47 11:59:47
12:00:04 12:00:04
12:00:05
12:01:44
The last record in Sybase is the one with a TIMESTAMP value of 12:00:04. FidToSyb take 1
minute from this giving 11:59:04. Then FidToSyb uploads the records from this timestamp
onwards which means the last four records from fidessa with timestamp values of 11:59:47,
12:00:04, 12:00:05 and 12:01:44.
3.2.3 Updates
During recovery, the most recent TIMESTAMP value is recorded for each service before the next
service is recovered. When all services have been recovered, FidToSyb enters its normal mode
of operation which is to poll the primary table on each service at regular intervals, attempting to
read records with a newer TIMESTAMP then the recorded value. Any records found are uploaded
to Sybase. The most recent TIMESTAMP value is then recorded again.
To avoid sending records to Sybase before the fidessa transaction which generated them has
been journalled, FidToSyb also listens to the fidessa a update queue for updates to the primary
table. FidToSyb needs to spot an update which corresponds to each newer record before
sending that record to Sybase. If such an update never arrives (because the record was updated

Page 26 FidToSyb
using Dbm, or because the transaction failed to post an update, FidToSyb will log the following
operator message:
30 JUN 1998,10:48:06,WAR,INF_F2S-0-Reliable stream lagging behind - check log

file
FidToSyb will then continue to periodically poll for updates, sending to Sybase those new
records which are over ten seconds old at the time of polling.
3.2.4 Replay Mode

On rare occasions, FidToSyb, during its normal operating cycle, may fail to load records into
New in Sybase. Typically, this can occurs if,
v5.3.0
Deadlock occurred when attempting to write the fidessa record to Sybase.
The fidessa update-queue wrapped, leading to lost updates.
In this instance, Recovery may not lead to the missing records being restored to Sybase. For
example, if a non-insert only service succesfully dowloaded a record into Sybase, following a
failed record, the failed record would never be recovered.

11:41:00 11:41:00
Downloaded
11:58:44 11:58:44
11:59:47
Failed
12:00:04
12:00:05 12:00:05
Downloaded
12:01:44 12:01:44
In this case, Recovery would attempt to recover all records from 12:01:44, meaning that
records 11:59:47, and 12:00:04, would never be downloaded to Sybase.
Further, situations may exist whereby a new field is added to a fidessa RTD table, which needs
to be populated to existing Sybase records, which are for example, less than a month old. There
is no way to perform this function from a standard FidToSyb.
Replay mode is intended to solve these two problems. When started in this mode, FidToSyb
passes a replayMode and replayAction parameter to the primaryKeySroc and
timestampSproc. The replayMode parameter simply indicates that replay is in progress,
and instructs the sproc to return only records between timestamps configured within the

FidToSyb Page 27
iTB_report_service Sybase table. The replayAction parameter may be one of the

following,
SKIP - Indicates that if records are already present within Sybase, they should be
skipped.
UPDATE Indicates that if records are already present within Sybase, they will be updated.
This action is specifically used to cater for the case where extra fields have beed added, and
need to be populated to Sybase.
When replay of all records for a particular service is complete, FidToSyb advances on to the
next service. When all services have been replayed, FidToSyb will exit. This allows replay to
run concurrently with a primary FidToSyb, the primary handling updates that occur in real time,
and the replay restoring missing records.
In order to allow sharing of configuration between FidToSyb processes run in replay mode, and
FidToSyb run in normal mode, two embedded variables may be used.
Variable Type Default Value in Value in Replay

Value Normal Mode Mode
replayMode string N N Y
replayAction string SKIP or UPDATE
3.2.5 sql vs sproc

If the dbUpdateMethod config item is set to sql then the following code is generated as
replacements for the sprocs. This example assumes a table HIST with a primaryKey
HIST_BY_ID containing a field HIST_ID and the table has fields NAME and PRICE.
Sproc SQL
dbSprocName if exists (select 1 from HIST where

HIST_ID = @hist_id)
Begin
update HIST
set TIMESTAMP = @timestamp,
HIST_ID = @hist_id,
NAME = @name,
PRICE = @price
where HIST_ID = @hist_id
end
else
begin
insert HIST (TIMESTAMP,
HIST_ID,
NAME,
PRICE)
values(@timestamp,
@hist_id,
@name,

Page 28 FidToSyb
@price)
end
timestampSproc select max(TIMESTAMP) from HIST
primaryKeySproc select HIST_ID from HIST whereTIMESTAMP =

(select max(TIMESTAMP) from HIST)
The SQL generated is of minimum complexity, does not use transactions and does not handle
errors in the sproc specifically. Any sprocs written to replace these should be based on the
above in terms of input parameter and row output.
From v5.3.0, the timestampSproc and primaryKeySproc items should be specified as

New in
expressions. The embedded variabled replayMode and replayAction (see section 3.2.4),
v5.3.0
are available to these expressions. For example,
primaryKeySproc iSP_fts_pk_trade_set + replayMode
Prior to v5.3.0, both the primaryKeySproc and timestampSproc were specified as strings. If
these config items remain configured, as strings, a warning will be displayed, the process will
however, operate as normal.
3.2.6 Sybase errors

When FidToSyb starts, it attempts to connect or login to Sybase. This should timeout after 60
seconds if there is no response from the SQL server. This timeout can be altered with the
dbLoginTimeout configuration item.
If the connect/login is timed out or fails then a reconnection attempt will be made after 60,000 ms
(1 minute). This interval can be changed with the pollPeriod configuration item.
During the execution of a Sybase sproc or sql statement, FidToSyb will be waiting for a
response from the SQL Server. If no response is received then FidToSyb will wait forever for
the command to 'complete'. FidToSyb can be made to limit the how long to wait for a response
before returning an error by setting the dbStaleTimeout configuration item.
Sybase errors are reported to trace and operator logs. If a severe or fatal error is detected then
FidToSyb will disconnect and reconnection will be attempted after 60 seconds as above.
When FidToSyb disconnects from Sybase, the polling of tables for updates is stopped. When
FidToSyb has reconnected to Sybase, it starts with recovery as if it had been run for the first
time, moving back to polling for updates when all services have been recovered.
3.3 FidToSyb.cfg Detail

The FidToSyb.cfg file contains configuration items as follows:
Path/Item Default Meaning
FidToSyb/general/
processName <none> Name used to identify process in trace

FidToSyb Page 29
and operator messages
Name of the stat n dump OA control

service
upqPollPeriod 1000 Period in ms between polling tables for

updated/new records.
upqMaxBatch 100 Maximum number of updated records

uploaded to Sybase on each poll of a
service.
recoveryRewindMins 1 The TIMESTAMP overlap period (in

minutes) used during recovery of non
insert-only services.
dbLoginTimeout Sybase Interval after which Sybase should Return

default an error on a login
(60
seconds)
dbStaleTimeout Sybase Interval after which Sybase should return

default an error on a command (sproc or sql)
(never)
pollPeriod 60000 Interval after which a failed Sybase login

is retried
New in healthCheckDir <none> This should specify a directory to hold the

v5.0.0 FidToSyb control file. Should be
specified as $PS_RUN/control. (See
section entitled Health Checking, below).
New in healthCheckPollSeconds 30 The time in seconds between writing

v5.0.0 health information to the control file (See
section entitled Health Checking, below)
flushAtClose TRUE If set to TRUE, FidToSyb will hold up

system closedown until it has propagated
all updates to Sybase.
If set to FALSE, FidToSyb will not

guarantee to flush updates when fidessa
closes down.
retryPeriod 10000 Unused

Page 30 FidToSyb
asyncBatchSize 10 Unused
asyncBatchDelay 100 Unused
maxRetryAttempts 2 The number of times FidToSyb will

attempt to upload a particular record if the
upload fails due to a deadlock.
FidToSyb/expression/ Standard expression block. See the fidessa expression

evaluator programmers guide.
FidToSyb/linkMgr/ Standard OpenAccess control block
FidToSyb/sqlConnection/
dbServer $DSQUERY Sybase SQL server name
dbName Sybase database name. If defaulted

then use default database for user
dbUser $REP_USER Sybase login name
dbPassword $REP_PASSWORD Sybase login password. This

configuration item should not be
specified if an encrypted password
is required.
New in encryptedDbPassword None For added security, the Sybase

v5.3.1 (optional) login password may be specified in
configuration in an encrypted
format. The encrypted form of the
password is generated by using a
TCL extension, run as,
ExecTclProc Fid_Utl
encrypt <passwd>.
If this config item is not specified,

then the DbPassword item is
used.
dbLoginTimeout 0 How long before a login attempt

times out. If defaulted to zero then
the Sybase default value is used
which is 60 (seconds)
dbStaleTimeout 600 How long to wait for a response

before the Sybase SQL server is
timed out (seconds). If set to zero,

FidToSyb Page 31
then an operation will never time

out and FidToSyb will pend until
the operation completed or is
cancelled by a DBA.
pollPeriod 60000 Interval between a failed login

attempt and a retry.
subst <none> A list of pairs of names where the

second word is used as a
replacement for the first in SQL
commands, and Sybase column
and table names. See Note 1
below:
FidToSyb/serviceManager/
standbyActive FALSE If set to TRUE, FidToSyb will connect to

Sybase and start uploading data in the
DATABASE_RECOVERABLE state, i.e. on
the standby system in a primary/standby
pair.
recoveryOrder <none> Order in which services are recovered.
New in statusSproc <none> Sybase stored procedure to invoke on

v5.0.0 DATABASE_AVAILABLE,
DATABASE_CLOSING, and on Sybase
login. (See section entitled Status Stored
Procedures, below)
New in dataSyncPollSeconds 60 Intervals at which to check the status of

v5.0.0 any DATA_SYNC_STATUS requests
(See section entitled Data Sync Status,

below)
New in FidToSyb/serviceManager/streamStatus/ (See section entitled Stream

v5.0.0 Status, below)
New in checkIntervalSeconds <none> The interval in seconds at which the

v5.0.0 status of the stream should be checked.
New in maximumLagSeconds <none> The maximum time in seconds that a

v5.0.0 stream can be behind, before the sproc
is invoked.

Page 32 FidToSyb
New in streamStatusSproc <none> The name of the sproc to invoke when the
v5.0.0 stream lags behind.
FidToSyb/serviceManager/service/
name <none> Identifies this service.
upqPollPeriod from Used to override the value specified in the

general/ general block on a per service basis.
upqMaxBatch from Used to override the value specified in the

recoveryRewindMins from Used to override the value specified in the

insertsOnly <none> Recovery mechanism (See 3.2.1and

3.2.2)
primaryKey <none> The key used to access fidessa data
dbUpdateMethod <none> sql or sproc (See 3.2.5)
Modified in primaryKeySproc <none> (See 3.2.5)

v5.3.0
Modified in timestampSproc <none> (See 3.2.5)

v5.3.0
dbTableName <none> Name of Sybase table
recoveryBatchSize 1000 How many records to recover on each

iteration during recovery.
recoveryInterval 1000 Interval between recovery batches during

recovery.
table <none> Standard database scanner tables blocks.
parameters <automatic> See 3.7
Notes:
1. FidToSyb automatically includes substitution for Sybase reserved words. Any further
substitutions added here would replace long fidessa field names of more than 30 characters
with shorter ones for Sybase. See Appendix B for a list of Auto Substitutions
3.3.1 Health Checking

New in Prior to v5.0.0, FidToSyb, would receive health check requests via OpenAccess, and respond to
v5.0.0 these indicating the health of the process. Due to the synchronous nature of FidToSyb,
however, it is possible for FidToSyb to be busy processing updates, and hence miss its health

FidToSyb Page 33
check. From v5.0.0, FidToSyb uses file based health checking, the directory in which the file
should be created in specified in configuration, as healthCheckDir. This must be set to
$PS_RUN/control, as this is the directory that Minder will check to determine the process
health.
To Configure health checking, the following line should be added to the

ProcessDetails.syscfg file for the FidToSyb process
CHECK_PROC=Fid_F2SCheck <process name>
This indicates the name of a tcl procedure that will be invoked by Minder at pre-configured
time intervals to check the health of the process.
It should be noted that if a pre v5.0.0 Fid_F2Scheck tcl procedure were used with a v5.0.0
FidToSyb process, health checking would be implemented using OpenAccess.
3.3.2 Status Stored Procedures

New in FidToSyb allows the specification of a status-stored procedure, the interface to which is as
v5.0.0 follows.
proc iSP_sdd_status
(
@status_message DT_FREE_FORMAT
}

should be specified in the Sdd configuration, with a trailing ,.
status_message will be used to record one of the following events.
status_message Sdd Event
DATABASE_CONNECTED FidToSyb login to Sybase
DATABASE_AVAILABLE Received DATABASE_AVAILABLE event
DATABASE_CLOSING Received DATABASE_CLOSING event
An example of a valid statusSproc definition would be,
statusSproc data_33..iSP_sdd_status 'fidessa-FTS-MAIN', 'FTS', '4.2.0',

Page 34 FidToSyb
3.3.3 Data Sync Status

New in In some instances, it is necessary to determine when all updates pertaining to a particular action
v5.0.0 have been downloaded to Sybase. An example of this, is Mark to Market, which updates many
RTD records. However, even though the Mark to Market may have completed, the updates may
not appear in the Sybase database for some time.
Data Sync Status is provided to indicate when tasks such as this are complete. FidToSyb
receives the following OpenAccess message to achieve this.
Field Name Reqd Format Notes
MESSAGE_TYPE Yes string Should be REQ_F2S_DATA_SYNC_STATUS
USER_NAME Yes string User making the request.
DATA_SYNC_NAME Yes string Name used to identify this request (e.g. MTM), used for
information only.
TIMESTAMP Yes string In the format YYYYMMDD HH:MM:SS. When each of the list
of services have synchronised up to this time-stamp, the sproc
will be invoked.
SPROC_NAME Yes string The name of the sproc, and any associated parameters. This
string will be passed as-is, to the SQL Server.
SERVICES Yes string[] List of service names to monitor.
On receipt of this message, FidToSyb may generate the following reply.
REP_F2S_DATA_SYNC_STATUS
Field Name Reqd Format Notes
MESSAGE_TYPE Yes string REP_F2S_DATA_SYNC_STATUS
ERROR_CODE Yes I32 10010 invalid sproc name

10011 invalid time stamp
10012 invalid service name
Other codes are possible but should only be used for enhancing
diagnostics. The range 0-99 is reserved for standard codes.
Where 0 indicates no error.
ERROR_TEXT Yes string Descriptive string of reject reason, only set if ERROR_CODE is
non zero.
It should be noted that the sproc defined in the OpenAccess REQ_F2S_DATA_SYNC_STATUS

message will only be invoked once all given services have synced up to the timestamp.

FidToSyb Page 35
It will be possible at any time to check the status of the data synchronisation status requests, by
use of OpenAccess Statistics. These will show all requests received, and those that are still
outstanding. It will also be possible to verify from these statistics, which services have not yet
synchronised up to the time-stamp.
An example of the output from OaReq follows. The first example shows a data sync status
request that is active, and waiting for the TB_CURRENCY_RATE service to Sync up to the time-
stamp.
CLASS[1].CLASS[0].DATA_SYNC_STATUS[0].SYNC_NAME.s = "MTM"
CLASS[1].CLASS[0].DATA_SYNC_STATUS[0].TIMESTAMP.s = "19990831 14:14:00"
CLASS[1].CLASS[0].DATA_SYNC_STATUS[0].SPROC.s = "FID_DATA_SYNC"
CLASS[1].CLASS[0].DATA_SYNC_STATUS[0].STATUS.s = "ACTIVE"
CLASS[1].CLASS[0].DATA_SYNC_STATUS[0].SERVICE.s = "TB_CURRENCY_RATE"
This example shows that two data sync status requests have been issued, the first is complete,
and the second is still active, and is waiting for both the TB_CURRENCY_RATE, and the
TB_INSTRUMENT_PRICE services to sync up to the time-stamp.
CLASS[1].CLASS[0].DATA_SYNC_STATUS[0].SPROC.s = "FID_INSTRUMENT_PRICE"
CLASS[1].CLASS[0].DATA_SYNC_STATUS[0].STATUS.s = "ACTIVE"
CLASS[1].CLASS[0].DATA_SYNC_STATUS[0].SERVICE.s = "TB_CURRENCY_RATE"
CLASS[1].CLASS[0].DATA_SYNC_STATUS[0].SERVICE.s = "TB_INSTRUMENT_PRICE"
CLASS[1].CLASS[0].DATA_SYNC_STATUS[1].SPROC.s = "FID_INSTRUMENT_PRICE"
CLASS[1].CLASS[0].DATA_SYNC_STATUS[1].STATUS.s = "COMPLETE"
3.3.4 Stream Status

New in FidToSyb can be configured to automatically invoke a stored procedure when streams are
v5.0.0 lagging behind. This is specified by the streamStatus configuration block. If this block is
specified, then all entries within it are mandatory.
The interface to the streamStatusSproc is as follows,
proc iSP_sdd_stream_status
{
@service DT_SYBASE_OBJECT,
@up_to_date boolean,
@lag_time int
}

Page 36 FidToSyb

would be specified in the FidToSyb configuration as per other sprocs. An example of a valid
streamStatusSproc definition would be,
streamStatusSproc data_32..iSP_sdd_stream_status 'fidessa-FTS-MAIN', 'FTS',

'4.2.0',
The additional parameters are defined as follows,
Parameter Name Description
service The name of the service as specified in the FidToSyb configuration.
up_to_date TRUE if the service is lagging behind, FALSE otherwise
lag_time The time in seconds that the service is lagging behind.
At start-up, FidToSyb will invoke the streamStatusSproc for each configured service, setting
the up_to_date flag to TRUE, and the lag_time to 0. Thereafter, FidToSyb will monitor the
status of the stream, and only invoke the sproc, if a service is behind by more than the
maximum allowable lag time. Once the service had caught up, the sproc would again be
invoked to indicate that it was now up to date.

FidToSyb uses environment variables as follows:
Name Meaning
REP_USER Default Sybase login name
REP_PASSWORD Default Sybase login password

FidToSyb has command line arguments as follows:
-c <file.cfg> FidToSyb.cfg Specifies the config file to be used
-r Specifies replay mode (See section 3.2.4)
-a <replayAction> Specifies the replay action, either SKIP, or

UPDATE, (See section 3.2.4)

FidToSyb Page 37

FidToSyb implements HEALTH and COMMAND=DUMP attributes and REQ_CLASS requests.
The response to the REQ_CLASS query includes the following information:
Connection details including, server name, database name and user name
State of the connected (connected or not connected)
Time of last successful connect
Time of last disconnect
Number of failed connection attempts
Number of failed commands (sproc or sql executions)
If FidToSyb is recovering, and if it is, the service it is currently recovering.
For each service:
Number of records recovered.
Number of records updated.
Various timstamps and counts indicating if the service is keeping sybase up to date wrt
changes to the rtd and if not, how far behind it is.
A summary of timestamps and counts, over all services, indicating if FidToSyb is keeping up
with changes to rtd and if not, how far behind it is.
A list of data sync status requests, showing both active, and complete. Active requests, will
show each service that has not yet synced up to the timestamp.
The OpenAccess Statistics are further detailed in the Appendix A.
3.7 fidessa to Sybase column mapping

if dbUpdateMethod is sproc then each field from the primary (first) table in the table join is
mapped to a sproc argument with the same name as the fidessa field but in lower case. If
dbUpdateMethod is sql the fidessa field is mapped to a Sybase column with the same
name. If the fidessa field has a dct_time type then the FidToSyb internal function
SybaseTime$() is applied to the value to make it look like a Sybase time. For example, a
fidessa table with two fields THE_TIME and THE_PLACE would behave as if there was a
parameters block like this:
parameters
{
SybaseTime$(t.THE_TIME) the_time
t.THE_PLACE the_place
}
for a dbUpdateMethod of sproc and

Page 38 FidToSyb
parameters
{
SybaseTime$(t.THE_TIME) THE_TIME
t.THE_PLACE THE_PLACE
}
for a dbUpdateMethod of sqlThese default mappings can be overridden by the

parameters block. If this block is specified, the default mappings are not added. Those from
the parameters block will be used instead.
In the example above, you could use:
MEETING LOCATION IS + t.THE_PLACE THE_PLACE
to specify that 'MEETING LOCATION IS ' will be prepended to the THE_PLACE field as it is
uploaded to Sybase. If a join is made to a secondary table then expressions using values from
that table can be used to replace values for existing sproc parameters or column name or to
specify new ones. Sproc parameters or column names should be specified as above, i.e. in
lower case without @. The SybaseTime$() function takes a string in the form HHMMSSCC as
output by the fidessa expression evaluator for conversion of a dct_time type to string, and
converts it to the form HH:MM:SS.CC which is recognisable by the Sybase parser as a time
value.

OAToDb Page 39
4. OAToDb
4.1 Design
OaToDb provides a way of running a Sybase stored procedure (sproc) from an OpenAccess
message. It can pass a number of arguments to the sproc and return the results. It accepts
standard OpenAccess transactions containing an embedded SDS with the arguments to the
sproc and will reply with a NACK containing the sproc error status or an ACK containing any row
output.
4.2 Transactions
OAToDb supports the Transaction protocol as described in fidessa OpenAccess Programmers
Guide with the following caveats:
Recognised Database transaction MESSAGE_TYPE values are TXN_ADD, TXN_MODIFY

and TXN_DELETE.
Only one Add/Modify/Delete or sproc call per transaction.
No validation is performed on USER_NAME.
Values for VALIDATE of 0 or 1 are accepted but ignored.
Values for IGNORE_WARNINGS of 0 or 1 are accepted but ignored.
OAToDb does not support source reference requests or reply recovery.
4.2.1 Generic Database Transactions

The embedded SDS, following the standard transaction fields, must match an entry in the
configuration file for OAToDb, from which the stored procedure name is obtained. OAToDb only
supports one sproc call per transaction so only one embedded SDS should be specified. It is an
error to specify any more and the transaction will be NACKed.
If the stored procedure has a parameter @txn_type, then this is set from MESSAGE_TYPE. The
remaining parameters are set from the fields in the embedded SDS. Each field specified must
match a sproc parameter. The fields may be specified in either upper or lower case and they
may or may not have a leading '@'. If the sproc has extra parameters, not specified in the SDS
then they will not be set in the call. If the sproc can be called without those parameters then it
won't cause an error.

Page 40 OAToDb
The fields in the SDS may be of type SDS_BYTE, SDS_I32, SDS_STRING or SDS_F64. No other
SDS types can be used. The sproc parameters may be any of the Sybase types int,
smallint, tinyint, float, real, money, char(n), varchar(n) or datetime. OAToDb will
automatically convert the SDS field type to the Sybase type.
4.2.2 Ack Reply

When the sproc returns, any row output is appended to the standard ACK response in a
repeating SDS called ROWS. There will be an instance of the ROWS SDS for each row produced
by the sproc. The fields will have the same names as the column names. The Sybase datatypes
are represented as SDS types as follows:
Sybase data type SDS type
int, smallint or tinyint SDS_I32
float, real or money SDS_F64
char(n), varchar(n) or datetime SDS_STRING
4.2.3 Nack Reply

When there has been an error, a standard NACK response is sent back. An error might be
generated by an incorrect or invalid transaction being received by OAToDb or the sproc returning
a non-zero exit status with the RETURN keyword. If the sproc fails then the exit code number and
any recent asychronous Sybase error text are used to generate the ERROR_TEXT field.
Otherwise, if some other error occurred such as the sproc not being found in Sybase then a
ERROR_TEXT will contain a self explanatory message.
4.3 Configuration
4.4 OAToDb.cfg Detail

The OAToDb.cfg file contains configuration items as follows:
OAToDb/general/
serviceName Name of the OpenAccess service to provide.
OAToDb/linkMgr/
ipcName Set to the same as serviceName.
OAToDb/linkMgr/stateLink
address Can be set to NONE to cause OAToDb to connect

to Sybase immediately on startup and provide the

OAToDb Page 41
named service. Otherwise, OAToDb will wait for a

DATABASE_AVAILABLE state notification from the
state server named here.
OAToDb/sqlConnection/
dbServer $DSQUERY Sybase SQL server name
dbName Sybase database name. If defaulted then use

default database for user
dbUser $REP_USER Sybase login name
dbPassword $REP_PASSWORD Sybase login password
pollPeriod 60000 Interval between a failed login attempt and a retry.
OAToDb/service/
tableName Used as a match for the name of the embedded

SDS in a received transaction
sprocName Corresponding sproc to execute.
There can be any number of OAToDb/service blocks.

OAToDb uses environment variables as follows:
Name Meaning
REP_USER Default Sybase login name
REP_PASSWORD Default Sybase login password

OAToDb has command line arguments as follows:
-c <file.cfg> OAToDb.cfg Specifies the config file to be used

Page 42 OAToDb

OAToDb implements HEALTH, COMMAND=DUMP attributes and REQ_CLASS requests.
The response to the REQ_CLASS query includes the following information:
Connection details including, server name, database name and user name
State of the connected (connected or not connected)
Time of last successful connect
Time of last disconnect
Number of failed commands (attempts to execute sprocs)
Totals for TXN_ADD, TXN_MODIFY and TXN_DELETE transactions received.
Totals for TXN_ACK and TXN_NACK replies sent.
The OpenAccess Statistics are further detailed in Appendix A.

Appendix A Page 43
5. Appendix A
5.1 Sdd OpenAccess Statistics
5.1.1 Class Name
<SERVICE_NAME>.INFO
5.1.2 Statistics
Field Name Format Notes
TXN_OUSTANDING I32 Number of transactions sent for which no reply

has yet been received
TABLE_UPDATE_FIRST I32 First Sequence Number Sdd read in the last

batch from Sybase
TABLE_UPDATE_LAST I32 Last Sequence number Sdd last read in the

last batch from Sybase
LAST_SOURCE_REF_ACKED I32 SOURCE_REF value from last TXN_ACK

message that contained one
LAST_SOURCE_REF_NACKED I32 SOURCE_REF value from last TXN_NACK

message that contained one.
SEQUENCE_NUM I32 SOURCE_REF of last transaction sent
SERVICE REPSDS There is one such block for each service from
the configuration file, formatted as shown
below.
SERVICE
NAME String Name of service
DOWNLOAD_ALL_COUNT I32 Total number of primary records retrieved from

Sybase
TXN_ADD I32 Total number of TXN_ADD sent

Page 44 Appendix A
TXN_MODIFY I32 Total number of TXN_MODIFY sent
TXN_DELETE I32 Total number of TXN_DELETE sent
TXN_REPLACE I32 Total number of TXN_REPLACE sent
MODIFIES_IGNORED I32 Total number of modifies discarded because

the data downloaded from Sybase was the
same as that held in fidessa.
5.1.3 Counters
None
5.1.4 Attributes
Attribute name Format Read/Write Notes
HEALTH I32 read only 0 connected to Sybase and

retrieving updates using the poll
mechanism
1 either, not connected to
Sybase, or still performing a
total download
HEALTH_TEXT string read only If HEALTH is not zero, then

contains a message saying why.
COMMAND string write only Set to DUMP to dump stats to

trace file.
5.2 FidToSyb OpenAccess Statistics

5.2.1 Class Name
<SERVICE_NAME>.INFO
5.2.2 Statistics
DB_SERVER string Name of Sybase SQL Server
DB_NAME string Name of Sybase database
DB_USER string Name of Sybase user

Appendix A Page 45
CONNECTED string Sybase connection flag (TRUE or FALSE)
CONNECT_TIME string Time of last Sybase connection
DISCONNECT_TIME string Time of last Sybase disconnection
NUM_CONNECT_FAILS I32 Number of failed Sybase connection attempts
NUM_COMMAND_FAILS I32 Number of failed Sybase commands (stored procs

or SQL statements)
SERVICE REPSDS Repeating SDS. Contains details for a FidToSyb

Service as below. The services are listed in the
order of their recovery, as configured.
TOTAL_RECOVERED I32 Total number of records recovered since start up.
TOTAL_UPDATES I32 Total number of records updated since start up.
LAST_TIMESTAMP string Timestamp of last record written to Sybase
LAST_UPDATE_TIME string Time that last record was written to Sybase
UP_TO_DATE I32 0 - one of the services is not up to date

1 - all services are up to date
MAX_UPDATE_TIME_LAG I32 Maximum time difference across all services (in

seconds) between the last rtd timestamp and the
last rtd timestamp uploaded to Sybase.
DATA_SYNC_STATUS REPSDS Repeating SDS. Contains details of the Data

Sync Status Requests issued, showing which are
ACTIVE and COMPLETE.
SERVICE
NAME string Service name (from the .cfg file)
NUM_RECOVERED I32 Number of records recovered since start up
NUM_UPDATES I32 Number of records updated since start up
LAST_RECOVERY_TIME string Timestamp of last record recovered
LAST_TIMESTAMP string Timestamp of last record written to Sybase
LAST_UPDATE_TIME string Time that last record was written to Sybase
LAST_RTD_TIMESTAMP string Timestamp of last record to read from rtd

Page 46 Appendix A
(sent or not)
TOP_RTD_TIMESTAMP string Timestamp of current last record in the

timestamp key.
UP_TO_DATE I32 0 - last record read from rtd is not the most
recent
1 - last record read from rtd is the most
recent, Sybase should be in sync.
DATA_SYNC_STATUS
SYNC_NAME string The name given to the Data Sync

Request.
TIMESTAMP string The time up to which the data should be

synchronised.
SPROC string The stored procedure to invoke when the

data is in sync.
STATUS string Status of the request, either ACTIVE, or

COMPLETE.
SERVICE string[] List of serivces to be synced.
5.2.3 Counters
None
5.2.4 Attributes
HEALTH string read only 0 connected to Sybase and

operating in update mode
Sybase, or still performing
catching up
HEALTH_TEXT string read only if HEALTH is not zero then


Appendix A Page 47
5.3 OAToDb OpenAccess Statistics

5.3.1 Class Name
<SERVICE_NAME>.INFO
5.3.2 Statistics
DB_SERVER string Name of Sybase SQL Server
DB_NAME string Name of Sybase database
DB_USER string Name of Sybase user
CONNECTED string Sybase connection flag (TRUE or FALSE)
CONNECT_TIME string Time of last Sybase connection
DISCONNECT_TIME string Time of last Sybase disconnection
NUM_COMMAND_FAILS I32 Number of failed Sybase commands (stored procs

or SQL statements)
TXN_ADD I32 Number of TXN_ADD messages received
TXN_MODIFY I32 Number of TXN_MODIFY messages received
TXN_DELETE I32 Number of TXN_DELETE messages received
TXN_ACK I32 Number of TXN_ACK messages sent
TXN_NACK I32 Number of TXN_NACK messages sent
SESSION_ABORTED I32 Number of abort session messages sent
5.3.3 Counters
None
5.3.4 Attributes
HEALTH I32 read only 0 connected to Sybase and

operating in update mode
Sybase, or still performing
catching up
HEALTH_TEXT string read only if HEALTH is not zero then


Page 48 Appendix A
COMMAND string write only Set to "DUMP" to cause stats to

be dumped to trace log

Appendix B Page 49
6. Appendix B
6.1 Built in subst ( ) substitutions in FidToSyb
The following words are reserved in Sybase and are substituted for the same word but with a z in
front of the name when FidToSyb refers to table and column names in SQL server commands:
ADD, ALL, ALTER, AND, ANY, ARITH_OVERFLOW, AS, ASC, AT, AUTHORIZATION,
AVG, BEGIN, BETWEEN, BREAK, BROWSE, BULK, BY, CASCADE, CHAR_CONVERT,
CHECK, CHECKPOINT, CLOSE, CLUSTERED, COMMIT, COMPUTE, CONFIRM,
CONSTRAINT, CONTINUE, CONTROLROW, CONVERT, COUNT, CREATE, CURRENT,
CURSOR, DATABASE, DATA_PGS, DBCC, DEALLOCATE, DECLARE, DEFAULT, DELETE,
DESC, DISK, DISTINCT, DOUBLE, DROP, DUMMY, DUMP, ELSE, END, ENDTRAN,
ERRLVL, ERROREXIT, ESCAPE, EXCEPT, EXEC, EXECUTE, EXISTS, EXIT, FETCH,
FILLFACTOR, FOR, FOREIGN, FROM, GOTO, GRANT, GROUP, HAVING, HOLDLOCK,
IDENTITY, IDENTITY_INSERT, IF, IN, INDEX, INSERT, INTERSECT, INTO, IS,
ISOLATION, KEY, KILL, LEVEL, LIKE, LINENO, LOAD, MAX, MIN, MIRROR,
MIRROREXIT, NATIONAL, NOHOLDLOCK, NONCLUSTERED, NOT, NULL,
NUMERIC_TRUNCATION, OF, OFF, OFFSETS, ON, ONCE, ONLY, OPEN, OPTION, OR,
ORDER, OVER, PERM, PERMANENT, PLAN, PRECISION, PREPARE, PRIMARY, PRINT,
PRIVILEGES, PROC, PROCEDURE, PROCESSEXIT, PUBLIC, RAISERROR,
READ,READTEXT, RECONFIGURE, REFERENCES, REPLACE, RESERVED_PGS, RETURN,
REVOKE, ROLE,ROLLBACK, ROWCNT, ROWCOUNT, ROWS, RULE, SAVE, SCHEMA,
SELECT, SET, SETUSER, SHARED, SHUTDOWN, SOME, STATISTICS, STRIPE, SUM,
SYB_IDENTITY, SYB_RESTREE, TABLE, TEMP, TEMPORARY, TEXTSIZE, TO, TRAN,
TRANSACTION, TRIGGER, TRUNCATE, TSEQUAL, UNION, UNIQUE, UPDATE,
USED_PGS, USER, USER_OPTION, USING, VALUES, VARYING, VIEW, WAITFOR,
WHERE, WHILE, WITH, WORK, WRITETEXT
For example, the USER table would be referred to in generated SQL

statements in FidToSyb as zUSER


Sybase Interface Processes Configuration Guide

Caricato da

Informazioni sul documento

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Sybase Interface Processes Configuration Guide

Caricato da

Copyright:

Formati disponibili

f I D E S S A

Making better use of trading technology

fidessa Product Infrastructure

Sybase Interface Processes

Issue Date: 2nd July 2001

Author: royalblue financial plc

The copyright in this work is vested in royalblue financial plc and

Copyright 2001 royalblue financial plc

fidessa is a registered trademark of

26 Jan 1998 1.0 First issue

New update regime for FidToSyb.

Modified config items for FidToSyb:

recoveryRewindMins (new, optional)

dbLoginTimeout (new, optional)

dbStaleTimeout (new, optional)

updPollPeriod (meaning altered)

upqMaxBatch (default changed, meaning altered)

pollPeriod (new, optional)

databaseMap (was missing from previous version of

29 Apr 1998 1.2 Added details on OAToDb

6 Jul 1998 1.3 Documented new FidToSyb configuration option flushAtClose

Documented amendment to new FidToSyb update regime

28 Oct 1998 1.4 Updated for infrastructure v4.2.2:

Documented new Sdd options dbServerName,

Explained that the default FidToSyb mapping list is not used

17 Nov 1998 1.5 Updated for infrastructure v4.2.3 patch A:

New Sdd option modifiedFieldsOnly documented.

21 Jun 1999 1.6 Updated for Infrastructure v4.3.1

New Sdd option upqMaxBatch documented.

21 Dec 1999 1.7 Updated for infrastructure v5.0.0

Sdd resync mode, status & nack sprocs, and

FidToSyb status & stream status sprocs, data synchronisation

OA Stats documented in Appendix A.

Corrected documentation of standbyActive config option

31 Mar 2000 1.8 Updated for infrastructure v5.1.2

22 Jan 2001 1.9 Updated for infrastructure v5.3.0

Added new replay and replayAction command line

Added section on FidToSyb replay mode.

FidToSyb, timestampSproc and primaryKeySproc

Added encryptedDbPassword for Sybase password encryption.

2.1 Design ............................................................................................................... 8

3.1 Design ............................................................................................................. 23

4.1 Design ............................................................................................................. 39

fidessa Product Infrastructure Copyright 2001 royalblue Issue: 1.10

5.1 Sdd OpenAccess Statistics............................................................................. 43

6.1 Built in subst ( ) substitutions in FidToSyb ...................................................... 49

Issue: 1.10 Copyright 2001 royalblue fidessa Product Infrastructure

Available data flows:

tcl program SybTcl

Open Access OAToSyb Sybase data

fidessa Product Infrastructure Copyright 2001 royalblue Issue: 1.10

Sybase SQL Server Fidessa

Static Sdd OA Txn

Sybase Open Fidessa

Issue: 1.10 Copyright 2001 royalblue fidessa Product Infrastructure

and to list a particular record, you might try:

FTSsddsel_ALT_COUNTERPARTY_ID RECORD, CTLA, VIEW

Tables: Triggers: Insert Table

fidessa Product Infrastructure Copyright 2001 royalblue Issue: 1.10

2. Waits for fidessa database available

3. Sends an OA TXN_GET_LAST_SOURCE_REF message to fidessa

4. Receives a response and saves it as last_sequence_number

6. Enter a loop that runs sddsel_TABLE_UPDATE sproc with the last_sequence_number

2.2.1 Download Modes