Just in Time (JIT) Business Rules Mining When Time is All You Have and the Documentation Just

Isnt There Shirley J. Sartin, PMP Abstract--- To maintain an application or reengineer it, you need to understand its functions or background processes and the business rules that drive them. However, there are times when supporting documentation is unavailable or insufficient to aid you. Without the budget for consultants or production solutions, how can the analyst manually mine the business rules? This paper provides methods for determining functions, events, and inferred business rules for existing applications or background processes, and describes how to derive and verify a logical process model from source code or a user interface. I. Introduction At the Seventh International Business Rules Forum (2004), Terry Moriarty spoke about current state business rules. In the session Get Ready for Your Business Rules Management, Ms. Moriarty asked how much of the current state Business Rules (BRs) should be harvested and suggested that we could reengineer to a new vision. At the least, existing systems could be engineered just enough to understand the current environment as the business understands itself. But then again, Ms. Moriarty wondered about the dead or forgotten business rules and suggested that maybe we could pull out just enough information to understand and cross link the discovered BRs with the new system. However, many analysts do not have access to data, synchronized code, or analysis artifacts. Harvesting current business rules by reverse engineering is sometimes necessary, and vendor tools are available to perform this task. These tools can be very expensive, but without this specialized software to automatically mine the business rules, what can analysts do? How can they verify that the inherent BRs found in the code are true and current? By returning to the basics of systems analysis and reverse engineering using only source code and/or Graphical User Interfaces (GUIs), a logical process model based on event-driven processes may be manually constructed and used to derive business processing requirements and BRs. Verification and validation of the resultant requirements and rules with both business and Information Technology (IT) application development personnel will serve to ensure inferred BRs are on target. This paper relates methods and steps used to determine functions, events, and inferred BRs by logical process modeling of existing applications or background processes for the following:
1

An e-Commerce Web site with no viewable source code. An order processing application built with Powersofts PowerBuilder. 1

PowerBuilder, a GUI development tool from Powersoft, is used for developing client-server database applications in PowerScript, a language loosely based on BASIC.

Structured Query Language (SQL)2 stored procedures automatically pushing master account data from one system to another.

The logical process model developed is an interim step in system design and may later be transformed into a physical process model (or application schema) for the chosen technical architecture. The next section provides steps in reverse engineering a department stores online catalog. Section III discusses manually mining BRs from events. Client-server application reverse engineering is covered in Section IV, Structured Query Language (SQL) stored procedures are discussed in Section V, and additional information is provided concerning the use of help text and training materials in Section VI. The paper concludes in Section VII. A glossary is provided in Appendix A and information concerning building a context diagram is presented in Appendix B. II. Reverse Engineering Using Logical Process Engineering The logical process model, also known as the conceptual or business model, has a process focus and a system owner and/or system-user perspective and is not concerned with implementation details or technology. Models are created using event partitioning, where a system is separated into subsystems based on business events and responses to those events. 3 A logical process model is especially effective for determining BRs because this type of modeling does not include redundant activities, is not technology dependent, and has no physical limitations or requirements. The completed model presents a business view of activities from which BRs may be inferred. This section provides details for creating a logical process model while reverse engineering an online catalog. The instructions provided may be used for building most logical process models. 4 Logical Process Modeling Artifacts In order to create a logical process model, a systems logic must be organized and documented. Since it is considered to be a conceptual or business model, the logical process model is intended to show what a system does, not how it is done, and thus does not reflect implementation of a technical solution. In creating a logical model for use in BR mining, specific steps are followed to produce the following artifacts: 5 System Context Diagram Indicates the events or transactions the system must respond to; system responses, sources and destinations for both; and the identity of all external data stores.

Structured Query Language (SQL) is an industry language for creating, updating, and querying relational database management systems. 3 Jeffrey L. Whitten and Lonnie D. Bentley, Systems Analysis and Design Methods, (Irwin/McGraw Hill, 1998): 236. 4 Primitive and system diagrams are typically created during logical process modeling. However, these diagrams may not be necessary for determining BRs, and thus the logical process modeling efforts may end after building all event diagrams. 5 Whitten and Bentley, Systems Analysis and Design Methods, 237.

Functional Decomposition Diagram Provides top-down functional decomposition or structure of the system. Event-Response List Outlines the business events the system must respond to along with appropriate responses. Also serves to identify how input data flow will occur for the system. Event Diagrams Derived from a combination of the event list and a further partitioning of the functional decomposition diagram, show inputs, outputs, and data store interactions for each event.

Reverse Engineering Steps The artifacts created during logical process modeling are used in analysis verification and validation (V&V) efforts and are invaluable for tracing features/functions through the process diagrams. Basic steps used when reverse engineering using logical process modeling methods are listed below.6 Appendix A provides a glossary of process modeling terms and definitions. 1) Identified Features A logical process model was built by first identifying features of the software or process. These features could be considered the bullet points on a shrinkwrapped software box. By looking at the Web pages of the department stores online catalog, the following features were identified: Search Product (by Category/Department) View Product List (by Department) Create/Manage Account Manage Shopping Cart View Store Locations and Events Status of Orders Inquire on Shipping Methods & Charges Login to Site Apply for Credit

Each of these features was considered to be a separate subsystem or process. Focus was placed on only one process at a time for each process modeling step, below, through to the final step where an event diagram was created for all processes. By focusing on individual processes, the analysis effort was more manageable. 7

Detailed instructions for developing logical process models may be found in systems analysis or design method textbooks. 7 Only the Create/Manage Account process modeling is documented in this paper.

2) Determined Functions In reviewing the online catalog site, the Create/Manage Account process appeared to provide for customer accounts with the functions listed in Table 1, below. These functions were identified by stepping through the online catalogs web pages and noting the actions that occurred. Function Create Account Remind Password Select Profile Functions Description Create user account. E-mail password hint to existing customer. Select a profile function: Update Account Profile Access Shopping Bag Review Order History/Status Setup Express Checkout

Table 1. Create/Manage Account Functions

3) Created Context Diagram By considering context diagram symbol descriptions (Appendix B) and reviewing each of the functions in Table 1, inputs, outputs, data flows, data stores and functions initiated by external agents of the Create/Manage Account process were identified and placed in Table 2, below. This information was determined by reviewing the online catalog pages in more detail.
Process (Feature) External Agents New Customer Sub Processes (Functions) Create Account Events/Responses External Data Stores

Remind Password Create / Manage Account Select Profile Function: Update Account Profile Access Shopping Bag Review Order History/ Status Setup Express Checkout

Existing Customer

Register Account User Data8 New Acct Info Account Info Creation Ask for Password User Data E-mail Address Get Password Reminder Function Selection Maintain Account User Data Account Info Updates Maintain Account User Data Order History/Status Orders Maintain Account User Data Express Checkout Shipping/Billing Maintain Account Shopping Bag Shopping Bag Items

Table 2. Create/Manage Account Function Information

The external data stores User Data, Shopping Bag, Order Table, and Shipping/Billing implied data relating to events/responses. In actuality, a data store may exist but the name is unknown at this point in the modeling process.

By using information in Table 2, above, and context diagram symbols in Appendix B, the context diagram in Figure 1, below was created.

Figure 1. Create/Manage Account Context Diagram

This context diagram provided an understanding of the Create/Manage Account process. It assisted in the next logical process modeling step where the sub processes were partitioned into a functional decomposition diagram.

4) Created Functional Decomposition Diagram A functional decomposition diagram (also known as a hierarchy chart) was drawn to partition the process into logical functions thus providing the beginnings of an outline for drawing data flow diagrams. The functional decomposition diagram provided a top-down functional decomposition or structure of the system. 9 The Create/Manage Account context diagram was decomposed into sub processes (functions). As shown in Figure 2, below, the sub processes were represented with meaningful names.10

Figure 2. Create/Manage Account Functional Decomposition Diagram

10

Whitten and Bentley, Systems Analysis and Design Methods, 237. Where necessary, processes are divided into sub-processes.

5) Created Event Decomposition Diagram By focusing on events in the Create Account function (reference dashed area in context diagram Figure 3, below), events were decomposed and compiled to identify and confirm business events to which the system must provide a response.11 The completed Event Decomposition Diagram appears in Figure 4.
User Name Account Info Updates Order History/ Status Account Info Creation Acct Info

User Data

Orders

Remind Password
E-mail Address

Select Profile Function

Express Checkout

Shipping/ Billing

Functi on

Create Account

New Acct Info

Function Selection

Shopping Bag Items

Shopping Bag

Register Account

Create/Manage Account

New Customer

Ask for Password

Maintain Account

Existing Customer

Figure 3. Create Account Function

11

Whitten and Bentley, Systems Analysis and Design Methods, 237, 245.

Derived from a combination of the context diagram (Figure 3, above) and a further partitioning of the functional decomposition diagram (Figure 2, above), the event decomposition diagram below was created to show decomposition of the Create Account function and its associated events tagged with unique numbers. These numbers relate to event diagrams or data flow diagrams in Step 7.

Figure 4. Create Account Event Decomposition Diagram

6) Event Decomposition Diagram In following the Create Account function in the context and event decomposition diagrams (Figures 3 and 4, above), triggers and responses for each event were determined as listed in Table 3, below. The event-response list outlines the business events the system responded to along with the appropriate responses. This list also served to identify how input data flow occurred for the system.
Event RA-01 Register Account RA-02 New Account Info Event Description New customer chooses option to register a new account. New customer provides account information. Trigger (Inputs) Create/Manage Account Module (View: Login Page) Choose Register Option Create/Manage Account Module (View: Registration Form) Provide account information: User Data User Name E-mail Address Password Zip/Postal Code E-mail Update Selection RA-03 Account Info Creation System writes data to user data store and provides account validation. Create Account Sub-Module (Background Process) Register User Data Create Account Sub-Module Validation/verification form inputs. (View: Error or acceptance messages.) Responses (Outputs) Create/Manage Account Module (View: Registration Form)

Create Account Sub-Module Validation/verification user data. (View: Welcome new user Error/Acceptance Message.)

Table 3. Create Account Event-Response List

7) Event Diagrams The following individual diagrams were created for each event in the event decomposition diagram. These diagrams depict elementary processes, inputs, outputs, and data store interactions, where applicable, for single events of the decomposed Create Account function (Figure 4, above).

Figure 5. RA-01 Choose Register Option Event Diagram

Figure 6. RA-02 Provide Account Info Event Diagram

Figure 7. RA-03 Register User Data Event Diagram

Individual event diagrams were then used to manually mine BRs for the Create Account function. The next section discusses how BRs may be determined using these online catalog event diagrams. III. Manually Mining Business Rules Ronald G. Ross noted, Rules in the business rule approach must be perceived and expressed declaratively, independent of processes and procedures.12 Modeling an existing systems processes, such as the online catalog, and decomposing its functions to single events in the previous section assisted in elevating an understanding from the business solution system to a business problem. Mr. Ross also indicated that rules must be enforced when certain events occur.13 By examining the business problem(s) with a focus on individual events, it became

12 13

Ronald G. Ross, Principles of the Business Rule Approach, (Addison-Wesley, 2003): 70. Ibid, 70.

10

easier to identify those certain events. Narrowing the focus from high level functions to individual events aided in identifying critical events and their associated BRs.14 In Figure 5, above, it was determined that a new customer must register an account. When the new customer chose the option to register an account, they were presented with a registration form. Should a BR be considered for this event? Since the external agent (entity) was identified as New Customer and the action was to provide a registration form, the BR could have been: Customers must have accounts to access the online catalog.

Figure 6, above, depicted the event where a new customer provided account information, and the New Account Info event (RA-02) returned either an error or an acceptance message. A BR for this event might have been: Only valid account information will be accepted.

While building a data flow diagram for the event Account Info Creation (RA-03), it was determined that this event should include a data store. As shown in Figure 7, this event passes validated user data to the User Data data store and either a Welcome New User or a System Error message is returned to the New Customer. BRs for this event could have been: Registered user account information must be maintained in a system data store. User account information must be valid.

Once all features had been processed as in the Create/Manage Account process, above, and all event diagrams for functions and identifying potential business rules were thought to be discovered, the logical process model was completed by creating a system diagram (not discussed in this paper). All discovered BRs were verified with the business users and a rule repository created and populated with the discovered BRs. Note that a repository could be a spreadsheet, a database, or tables in a word processing document. Reverse engineering by logical process modeling has also been used for client-server systems, as discussed in the next section. IV. Reverse Engineering a Client-Server System The previously documented logical process modeling methods were used to reverse engineer an online catalog. These same modeling methods were also used while reverse engineering a telecommunication companys order processing system built with PowerBuilder.

14

Interpretations of these events may vary depending upon the business. BRs created during the logical process modeling exercise above, serve as examples of BR mining and may not, in fact, be actual BRs created during a true modeling situation.

11

Logical process model diagrams were created based on features discovered by stepping through GUIs and reviewing help text and training documentation. An objective of this project was to provide enough technical details so the development staff could correct application defects; thus it became necessary to identify technical details such as table names, field attributes, etc. By using PB Code Analyzer,15 cross references (relationships) to objects were discovered and the tools Table Locator function identified tables used by each of the objects. Objects identified by the tool were included as external agents in the logical process model. For example, in the Edit Request event, external agents were identified as the Request Originator (a person), and the two objects PROV_DL and NUM_ASGN were provided the meaningful names Provisioning Directory Listing and Number Assignment. The analyzer tool also served to verify functions/events. The tables used in this event were identified by the tool and appeared in the event response list as below:
Event Edit Request Event Description Request Originator, Provisioning Directory Listing (PROV_DL) and Number Assignment (NUM_ASGN) edits general information or creates number of lines on open number reservation request. Trigger (Inputs) Number Reservation Module (View: Open Reservation Request, double-click on selected request, perform edit in left frame, click Save icon.) General Info Update Line Count Increase Table 4. Edit Request Event on the Event-Response List Responses (Outputs) Open Request general information is updated.

The Edit Request event (2-4d) was represented in an event diagram as follows:

Figure 8. Edit Request Event Diagram

Once all modeling efforts were completed and BRs were identified, they were verified with the business users and the development team.
15

Ascension Labs, LLC.

12

SQL stored procedures automatically pushing master account data from one system to another were reverse engineered using this logical process modeling method. The next section provides information concerning process modeling for SQL procedures. V. Reverse Engineering SQL Stored Procedures Mainframe legacy systems are sometimes used to host master account information. With the mainframe acting as a data repository, the account information is pushed on a scheduled basis to subsidiary repositories on other platforms. This pushed data is consumed by applications not located on the mainframe platform. During a reengineering project, reverse engineering became necessary in order to determine the processes and BRs of the stored procedures pushing master data into tables on a SQL server. These procedures involved one set of 10 SQL procedures and one set of seven SQL procedures accessing and pushing data from two separate data sources. The procedures were created by a consultant in 1999, and only a brief definition of the SQL steps and one process flow were available as documentation. It was acknowledged that the original code had undergone several changes, but it could not be readily determined what BRs were underlying the code. By using the limited documentation and reviewing comments embedded in the code, a potential list of functions and events were discovered. These functions and events were then used to build a partial logical process model including a context diagram and a functional decomposition diagram. SQL keywords INSERT INTO, CREATE TABLE, and UPDATE were considered events in the procedures. Code leading up to the keywords was reviewed to document the actual event that was occurring. Eventresponse lists were then created and used in hyperlinked word processing documents for use when reviewing events linked to specific code. V&V of the process model documentation and code was held with the Data Base Architect and a Data Base Administrator (DBA) to verify code interpretation. Upon approval of the process model, the Analyst, Project Manager and DBA reviewed the approved hyperlinked documents and derived BRs. Business users verified the inferred BRs and the reengineering project continued. VI. Help Text and Training Materials The cases provided in this paper described reverse engineering by viewing online catalog pages, stepping through a client-server system using help text along with analyzed code, and by reviewing SQL program code. When preparing to reverse engineer, it is also helpful to gather documentation from sources such as help text or training materials. The help text table of contents assisted with reverse engineering the client-server system because it was organized by system features. These system features narrowed the focus for the logical process modeling effort. Even without a table of contents, the help text alone would have provided assistance since it contained information concerning system functionality. Depending on format and content, training materials may also be helpful in identifying features or functions.

13

VII. Conclusion In order to provide effective software maintenance and/or reengineering, it was necessary to understand functions of a software application or system background processes and the business rules that drove them. When support documentation was not available or insufficient for this purpose, the Analyst was able to manually mine BRs by using proven methods and steps, thus combining comprehensive business requirements analysis with logical process modeling.

14

Appendix A. Glossary Term Context Diagram Definition Defines the scope and boundary for one process and identifies inputs and outputs. External agents of the process are indicated as well as data stores and interactions with the process. Logical unit of work that must be completed as a whole. Data flow diagrams of individual events based on the context diagram. Factors a system into subsystems based on business events and responses to those events. External agents/entities are the source for net inputs into the process and/or destination for net outputs from the process. Set of related ongoing activities of the business that group logically related activities and tasks. Activities that have a beginning and an end, transform data, and can be decomposed into other processes or into events. High level diagram showing a composite of all events in the system or subsystem (optional for BR discovery). The system diagram is constructed by merging event diagrams.

Event

Event Diagrams

Event Partitioning

External Agent

Function

Processes

System Diagram

A-1

Appendix B. Developing a Context Diagram Context diagrams are built using diagram symbols as listed in Table 5, below. 16 Diagram Symbol How Used

Process a single feature. Process Sub process(es) are functions of the single feature or process. Each of the sub processes identified will be used in the functional diagram to decompose events/responses that have been identified. External agents/entities are the source for net inputs into the process and/or destination for net outputs from the process. External data stores are identified to assist with understanding the logical process. Individual (primitive) event diagrams will be created for each event in the event decomposition diagram and will show net inputs, net outputs, and data store interactions for each sub process. Data flows indicate events and responses for each of the processes and sub processes. These events/responses will be used during the functional decomposition of each process/sub process.
Table 5. Context Diagram Symbols and Their Use

Sub Process

External Agents

External Data Stores

Events/Responses

16

Whitten and Bentley, Systems Analysis and Design Methods, 241.

B-1

Using Table 2. Create/Manage Account Function Information and applying the context diagram symbols in Table 5, above, a context drawing was created (Figure 9, below). Note the context diagram shows only one high level process (feature) as Create/Manage Account and all associated sub processes (functions) as Create Account, Remind Password, and Select Profile Function. The Create Account functions events are called out as Register Account, New Acct Info, and Account Info Creation.
User Name

User Data

Account Info Updates Order History/ Status

Orders

Get PW Account Info Creation Reminder

Remind Password
E-mail Address New Acct Info

Events
Create Account

Select Profile Function

Express Checkout

Shipping/ Billing

Function Selection

Shopping Bag Items

Shopping Bag

Sub Process (Function)

Register Account

Create/Manage Account

New Customer

Proc (Fea ess ture )

Ask for Password

Maintain Account

Existing Customer

External Agents
Figure 9. Online Catalog Create/Manage Account Context Diagram

B-2

Shirley J. Sartin, PMP (BSBA MIS, MS CIS) is currently performing requirements analysis. Her 20+ years in Information Technology have encompassed roles in project management, systems analysis, business analysis, programming analysis, systems administration, operations administration, and technical training in a variety of industries, including broadband cable, telecommunications, aerospace engineering, engineering product development, software, and transportation. Skilled in all aspects of IT including back- and front-office, Shirley effectively interfaces with development groups and customers with a focus on empathizing and appreciating the needs of the end-user. She can be reached at shirleysartin@hotmail.com.