A case study in IT audits program risk management role

on a large-scale system implementation

By Scott Kamenick, Protiviti Managing Director
July 16, 2007

Overhauling technology application systems is not for the faint of heart. With the careful guidance of
internal audit, however, this fearsome task can be an opportunity for positive change. The numbers
associated with a recent application upgrade at a financial institution were unusually large, and the
challenges were somewhat uncommon. However, the take-home lessons from this experience can be
applied to nearly any organization looking to overhaul its systems and related business processes.
This particular case study involved a bank with $6 billion in assets, much of it centered on a regional
service area. The bank, which had some 1,500 employees, was undertaking a significant upgrade of its
entire underlying technology infrastructure. Approximately 30 project teams were involved in this
endeavor, each addressing a separate, full-blown system implementation or project. Each team was
divided into different application development and delivery teams. To complicate matters, the project
teams in the application group consisted of company personnel; a system integrator (from a professional
services firm hired to assist with development and deployment); and, where applicable, professionals
from associated vendors.
The result of this structure was three different people, three different approaches, and three different
personalities creating an interesting dynamic.
Big-bang approach
The project teams charge was nothing less than massivea big-bang theory of sorts. Among the
application systems being implemented were all core banking systems (including commercial loans,
consumer loan servicing, customer information, deposits and service delivery); financial systems (Oracle
Financials); customer relationship management (Onyx); items processing (AFS); data warehouse; and
voice response unit.
On the infrastructure side, the company was moving from a predominantly mainframe environment to a
client server setupa significant change, as the new user interface required different skills. Bank tellers
literally had to learn how to use a mouse and a PC after years of using green screens and typing
commands using shortcut keys.
Obviously, the technology leap was huge, and the scope highly ambitious. Where did internal audit fit into
this complex puzzle?
Originally the IT audit plan called for a pre-implementation review, which was easier said than done.
Given the scope of the effort, the obvious question became, pre-implementation of what? Preimplementation of one of the aforementioned 30 projects, all 30 or some combination?

2007 Protiviti Inc. All rights reserved. An Equal Opportunity Employer

Page 1

Ultimately, IT audit looked at the project plan to overhaul every system. The focus was on the program
management office: how it performed tracking, risk monitoring, status reporting, project planning and the
integration of the different project teams to set proper milestones, all with some basis in Project
Management Institute (PMI) concepts. IT audit also looked at overall systems in terms of the system
development life cycle (SDLC), based on their design and possible risks. The approach was, in essence,
more process-oriented as opposed to monitoring the functionality of specific applications.
The risk areas identified included:

Program management
Project planning and monitoring
Resource management
Scope management
Change management (e.g., scope, schedule, cost, quality, risk or contract administration)
Issue monitoring and management
Risk monitoring and management
Quality assurance and control (including testing)
Informal SDLC policies and procedures
Security

Certainly the pre-implementation review created more questions than answers. For example, no test
strategy had been defined, which raised the simple questions: What is meant by test? Is it unit testing?
System testing? Security testing? Control testing? Even within a specific areasystem testing, for
exampledefinitions of concepts such as pass and fail were non-existent.
Such informal adherence to SDLC policies and procedures empowered members of each project team to
develop and deploy a system as they saw fit. That led to other issues, including securityor lack of it.
Security had not been considered, so there were no requirements for access controls within the new
systems.
IT audits observations and recommendations were met with no small amount of resistance. The system
integratorarguably the party with the most to lose in this scenariocontended that IT audits approach
was one of many possible approaches, ignoring the fact that any system integration carries with it core
requirements. The primary vendor agreed with the system integrator. Likewise, management responses
varied and were often driven by relationships, personal experiences and, occasionally, personal agendas
or politics.
A new approach
Shortly after the aforementioned issues were summarized and reported to the audit committee, the
organization underwent a re-planning of all project team activities:

Overall program milestones were established, and formal resource requirements were developed;
A test strategy was created;
A more formal issue-tracking process was deployed; and
Definitions were created for application-level security requirements.

Where did that leave IT audit? It assumed the role of program risk monitoring and management,
essentially rolling up its sleeves, joining the program management office and forcing management to deal
with appropriate risks.

2007 Protiviti Inc. All rights reserved. An Equal Opportunity Employer

Page 2

This transformation was supported by an internal audit director who believed internal audit could and
should serve in a consultative role, and by internal audits own professionalism, responsiveness, flexibility
and relevance. It was also becoming clear that risk management was a vital part of the overhaul program
with clear potential business impacts.
IT audits new role had three distinct aspects:

The periodic risk scorecards, which identified 12 key business risks and key controls to help
mitigate each of these risks. Every two weeks, IT audit assessed whether those controls were in
place; and, to the extent they were not in place, it would describe the risk to the program or track
the risk itself.

The readiness review, which entailed validating management progress toward defined exit and
entrance criteria. This was a lesson learned from the pre-implementation review, during which,
as previously mentioned, the system integrator, client and vendor questioned IT audits basis for
evaluation. Thus, the readiness review was anchored on company-defined exit and entrance
criteria that had been approved by the executive steering committee (the sponsor of the overall
project).
It should be noted that many of the must-have exit/entrance criteria did not exist or were
incomplete during the period prior to go-live. The readiness review ended with a presentation to
an executive team during a series of go/no-go meetings; internal audit was one of perhaps 20
groups that appeared before this executive team. Internal audits presentation was fact-based; it
did not offer a decision regarding whether to go-live, but rather stated the matter was a business
decision for the executive team to make. Internal audits role was to inform them of the risks.

Transition risk management, which took place in the command center during conversion. This
occurred during two and a half days of non-stop integrated, sequential-dependent sets of tasks.
IT audit played a role in tracking certain controls, under a risk management/monitoring function.
It verified that documentation for validation of data conversion was obtained, with appropriate
sign-off. It monitored the issue list to make sure each issue was assigned, worked and resolved.
IT audit continued in this role several weeks after implementation as well, as the organization
continued to triage urgent issues.

Lessons learned
This case was extraordinary in some ways, but quite ordinary in others. While the scope may have been
extreme, the issues are common to large-scale system implementations. Regardless of size, any such
project requires planning and rigor because its significance to the organization is enormous. Throughout
the entire process, keep the following in mind:

Regardless of the size and scope of a systems overhauland this one was enormousit is
incumbent on IT audit to become engaged early and often. In this case, IT audit came aboard
approximately one year into the project. An earlier intervention may have helped stem the tide of
difficulties encountered.

IT audit should always address how it is perceived within the organization and promote itself as
internal consultants. Internal auditors are service providers, a role that requires a high degree of
customer service. That, in turn, demands that auditors develop professional, responsive and
relevant relationships with auditees.

When looking at big system changes, IT auditors should make sure risk management is an
element of every implementation. From a business perspective, the involvement of multiple
teams and players requires a common process and language.

2007 Protiviti Inc. All rights reserved. An Equal Opportunity Employer

Page 3

IT audit also needs to help establish the key, interconnected criteria that will help an organization
move through each step and phase of the process. These are the key control points that will
make an organization comfortable with moving forward. It is critical to assess those criteria and
ensure they are met. If they are not met, it is important to provide justification and a documented
rationale for moving forward (if the decision to do so is made). Too often organizations establish
such criteria upfront, but then overlook it in their haste to press ahead.

Article from Protiviti KnowledgeLeader www.knowledgleader.com.

KnowledgeLeader is a subscription-based website that provides audit programs, checklists, tools, resources and best
practices to help internal auditors and risk management professionals save time, manage risk, and add value. Free
30-day trials available.

Protiviti is a leading provider of truly independent internal audit and business and technology
risk consulting services. We help clients identify, measure and manage operational and
technology-related risks they face within their industries and throughout their systems and
processes. And we offer a full spectrum of audit services, technologies and skills for business
risk management and the continual transformation of internal audit functions.
Protiviti is not licensed or registered as a public accounting firm and does not issue opinions
on financial statements or offer attestation services.

2007 Protiviti Inc. All rights reserved. An Equal Opportunity Employer

Page 4