IT Measurement, IT Audit and

Evaluation

IT Measurement

Objectives
Management (IT and business) can be kept
informed of the activity and progress of the IT
organization.

Introduction
Measurement is needed to measure how well IT
is performing or wheter IT is meeting the
demands of the firm
In general organizations want to measure, to
know how effective and efficient the Information
technology performing
The Challenge is choosing what and when to
measure:
Financial measure value of IT, but do not
provide indication of how well the process being
performed and how they can be enchanced

Introduction
Questions :
Is the process meeting the needs of the business
partner ?
Is the process operating a its optimum levels?
Is the process often non functional or inoperable ?

MEASURING IT

Pressure to IT managers
Evaluate and justify the contribution of IT
Expenditures to:
Productivity
Quality
Competitiveness of the organizations

Frequently IT is used without a full understanding

of its: Applicability, effectiveness or efficiency.
Lack the tools needed to decide if they are
accomplishing the right activities.

Overview of Principal IT Evaluation

Models: Tools For IT Auditors
Clarence Kimpton and Denys Martin, FCPA, CIA
http://www.isaca.org/Journal/PastIssues/2001/Volume-5/Pages/Overview-of-PrincipalIT-Evaluation-Models.aspx

Frameworks, Models and sets of best

practises
1. Control Objectives for Information and related Technology (COBIT), Information
Systems Audit and Control Foundation (ISACF) and IT Governance Institute (ITGI)
2. Information Technology Control Guidelines, Canadian Institute of Chartered
Accountants (CICA)
3. Information Technology Investment Management: Framework for Assessing and
Improving Process Maturity, United States General Accounting Office (US GAO)
4. SysTrustMS/MD Principles and Criteria of System Reliability, Canadian Institute of
Chartered Accountants (CICA) and American Institute of Certified Public
Accountants (AICPA)
5. Control Objectives for Information and related Technology (COBIT), Information
Systems Audit and Control Foundation (ISACF) and IT Governance Institute (ITGI)
6. Information Technology Control Guidelines, Canadian Institute of Chartered
Accountants (CICA)

Frameworks, Models and sets of best

practises
7. Information Technology Investment Management: Framework for Assessing and
Improving Process Maturity, United States General Accounting Office (US GAO)
8. SysTrustMS/MD Principles and Criteria of System Reliability, Canadian Institute of
Chartered Accountants (CICA) and American Institute of Certified Public
Accountants (AICPA)
9. Software Development Capability Maturity Model, Software Engineering Institute
(SEI)
10. Managing Systems Information: A Practical Assessment Tool, Information
Technology Resources Board (ITRB)
11. A Guide to the Project Management Body of Knowledge, Project Management
Institute (PMI)
12. Systems Security Engineering Capability Maturity Model, United States National
Security Agency (US NSA)
13. Information Security Management: Learning From Leading Agencies, United States
General Accounting Office (US GAO)

CONTROL OBJECTIVES FOR INFORMATION AND

RELATED TECHNOLOGY
(COBIT)

Overview
COBIT consists of a framework of IT processes and
control objectives that can be implemented to control,
audit and manage the IT organization.
The framework is based on best practices and audit
and control of information systems. It particularly aims
at helping leaders understand and manage the risks
relating to IT and the links between the management
processes, the technical questions, the need for control
and the risks.
COBIT is structured around four main fields of
management implying 34 processes of management
associated with information technology.

Four main fields of management

Planning and Organization

Acquisition and Implementation
Delivery and Support
Monitoring

Planning and Organization

1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.

Define a strategic IT plan

Define the information architecture
Determine the technological direction
Define the IT organization and relationships
Manage the IT investment
Communicate management aims and direction
Manage human resources
Ensure compliance with external requirements
Assess risks
Manage projects
Manage quality

Acquisition and Implementation

12.Identify automated solutions
13.Acquire and maintain application software
14.Acquire and maintain technology
infrastructure
15.Develop and maintain procedures
16.Install and accredit systems
17.Manage changes

Delivery and Support

18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.

Define and manage service levels

Manage third-party services
Manage performance and capacity
Ensure continuous service
Ensure systems security
Identify and allocate costs
Educate and train users
Assist and advise customers
Manage the configuration
Manage problems and incidents
Manage data
Manage facilities
Manage operations

Monitoring
31.Monitor the processes
32.Assess internal control adequacy
33.Obtain independent assurance
34.Provide for independent audit

INFORMATION TECHNOLOGY
CONTROL GUIDELINES (CICA)

Overview
This model, developed by the Canadian Institute of
Chartered Accountants (CICA), is based on the concept of
organizational roles and establishes responsibilities for
security and control processes.
In this context, the roles are classified into seven
categories:
1.
2.
3.
4.
5.
6.
7.

Senior management
CIO
Owners
Stakeholders
End users of the information systems
Suppliers of services for development
IT operations and systems support

Activities associated with each organizational Role

Roles
Senior management

Activities
Approve the strategies, policies and standards; define
responsibilities; and develop and approve the business plans.

Chief Information Officer

Develop of the strategies, policies and standards; direct the

technical support services; and direct centralized services.

Owners

Define requirements; maintain control and security; evaluate

risks; classify the level of control and security for each type of
activity; and delegate tasks.

Stakeholders

Comply with the policies and standards; authorize logical and

physical access; and control changes.

End Users

Comply with the requirements of the owners; maintain IT

resources.

Suppliers of services development

Develop and acquire application systems;comply with policies

and standards.

Suppliers of ServicesIT operations

and systems support

Define service level agreements; plan and operations systems;

manage the problems; develop safeguards, disaster recovery
plans and systems support.

 The model addresses the concepts of authority,

responsibility and accountability.
The model specifies responsibilities regarding the
management of risks and related controls, and then
proposes an assessment framework for evaluating these
controls, consisting of objectives, standards and techniques
for the following areas:

IT planning
Acquisition, development and maintenance of IT systems
IT operations and support of systems
IT security
Continuity plans and resumption of IT services
Applications controls

INFORMATION TECHNOLOGY INVESTMENT

MANAGEMENT: FRAMEWORK FOR ASSESSING AND
IMPROVING PROCESS MATURITY

Overview
This model, developed by the US General Accounting
Office, identifies the critical processes that ensure the
success of investments in IT and organizes them
around five levels of maturity.
This model focuses on investments in IT, according to
the following phases:
Selection of the projects: determination of the projects
that best support the mission of the organization, by taking
into account the risks and returns on the investment
Control of the projects: assurance that the projects
continue to meet the needs and the required service levels
Evaluation of the projects: comparison of the anticipated
and reached results

Five levels of maturity and the fifteen

critical processes
Level of maturity
1 Creating investment awareness
2 Building the bases for investment

3 Developing a complete
investment portfolio

4 Improving the investment process

5 Leveraging IT for strategic outcomes

Critical processes
IT expenditure without a structured investment
processes
IT investment board operations
IT investment board operations
IT project oversight
IT asset alignment
Business needs identification for IT projects
Proposal selection
IT investment board
Portfolio selection criteria definition
Investment analysis
Development portfolio
Portfolio oversight performance
Post-implementation reviews and feedback
Portfolio performance evaluation and improvement
Investment benchmarking process
IT-driven strategic business changes
In addition, the GAO provides a structured process of
evaluation using the above described model.

SYSTRUST PRINCIPLES AND

CRITERIA OF SYSTEM RELIABILITY

Overview
The American Institute of Public Certified
Accountants (AICPA) and the Canadian Institute
of Chartered Accountants (CICA) have jointly
developed SysTrust, a framework for evaluating
the reliability of information systems.
The two organizations also are offering an
education program to train assessors in providing
a certification of IT organizations based on
SysTrust principles.

Four essential principles:

Availabilityability of the systems to meet
functionality and performance requirements in all
situations including peak times
Securityadequate and up-to-date preventive and
detective measures to dynamically protect the systems
Integritymeasures to ensure the data entered are
correct and cannot be altered unless authorized
Maintainabilitycapacity of the systems to upgrade
and expand to accommodate a higher volume of traffic
or more complex tasks

The criteria established

The criteria established for the four principles consist of:
The definition and documentation of the performance
objectives, the policies and standards for evaluating the
expected performance of the systems, the involvement of
the entity's senior staff and their means of communicating
with staff to ensure proper direction
The implementation procedure, with the goal of achieving
the performance objectives in accordance with policies and
standards
The activities for monitoring the systems and business
environment, allowing for the identification of any
potential performance degradation and the
implementation of appropriate corrective initiatives

SOFTWARE DEVELOPMENT
CAPABILITY MATURITY MODEL

Overview
This model was developed by the Software Engineering
Institute (SEI) of the Carnegie Mellon University in
Pittsburgh, PA, USA.
It is used to evaluate the capability of the IT
organization in the development and maintenance of
information systems.
It includes 18 key sectors to be addressed at five levels
of maturity
The SEI capability maturity model provided some of
the basic concepts used in developing process maturity
models for the 34 IT processes of COBIT, as published
in the COBIT Management Guidelines in July 2000.

Level of maturity
1 Initial
2 Can be replicated

3 Defined

4 Controlled
5 Optimized

Key sectors
None
Management of needs and requirements
Planning of software development projects
Follow-up and supervision of the software
development projects
Management of subcontract software
Quality assurance of software
Software configuration management
Organizational focus on the processes
Definition of the organization's processes
Training approach
Software integration management
Software engineering of products
Coordination joint committee
Peer review
Quantitative management of processes
Software quality management
Prevention of deficiencies and weaknesses
Management of technological changes
Management of process changes

MANAGING INFORMATION SYSTEMS: A

PRACTICAL ASSESSMENT TOOL

Overview
This reference work is the result of the
accumulated experiences of the Information
Technology Resources Board (ITRB), a group of
leaders in IT from US federal government
agencies.
It is a tool for evaluating how the strategic
implementation of IT supports the organization's
mission and improves its products and services.
The analysis is supported by nearly 300
questions, with yes/no answers, gathered around
critical issues

Critical Issues

Determination of a mission and a vision

The strategy defining where the organization is going
Identifying and addressing customer needs for IT products and service
The presence of a business plan
The management leadership that makes it possible to mobilize the staff
Actions taken by management at various levels in the organization
The strategic planning decision-making process
Project management processes
Managing performance
The technology that makes it possible to implement information systems
The process of acquisition of goods and services
The presence of architectures (work, data, systems, technology and flow
of information)

A GUIDE TO THE PROJECT

MANAGEMENT BODY OF KNOWLEDGE

Overview
This guide from the Project Management Institute
focuses on best practices for project management.
It positions the various phases of a project (initiation,
planning, implementation and control) as well as the
practices of project management related to each one
of these phases.
The practices of project management can be
considered according to two dimensions: the processes
to be satisfied (e.g., management of the cost of the
projects) and the phases of the projects (e.g., planning
of the activities to be performed)

Nine expertise
1.

2.

3.

4.

Management of the integration of the projects: to coordinate the

various components of a project such as planning and execution
of the project, and control the change, cost and quality of the
project
Management of the scope of the projects: to determine how the
project will be initiated, planned, subdivided into manageable
components, accepted by the stakeholders and controlled by an
effective process
Time management of the projects: to determine what specific
activities must be done to produce the deliverables, their
sequencing and related duration estimates as well as controlling
all changes to the schedule tasks
Management of the cost of the projects: to identify resources
required, estimates of their costs, their allocation as well as
controlling changes to the project budget

Nine Expertise
5.
6.
7.

8.
9.

Management of the quality of the projects: to evaluate the

adequacy of the standards applied, project performance and
results with proposed corrective actions
Management of the human resources assigned to the projects: to
determine and assign respective roles and responsibilities, acquire
and deploy appropriate resources
Management of communications in projects: to define what
information and communications will be needed and to whom
and how these will be collected and disseminated in a timely
manner
Management of the risks of the projects: to identify, evaluate,
detect and address the risks associated with the project
Management of acquisitions: to ensure services or products are
acquired from the right supplier in a timely manner as well as
meeting the needs and requirements of the projects

SYSTEMS SECURITY ENGINEERING-CAPABILITY MATURITY MODEL

Overview
Initiated by the US National Security Agency, describes
the essential characteristics of an organizational
architecture of IT security, according to practices
generally observed in agencies of the US federal
government. It covers the following aspects:
1. The complete life cycle of an information system
2. A review of activities covering the entire organization,
including those with a technical focus such as software
engineering and the interoperability among the various IT
areas of the organization
3. Coordinating activities with other operational areas of
the organization, such as asset acquisitions, systems
management and internal auditing

Overview
This set of best practices is based on reviews
conducted by the US GAO in eight agencies
recognized as leaders in IT security.
It identifies critical issues required to ensure
an adequate management of IT security.

Overview
The model analyzes IT security according to
three criteria:
1. The evaluation of the riskdetermination and
scope of the threats,
2. The setting in place of measurementdesign
and implementation of the required solutions,
and
3. Assurancecorroboration and need for security.

Overview
It examines the status of 22 practical issues of IT
security and management and compares them
with a table of maturity levels that is defined in
terms of expected results.
Level of maturity:
Level 1 implies that all the activities related to security
are carried out at least in a basic way,
Level 2 the activities are planned and followed
Level 3 well defined
Level 4 Controlled quantitatively
Level 5.

INFORMATION SECURITY MANAGEMENT:

LEARNING FROM LEADING AGENCIES

These best practices address

Coordination of activities
Evaluation of risks and determination of
Establishment of appropriate policies and
controls needs
Implementation of programs to ensure
organizational involvement
Measurement and evaluation of the
effectiveness of policies and controls

Coordination of activities
Establish an organizational group dedicated to
IT security.
Ensure access to senior management by the
management of the security function.
Provide the security group with the required
human and financial resources.
Ensure continuous training and required
professional certifications.

Evaluation of risks and determination

of needs
Evaluate specific security risks.
Identify and determine IT resources that are
critical to the mission and objectives of the
organization.
Ensure that end users are well trained and
accountable for their activities.
Manage the risks on a continuous basis.

Establishment of appropriate policies

and controls
Define policies and controls required to
manage risks.
Ensure that all operational areas can support
and implement these policies

Implementation of programs to
ensure organizational involvement
Train end users and stakeholders to address
identified risks and comply with policies.

Measurement and evaluation of the

effectiveness of policies and controls
Evaluate the factors that affect risks and
undermine security.
Consider the results of the evaluation to
determine future needs and to report to
senior management

Conclusion
Following the survey and assessment of evaluation models that
address the needs of the audit of electronic service delivery, the
authors selected COBIT 3rd Edition as the main evaluation tool.
The main rationale for selecting COBIT is that it is up-to-date
(released in July 2000) and that it integrates the key issues that
must be considered when managing and controlling IT activities.
COBIT integrates the key IT processes of an organization and
supports the different perspectives and needs of IT management
and of audit and control professionals.
As COBIT is a process-oriented framework, it also provides the
basis for communicating well with end users and general
management.

 Nevertheless, the authors decided to complement

COBIT with other models that will enable them to
more specifically address areas such as e-commerce
security, enterprise architecture, change management
and benefits management.
To address these specific concerns, they integrated
information from the other models discussed in this
document and created a database of best practices
that also reflects the experiences of other Canadian
jurisdictions and the US GAO.

Conclusion
Selected 11 of the 34 IT processes of COBIT, which they believed to
be most pertinent, from the following areas:
Planning and Organization:

Define a strategic IT plan

Define the information architecture
Manage the IT investment
Ensure compliance with external requirements
Assess risks

Delivery and Support:

Define and manage service levels

Ensure continuous service
Ensure systems security
Educate and train the users

Monitoring:
Monitor the processes
Obtain independent assurance

Hubungan antara audit keuangan dan sistem

informasi

Teknik Audit
Auditing Around The Computer
Untuk SI yang simpel, tidak terintegrasi, sehingga
pemrosesan di dalam komputer dianggap benar. Piranti
lunak pemroses dianggap sebagai black box. Audit hanya
meliputi input dan outputnya saja.

Auditing Through The Computer

Untuk SI yang kompleks dan terintegrasi. Yang di cek input,
proses dan output. Kebenaran piranti lunak harus dicek.

Auditing With The Computer

Komputer hanyalah alat bantu yang dilengkapi piranti
lunak untuk membantu auditor memroses data elektronik.
(CAATs: GAS)

Performance Measurement
Approaches

Business Value Hierarchy

Monitoring organisational
effectiveness

Information Economics

Generic IT Balanced Scorecard

Assurance Activities Linked to COBIT

Components

Urutan Kelompok Presentasi Tugas

Akhir

Selasa 29 Nov 2016

Kel 1 : Muh Fauzan
Kel 2 : Filzam
Kel 3 : Dimitri
Selasa 6 Desember 2016
Kel 4 : Harya Samudra
Kel 5 : Maurid
Kel 6 : Adil

Paper full dikumpulkan ke scele, senin 28 November

2016, maks jam 22.00