MNsure Phase II Project

Deliverable #2 Program and Project

Management Assessment

July 11, 2014

Document Control Information

Document Information
Document Name

Deliverable #2- Program and Project Management Assessment

Project Name

MNsure Phase II Project

Client

State of Minnesota

Document Author

Deloitte Consulting LLP

Document Version

2.0 FINAL

Document Status

FINAL

Date Release

07/11/2014

Document Edit History

Version

Date

Additions/Modifications

Prepared/Revised by

1.0

6/03/2014

Draft

Deloitte Consulting LLP

2.0

7/11/2014

Incorporated Feedback from MNsure, DHS, and MN.IT

stakeholders

Deloitte Consulting LLP

-2-

Table of Contents
Sections

Page Numbers

Lead Vendor Project Background

45

Executive Summary

68

Approach and Scope

9 11

Governance, Decision Making, and Accountability

12 28

Immediate Key Resource Needs

29 30

Communication and Information Flow

31 36

Status Reporting

37 42

Risk and Issue Management

43 47

Change Control

48 53

Defect Management

54 68

Test Management

69 83

Release Management

84 93

Appendices
Appendix A: Project Management Processes and Tools
Appendix B: Roles and Responsibilities
Appendix C: Interviews Conducted

94 115

-3-

Lead Vendor Project

Background

Lead Vendor Project Background

Project Background and Objective
Deloitte Consulting LLP (Deloitte) was engaged by the State of Minnesota to assess, identify potential impacts and provide
recommendations for the States consideration on the go-forward strategy for ongoing operations, 2015 open enrollment and beyond

Project Scope
1. Conduct an assessment of governance structure, decision-making processes, program and project management practices
and provide recommendations for consideration to implement governance structure, program and project management
controls and oversight
2. Conduct an assessment of the current state of the MNsure system from functional and technical perspective and provide
recommendations for consideration for the short-, mid-, and long-term
3. Perform the following project activities:
Program and Project Management
Project Planning
Functional and Technical Systems Assessment
Scope of this
Release Management
Defect and Issue Tracking
deliverable
Leadership and Planning of User Acceptance Testing (UAT)

Project Deliverables
Deloitte is contracted to produce five deliverables:

1. Report and reconciliation matrix of current status of Deliverables across existing vendor agreements
2. Project Management Analysis and Considerations Report
3. Phase 1 Functional and Technical Assessment Report with a categorization of key functional and system gaps and
considerations for a near-term system roadmap
4. Application Project Work Plan
5. Phase 2 Functional and Technical Assessment Report with a categorization of key functional and system gaps and
considerations for a mid-term and long-term
The focus of this deliverable is the Project Management Analysis and Considerations Report

-5-

Executive Summary

Executive Summary
Deloitte Consulting LLP (Deloitte) was engaged to conduct an assessment of the governance structure, accountability and
decision making, project management controls and software development lifecycle (SDLC) phases of testing, defect, and
release management. This assessment focused on identifying considerations for the State for the immediate term (calendar
year 2014) and for a sustainable project management structure and lifecycle.

During the Fall of 2013, much of the States efforts were focused on addressing issues that arose at the time of initial open
enrollment (October 1, 2013). During this period, we understand the governance and project management processes for the
project became less effective and resulted in a lack of coordination, integration and decision-making across the project teams
and stakeholders. Recognizing these challenges in early 2014, the State began to refresh efforts to reinstate its governing and
project management processes it had instituted at the outset of the project.
Deloitte identified observations, impacts and considerations in the following areas: (1) Governance; (2) Communication and
information flow; (3) Status reporting; (4) Risk management; (5) Issue management; (6) Change control; (7) Defect
management; (8) Testing management and (9) Release management. For each of the areas, the overall maturity of the
process/area was assessed against Deloittes proprietary project management methodology.
Governance: While positive efforts were noted in the reestablishment of a model of governance and related processes earlier
this year, their effectiveness remain diluted for a variety of structural, procedural, role definition, decision-making and
accountability challenges. The cumulative effect has been to create confusion among most leads and stakeholders, inconsistent
adherence to processes, untimely decision making and issue resolution. In addition to streamlining project execution
responsibility under a new Project Director role (within the Minnesota IT organization and has day to day responsibility for the
MNsure IT system project), the full establishment of a MN.IT MNsure Project Management Office (PMO), empowerment and
staffing of all governance bodies (including Change Control Board) was identified.
Prioritization of key tasks, activities and decisions made, need to be documented, communicated, and not revisited or changed.
MNsure IT system project work needs to be documented in an integrated project work plan to include testing and release
management activities built into the approach. Clarity of roles and establishing measurable accountability are key takeaways of
the observations.
-7-

Executive Summary (cont.)

Project management processes: Reconstitution of many critical project management processes were evident, however many
of these processes lacked consistency in operation and varied in maturity. The primary impacts to project effectiveness are
concentrated in a number of areas that are prioritized below:
It was observed that testing of complex functionality (such as batches, interfaces and notices) often occurs directly in the live
production environment - where actual user processing occurs. This specific functionality was not tested with State
involvement, and broader testing was limited or entirely missing prior to promotion to production. The State needs to address
the barriers preventing thorough testing in the lower (earlier) environments. In addition to significant disruption risk to t he
production environment, the cost of remediating an issue found in production is generally significantly more costly than when
found much earlier in the testing cycle.
Controls for risk, issue, and decision management (including logs and spreadsheets) are available for the project but there is
not active or consolidated management of these logs. Specifically, prioritization of risks and issues at an appropriate level does
not occur, nor does timely decision making occur. This can lead to issues, risks, and decisions not being fully understood,
communicated, or acted upon with the appropriate degree of prioritization.
Tracking and timely reporting of current and cumulative project status is critical to understanding where the project stands at
any point in time and thereby allowing leadership to respond to issues, unplanned events, and resource impacts in particular.
Comprehensive status reporting for the project was not timely, consistent or fully representative of all IT vendor partners a nd
agency groups.
System defects do not appear to be comprehensively captured, resulting in a far lower number of total open defects. Initial
reports showed only 60-162 total open defects. Upon follow-up and detailed analysis, 399 total open defects were identified.
The defect types are split roughly in half between product and functional issues, have been identified in the production
environment and fixes pending to be delivered by vendors were identified in lower environments. The State should validate and
confirm that this is the exhaustive list of defects and one system should be used to track and manage all defects. The non capture and active management of system defects will challenge system improvement efforts and may pose additional financial
burden on the State.

While our observations are pervasive across the governance model and project management processes addressing these
needs with a positive impact to project momentum can usually be achieved in a short timeframe. The remainder of this
document provides the detail and considerations to affect this effort.
-8-

Approach and Scope

Approach
Deloittes approach to assessing the current project governance, project management and software development lifecycle
processes and tools was to interview stakeholders, review documents and processes, and identify gaps. Gaps were compared with
Deloittes Project Management Body of Knowledge (PMBOK) based project management methodology to develop considerations
for each of the assessment areas.

Inputs and Activities

Identify
Stakeholders

Review Existing
Analysis

Deloittes Project
Management
Methodology

Process Walkthroughs

Interviews

Document
Reviews

Outputs
Observations, Impacts and Considerations for:

Governance, decision-making and

accountability

Communication and information flow

Status reporting

Risk management

Issue management

Change control

Defect management

Testing management

Release management
Proposed processes and tools (to-be implemented
by the MN.IT MNsure PMO) for:

Status reporting

Risk management

Issue management

Change control

Defect management

Testing management

Release management
- 10 -

Scope
The scope of this assessment is to provide observations and considerations focused around governance, prioritization,
communication and information flow, status reporting, risk and issue management, defect ,test and release management

Governance

Defect Management

Governance scope:
Governance structure
Accountability and decision-making

Defect management scope:

Defect triage
Defect prioritization, ownership, resolution, closure
Defect management tool
Defect dashboards and metrics

Communication and Information Flow

Communication and information flow scope:
Internal and external stakeholders (agencies, vendors, health
plans, counties, navigators, brokers
Information flow

Test Management
Test management scope:
Testing plan
Testing lifecycle spanning unit test, integration, system test,
user acceptance test, production smoke test, and regression
test
Performance test, security test, and American Disabilities Act
(ADA) testing types

Status Reporting
Status reporting scope:
Reporting of status content
Status preparation and distribution

Risk and Issue Management

Release Management

Risk and issue scope:

Risk/issue plan
Risk/issue tools and maintenance
Risk/issue prioritization and categorization

Release management scope:

Release management plan
Release schedule and calendar
Release estimates standards
Release checklists
Release notes
Deployment standards

Change Control
Change control scope:
Change control board
Change control request process
Change control log and request form

- 11 -

Governance, Decision-making, and Accountability

Project Governance Overview

Deloitte was engaged by the State of Minnesota (the State) to assess the project governance, organizational structure,
and project management approach and to recommend critical changes needed to improve overall management of the
project. The project is defined as the MNsure Phase II Project - which in short is the project to effect remediation and
enhancements to the system to fully enable the enrollment process for 2015 (which starts on November 15, 2014).

Three primary state entities have a stake in the project the Minnesota Insurance Marketplace (MNsure), the
Department of Human Services (DHS) and the Minnesota Information Technology agency (MN.IT). Our review included
understanding the business interests, relationships and impacts of these organizations on the project.

Today a Board of Directors governs the relatively newly formed MNsure organization (MNsure). At a summary level, the
Board by its charter predominantly determines strategy and delegates day to day operational management to its
appointed Executive Director, while also maintaining particular focus on the financial underpinnings of the organization.
The business of the organization (including policy setting) is exclusively that of the Board, who nonetheless can delegate
responsibility to its executive director or a committee(s).

The Department of Human Services (DHS), a more mature organization, is headed by a Commissioner with an
underlying executive team. The Commissioner has a permanent appointment on the MNsure Board of Directors.

The Minnesota Information Technology agency (MN.IT) is led by a Commissioner and the organization has broad
ownership of the States technology assets and resources, and operates as a shared services organization for their
respective business customers, including DHS and MNsure.

Although the MNsure and DHS organizations have unique business goals and interests, they share common interests as
they relate to providing health coverage to Minnesotans, and the underlying processes and system (MNsure system)
that enables that processing. MN.ITs stake in the relationship relates to the enablement and ongoing management of the
technology system as a shared services entity.

- 13 -

Project Governance Overview

We understand from interviews, that events following the initial open enrollment period (October 2013) led to a significant
breakdown in project governance and management processes. This was due to the State being primarily focused on
addressing and remediating the issues that arose at the time of open enrollment.

Following the departure of the Executive Director of MNsure, the Board of Directors essentially stepped into the operational
leadership role left void by her departure. Since that time the Board has continued to play a significant role in the operations
of the Exchange, was successful in appointing a new Executive Director, and working collectively with DHS and MN.IT has
started to reestablish critical governance and project processes.

Unclear roles and responsibility, authority, lines of communication, reporting relationships, team and governing bodies
composition have added significant inefficiency and have fostered no or poorly informed decision-making across the project.
Priorities or decisions are not timely or at the appropriate governing level and often are revisited and or changed.

Of particular concern, was the inconsistent and unclear engagement of MN.IT, the states information technology agency,
with broad responsibility for all state IT activities and assets, and their role in managing and delivering the system
(particularly within the context of the systems plan for the state enterprise).

Management of the MNsure IT system development vendors has been inconsistent and appears to have impeded outputs
and progress.

The dominant engagement model has tended towards a siloed approach among key stakeholder agency staff - further
exasperated by loose vendor management who themselves have operated in a silo from one another.

Absence of a baseline and updated/maintained consolidated work plan for the project - that is comprehensive of all task
level details for all contributing resources and vendors through system delivery and post system go live stabilization has
made project direction, execution, progress tracking and management challenging.

A number of essential roles/positions were not defined/ vacant for the vast majority of the project (including Project Director,
Testing Lead) that further challenged the governance and project management processes and project effectiveness.

- 14 -

Project Governance Summary Observations

Governance, Decision-making, and Accountability
Component Expectation

Summary Observation

Relevant business interests, strategic intent and priorities

of all agency stakeholders are defined, duly aligned and
represented in the form of a project long-term plan.

Partially present
While most near-term interests are known and a long term MN.IT@ DHS strategic plan exists;
alignment, prioritization and longer-term plans need to be finalized and communicated.

A governance and enabling organizational model for the

project exists, is well defined, been duly constituted and
understood by all impacted parties.

Partially present
Model components exist and are operating clarification of select roles and project ownership,
role realignment and addition of new roles should be considered. Upon finalization, clear and
broad communication is needed.

Roles and responsibilities of the project governing

body(ies) and participants are well defined and align with
the governance and organization model

Partially present
For a select few governance bodies Roles and Responsibilities were reasonably well defined, in
general they were are not uniformly defined, clear or well understood across all impacted
parties.

Clear delegation of responsibility and role definition of all

committees (including appropriate peering of all
participants).

Partially present
A number of committees exist (from direction setting, control to quasi-operational). Certain
committees roles, operation and effectiveness may be diluting the overall governance process.
We observed some peering inconsistencies that should be addressed.

Responsibility for priority setting is clear and priorities are

adhered to once establish.

Partially present
Many governing groups at MNsure set priorities but they are not established with a clear
methodology. Once established, priorities are often modified or fully changed.

Decision-making authority and escalation pathway is well

defined and understood by all affected parties.

Partially present
Although partially evident and improvements over time were noted - to avoid decision-making
delays and to engage the most relevant experience/skills at the right time, there is a need to
(re)align and empower the decision-making process.

Key project governance and organizational roles are

staffed with appropriately experienced and skilled
resources.

Partially present
Although many roles exist, critical ones are absent (including Project Director and a number of
subordinate but important roles such as a Testing Manager).

The enabling governance processes (including tools,

reports and meetings) to support an effective project
exist.

Partially present
While most processes exist, consolidation and further role and expectations clarifications are
needed to improve effectiveness.

- 15 -

Project Governance Detailed Observations

ID

Observation

Impact

Considerations

Although the interests and goals of the

Business Owners of the project
(MNsure and DHS) are generally well
defined and understood for the nearterm, they and their enabling tactics lack
prioritization. Further, the long term
strategic intent has been less clear.
DHS has developed a 5 year
project/system modernization plan,
however MNsure s longer-term needs
are still in progress. The absence of
alignment and harmonization of these
longer-term interests is a barrier to
completing the project long-term plan

Governance is reactive to latest

developments and decision-making is not
fully guided by a immediate and longerterm strategy. Staff and vendors are
unclear on priorities, significant milestones
or targets, and objectives for longer-term

Complete long-term business planning within the

respective project Business Owners and
consolidate those as they relate the project
(system). Together with near term interests set
the business prioritization for the project (with the
advice and guidance of the MN.IT organization).
Finalization of this process should include
relevant staff, vendors and other stakeholders

Fragmented and unclear decisionmaking authority and role confusion

across the project at multiple levels is
leading to decision-making delays,
bottle-necking of issues in need of
resolution, protracted activity planning,
and unmet objectives

Project delivery resources progress is

impeded as issue resolution, prioritization
or other decision-making is bottle-necked
or protracted

Clarify decision-making authority for each

governance body and representative role(e.g.
PMT,CCB, EST); define the issue and decisionmaking escalation pathway; set expectations and
accountability measures for each governance
body and role. Monitor and report periodically on
expectations and accountability measures

Vendors are given conflicting direction

on priorities of work and system
requirements from business,
technology and management teams
operating in silos, without coordination
or integration across State and vendor
teams. This appears to have
exasperated the lack of coordination
and teaming across the legacy vendors

Absent a common control point, vendors

are unsure how to proceed, or proceed in
conflict with other activities of the project
and lose production time clarifying tasks,
priorities, and requirements

MN.IT should provide project execution

leadership and management of all SDLC duties
including tasks such as managing technical
teams to the project plan activities,
communicating priorities, managing system
requirements, daily supervision of IT vendors,
coordinating and integrating across State and
vendor teams, and providing to progress
reporting through the project governance
leadership and business owners

The absence of a full and clear long-term

plan adds the risk of inefficient investment
being made in how future system
components are developed

- 16 -

Project Governance Detailed Observations

ID

Observation

Impact

Considerations

Participant roles and

responsibilities are not clearly
defined and adhered to within
governing bodies

Participants are often unsure of what

contribution they should provide to the
governing group leading in some cases to
both under and over-involvement of
participants, lack of focus on critical aspects
and inefficient use of senior resources

Establish/clarify governing body participant

expectations, roles and responsibilities,
accountability and decision-making authority

Participants in certain governing

bodies are not peered at the
same level, which may inhibit
engagement and equitable
decision-making

By mixing participants of different

organizational levels on a governance body,
there is a risk of representation bias, uneven
engagement and value creation and
decision-making independence

Wherever possible, participants on the project

governance bodies should be at the same peer
level. A review of current participants peering level
across governing bodies and realignment as
needed is recommended

- 17 -

Project Governance Detailed Observations (cont.)

ID

Observation

Impact

Considerations

The Minnesota Insurance Marketplace (MNsure)

governance has been undergoing a transition
over the past year (separate and distinct from the
project governance)

Over the past few months as the

new Executive Director and
Leadership team established itself
and began operating the business
(including managing its stake in
the project to remediate and
enhance the underlying system),
one of the unintentional
consequences of the Board and
Committee/Workgroups prevailing
operating mode may have
contributed to project inefficiencies

As the new Executive team solidifies, a

new project governance model (with
clear implementation accountability),
full and appropriate engagement by
key agency stakeholders (DHS and
MN.IT) is completed and
accompanying project processes
(including transparent performance
reporting) is established, we would
encourage a realignment of roles,
responsibilities and decision-making
between the Board, the relevant Board
workgroups/subcommittees and the
Executive team in the governance of
the project

After the departure of its initial Executive Director

(ED) and in the aftermath following the open
enrollment period, we understand the MNsure
Board (Board) essentially stepped into the ED
role, responsible primarily for daily operations.
We further understand, that as a result of a
historical lack of operational transparency, timely
and accurate reporting and appropriate level
decision-making, the Board decided to further
fortify the organizational leadership by
establishing several Board workgroups including
one focused on Technology and the project which
is the focus of Deloittes review
In more recent months, MNsure has successfully
appointed a new Executive Director, added new
senior leadership capability and begun to realign
the legacy organization. Coincident with these
changes, cooperation and coordination with the
other key agency stakeholders (DHS and MN.IT)
accelerated rapidly

Until operational management and

decision-making rebalances
between the Board and Executive
Leadership there exists another
layer of decision-making, coupled
with the risk of conflicting project
intervention and direction which
may impact overall project
effectiveness

- 18 -

Project Governance Detailed Observations (cont.)

ID

Observation

Impact

Considerations

Vendor management and direction

setting for the project system vendors
has vacillated and was often unclear

Vendors and staff are confused about

who is responsible and has authority to
direct System Development Life Cycle
(SDLC) activities, and how to prioritize
their related activities, adversely
impacting project progress

State and vendor leadership,

managers, and subject matter experts
(SMEs) are not involved at an
appropriate level in the governance
and decision-making

State and vendor SMEs are not utilized

appropriately to inform the governing
groups, and leadership and managers
are not providing appropriate input. It is
unclear as to the how or why decisions
are made, and have limited visibility into
the process

The role of an overall Project Director

for the project is not defined and
accordingly unfilled. Furthermore,
there is no central PMO coordinating
project functions

Inconsistent application of Project

leadership and management tasks and
many are conducted to varying degrees
by multiple stakeholders resulting in lack
of coordination

Develop scope, roles and responsibilities,

and reporting structures for a MNsure IT
Project Director and a MN.IT MNsure PMO.
Communicate to stakeholders the roles,
responsibilities, and authorities of the Project
Director and the MN.IT MNsure PMO.
Coordinate and integrate the activities of the
MN.IT MNsure PMO with other agency
PMOs within DHS, MNsure and MN.IT

10

Turnover of staff associated with the

project has occurred in participant
roles in governance

Institutional knowledge of governance

goals, activities, and outcomes is lost
when turnover occurs. New staff in
governance roles need to be on-boarded
to the governance participant role

Develop a workforce transition plan that

identifies project governance participant
roles and documentation so that knowledge
transfer from one State staff person to a new
State staff person to fulfill the governance
participation role

- 19 -

MN.IT be responsible for day-to-day

management of SDLC duties including tasks
such as managing teams to the project plan
activities and supervision of IT vendors

Participation roles in governing groups

should be defined for State and vendor
leadership. Management and staff should be
utilized at meetings as appropriate to provide
insights necessary to inform the governing
groups and provide clear communication
and visibility to the decision-making process

Project Governance Detailed Observations

ID

Observation

Impact

Considerations

11

Agenda topics and discussions

at governing groups are not at
the appropriate level needed to
meet the purpose of the
governing group

Topics and discussions outside of scope of

the group reduces the ability of the group
to fulfill its role and responsibilities and can
result in conflicts. Project staff also
consume time and effort providing
materials that are outside of scope of the
governing groups

Provide agenda items that are within scope of the

governing group. Maintain facilitation for each
group that manages adherence to the scope for
the governing groups

12

Meeting cadence including

sequence and frequency of
meetings for governing groups
is not appropriate for the
objectives, activities, and
outcomes required of the
governing groups

Improper meeting cadence, for instance

too frequent, encourages discussions that
are not within scope, or cadence that is not
frequent enough prevents discussions that
are in scope. Currently activities such as
meeting preparation, meeting time, and
post meeting activities are consuming
participant and project staff time

Meeting cadence should be defined that allows

the goals, activities, and outcomes of the
governing groups to be met while reducing
unnecessary meetings. A master schedule of all
meetings should be developed to manage
duplication, inefficiency and resource conflicts

- 20 -

Project Governance Detailed Observations (cont.)

ID

Observation

Impact

Considerations

13

An integrated project work plan is not

established with project activities,
dates, milestones, releases,
interdependencies, and resource
ownership of project activities

The State leadership and management

have limited insight into the activities of
specific vendors and dependencies which
reduces their ability to make decisions
based on planned activities

Develop an integrated work plan for all ITrelated project activities. The plan serves as
the primary document governing the activities
on the project including dates, milestones,
deliverables, responsible parties, and
dependencies

14

Processes for deliverable submittal,

review, acceptance or rejection,
remedy, and invoicing are unclear to
the State and vendor partners

State and vendors use time and effort

determining what has been submitted,
what should be approved or needs
additional activities to remedy, and what
decisions should be made regarding
payment of invoices

Establish and document the standard

approval process for deliverables and
communicate to appropriate vendors and
State staff

15

The naming convention of the IT

system (MNsure system) being
synonymous with the governing
organization (MNsure) for the health
insurance exchange creates
confusion in communication and
direction-setting

The system is intended to support multiple

agency/organizational interests.
Unwarranted and unintended confusion
can be caused when stakeholders
address organizational needs and issues
rather than the enabling IT system
demands

For clarity purposes - the system should be

assigned a unique name/moniker that
allows clear differentiation of the enabling
system (project) from any vested
agency/organization

- 21 -

Project Governance Detailed Observations (cont.)

ID

Observation

Impact

Considerations

16

MNsure work is divided into many

projects with out a full documentation
of dependencies or an overall project
work plan or consolidated schedule to
drive project work

The State and vendor project leaders

lack visibility into project dependencies
and activities on the project are not
managed based on a plan or schedule

Develop an integrated work plan for all ITrelated project activities including: design,
development, testing, and release
activities. Empower the Project
Management Team, Project Director, and
the MN.IT MNsure PMO to manage and
drive the activities of the project based off
the project work plan

17

No one person is in charge of the day

to day operations for the MNsure IT
project

Gaps in accountability develop as

governing groups spend energy
determining who is responsible for a
particular issue rather than effort to
resolve the issue

Create a MNsure IT Project Director

position to manage the day to day work of
the project for both vendors and State
staff. This position should report to MN.IT
staff and coordinate frequently with
MNsure, DHS, MN.IT, and vendor
stakeholders

18

MNsure board working groups lack

clear cadence, definition, duration, or
role for MNsure

Frequent meetings drain time and

resources away from the MNsure
leadership and staff as they prepare
and conduct briefings for Board
members

Clearly articulate the role and objective of

MNsure Board working groups. Consider
a sun setting or postpone work groups
during times of reduced activity on the
MNsure project

- 22 -

Project Governance Proposed Model

The structure supports the two Business
Owners/Sponsors setting overall direction, policy and
reviewing progress of the project based on their
strategic business needs. While some of those
interests are shared, many are independent of one
another. A process for setting, reconciling and
reviewing the related demands on the project will need
to be established

Business Owners/Sponsors

MN.IT

MNsure

DHS

Executive
Steering
Committee

The MN.IT agency, as the States Information

Technology shared services agency is positioned to
provide input to this process and serve as technology
advisor to the Business Owners

Project
Management
Team

The Executive Steering Committee has operational

responsibility for the success of the project and
primarily with the setting and monitoring of the tactics
to achieve the goals set by the Business Owners,
including major operational issue resolution. The
process for reviewing and reconciling competing
business demands on the project will be handled by
the Executive Steering Committee. The EST will
provide progress updates to MNsure and DHS and
inform them of any serious risk s or issues related to
the goals the business owners have set

Change Control
Board

Project Director
MN.IT MNsure
Project
Management Office

Change

Training

Communications

Business
Analysts/SMEs

IT vendors

Test

Release

Defect

Technology
staff
- 23 -

The Project Management Team serves as the Project

Directors immediate cross-agency operating group to
aide in resource, project and other operational issue
resolution
The Project Director who is staffed by MN.IT as part of
their delivery responsibility for the project and system,
manages and oversees the day-to-day activities of the
project and the system delivery teams

Project Governance Proposed Model

Governance Model Detail Senior Level
Business Owners/Sponsors

MN.IT

MNsure

DHS

MNsure Board
Compliance
Workgroup

Other .
Workgroup

Financial
Workgroup

Technology
Workgroup

Executive Steering Committee

CORE MEMBERS:
MNsure CEO
DHS Deputy Director
MN.IT CEO

Today the MNsure Board executes its

responsibilities largely through several workgroups
comprised of Board Members who in turn report
back to the collective Board on their respective
focus area (including Compliance, Financial,
Technology). Similar responsibilities for the DHS
agency are incorporated within the role of the DHS
Commissioner
As it relates to the project, the Business Owners
role is to set the strategic direction, based on their
business interests and mission/charter, for the
project and underlying system. The Board should
expect regular timely and relevant updates on
progress from the Executive Steering Committee,
as well as being made aware of significant critical
risks and issues as they arise, that may impact the
project and attainment of their goals. (In the case
of MNsure, the Board may exercise these
responsibilities through its current workgroup
structure or another model)

Supported by NON-CORE MEMBERS:

Senior Management Representation
(incl. Legal, Finance, Technology, Admin, Operating)

- 24 -

The Executive Steering Committee has

operational responsibility for the success of the
project to achieve the business goals set by the
Business Owners. It will determine and execute
operational level tactics to achieve those goals,
resolve all issues escalated to it by the
downstream governance team(s), monitor project
progress and keep the Business Owners
accordingly apprised

Governance Proposed Business Owners Framework

Role for the MNsure Project
The Business Owners provide overall guidance, policy setting and direction for the project based on their respective organizational strategy and
business needs. Where the Business Owners needs and priorities differ, due to mutually exclusive charter or strategies; they must reconcile those
differences and priorities such that the project direction is clear and unimpeded. The Business Owners retain broad oversight for the project and
should be kept apprised of progress through pre-determined executive updates. The Board should also expect to be made aware of any
significant critical events and risks that may impede progress and/or success of the project

Key Responsibilities

Members Roles and Representation

MNsure Board: Represented in the form of either its Board Chair
and/or another Board delegate (this could take the form of a
representative Board workgoup)

DHS Commissioner: The Commissioner for the Department of

Human Services
Note: While MN.IT is not considered a Business Owner, their value
as State IT shared services agency is recognized as important
and as such as should provide input and advisory support to the
Business Owners

Key Relationships

Strategic Planning
Policy determination/setting
Governance
Organizational/project direction setting
Project review and monitoring (in particular where
significant events or risks may impede progress or
success)

Key Decisions

Governor: Direct responsibility for the Department of Human

Services;
Legislature: Authorized the establishment of the Board and
authorized the Commissioner of DHS to serve as a Board member.
Exercises oversight of MNsure:

Meeting Cadence

- 25 -

Strategic
Policy
Communications themes and approach

One meeting per month (up to quarterly meetings when

project is stabilized and operating)

Governance Proposed Executive Steering Committee Framework

Role for the MNsure Project
The core Executive Steering Committee is comprised of the senior leadership peers from MNsure, MN.IT, and DHS, and provides the project
direction, monitoring and management of operational tactics needed to achieve the goals set out by the Business Owners. The Committee
provides overall coordination of efforts among leadership at MNsure, DHS and MN.IT. The process for reviewing and reconciling competing
business demands on the project will be handled by the Executive Steering Committee. In addition, the Committee serves to resolve issues and
assist with major project risk mitigation that is escalated up to them from downstream governance team(s)

Members Roles and Representation

Key Responsibilities

CORE Members: MNsure Executive Director, DHS Deputy

Commissioner, MN.IT CIO
SUPPORT Members: Key Operating, Technical, Administrative,
Legal, Financial representation. These members are not considered
voting members but advisory and support to the Core Members.

Executive level coordination between MNsure, DHS, MN.IT

Direction to the Project Management Team on guidance and
policy set by the Business Owners
Overall management responsibility for operations, policy,
technology, and communications on the MNsure IT system
project
Review and approve resource plans and commitments
Establish priority criteria for project activities
Resolve issues escalated by the Project Management Team

Key Decisions

Key Relationships
Business Owners: Overall Strategy and Policy setting
MNsure, DHS and MN.IT: Communication of plans and operational
impacts and coordination with respective organizations
PMT: The PMT provides project updates to the Executive Steering
Committee and escalates issues and risks for final resolution
Public and Media: These stakeholder groups look to the
leadership for information about the status of project
Committees (incl. Health Industry Advisory Committee and the
Consumer and Small Employer Advisory): These groups consult
with and provide recommendations to operational leadership

- 26 -

High level project operational setting and that provides more

detailed direction to the project
Issues resolution (escalated by the PMT)
Strategic recommendations to the Board
Statusing and communications recommendations to the
Board

Meeting Cadence
Weekly or bi-weekly until project key milestones are achieved
(can then revert to monthly)

Governance Proposed Project Management Team (PMT) Framework

Role for the MNsure Project
The Project Management Team (PMT) is comprised of business and technology managers that are peers from the three key stakeholder
agencies of MNsure, MN.IT, and DHS. The PMT reviews and approves more detailed administrative and operational project level activities and
decisions including forecasting, resourcing, planning, and prioritizing project activities, major enhancements, continuous improvements, and
maintenance of service delivery. Their direction to the Project Director is based on effective demand and capacity management of business and
technology agency resources, and management of cross agency interdependences and impacts. The PMT addresses risks, issues, and action
items escalated from the Project Director. The PMT operates within its authority and escalates issues to the EST as needed/required

Key Responsibilities

Participant Roles and Representation

MN.IT: MN.IT is a member representing the overall technology
goals of the project and MN.IT provides technology knowledge and
expertise to the PMT. The MN.IT representative acts with MNsure
and DHS input, as the decision authority for MNsure IT system
related decisions brought to the PMT
MNsure: The MNsure representative is responsible for representing
the business and technical goals of MNsure
DHS: DHS is a member representing the interests of DHS
programs that are affected/dependent by/on MNsure
Vendors: Representatives from each of the IT vendors attend the
meeting and provide input as requested by the PMT members

Provide direction to the Project Director for managing all

areas of the project including: scope, schedule, budget,
quality, resources, communications, risk, procurement, and
integration
Monitor the progress of project activities through the planning,
execution, monitoring, controlling, and closing of project
phases
Finalize recommendations from the Change Control Board
regarding change requests

Key Decisions

Key Relationships

Executive Committee: Issues outside their authority or that cannot

be resolved by the Project Management Team should be escalated
to the Executive Committee for final decision/resolution
Project Director: The PMT receives status from the Project
Director and the PMT provides guidance, decisions and issue
resolution support to the Project Director

Remediation steps for issues that are impacting scope,

schedule, budget, quality, human resources,
communications, risk, procurement, and integration
Prioritization of risks and issues
Recommendations for change orders
Release schedule
Identification of issues for escalation to the Executive
Steering Committee

Meeting Cadence
One meeting per week

- 27 -

Governance Proposed Project Management Office (PMO) Framework

Role for the MNsure Project
The MN.IT MNsure Project Management Office (PMO) provides support to the Project Director and to the project by providing tools and
processes, templates, standards, methodology, policies and procedures for activities including the project work plan, status reporting, risk and
issue tracking, change control, defect management, release management, testing management, and communication. The MN.IT MNsure PMO
coordinates with MN.IT PMO, MNsure PMO and DHS PMO. The MN.IT MNsure PMO has responsibility for rolling-up (consolidating) the
respective stakeholder PMO and vendor work plans and status reporting into a master plan and status report

Members Roles and Representation

Key Responsibilities

MNsure IT system Project Director: Directs the MNsure IT system

PMO and the activities of the MNsure IT system work plan
MN.IT MNsure PMO: The MN.IT PMO maintains the integrated work
plan for the MNsure IT system, and manages status reporting, risk
and issue tracking, change control, defect management, release
management, testing management, and communication. The State
provides necessary staff to assist the PMO including staff such as:
change manager, release manager1, test manager1, defect manager1
and communications manager.
Vendors: Project managers from each of the IT vendors are to
required to provide input to the PMO for each of the areas managed
by the PMO

Manage the project work plan, status reporting, risk and issue
tracking, change control, defect management, release
management, testing management, and communication
Provide reports to the Project Director on areas managed by
the MN.IT MNsure PMO
Communicate with key stakeholders
Develop project status reports and distribute to stakeholders

These roles are described in their respective section on following slides

Key Decisions

Key Relationships
Project Director: The Project Director guides the activities of the
MN.IT MNsure PMO. The PMO supports the activities of the Project
Director in managing the MNsure IT system
MN.IT and DHS: Other PMO staff coordinate with the MN.IT MNsure
PMO to maintain MN.IT MNsure PMO alignment with tools,
processes, templates, standards, methodology, policies and
procedures used by other State PMOs.
Vendors: Vendor work plans are integrated into the master project
work plan, vendors provide input to the areas managed by the MN.IT
MNsure PMO

- 28 -

Determination of tools and processes, templates, standards,

methodology, policies and procedures for project activities
Assignment of risk and issue owners

Meeting Cadence
Coordination occurs daily as a matter of on-going operations

Immediate Key Resource Needs

Immediate Key Resource Needs for MNsure IT System Project

Top 5 Positions and Descriptions
Priority

Position

General Description

Project Director

The Project Director for the MNsure IT system manages and oversees the day-to-day
activities of the project management lifecycle. This full time resource is empowered to make
key decisions for the MNsure IT system and will escalate issues to the Project Management
Team as needed. The MN.IT MNsure PMO will support the Project Director in driving
MNsure IT system activities

Test Manager

Test Manager is accountable for testing, test plan development and execution in the
MNsure IT system including: test cases and scenarios, results and defect management,
testing status communications and defining entry and exit criteria across all test phases
(integration, system, performance/regression, and UAT)

Release
Manager

The Release Manager will lead release management end-to-end activities and ensure
compliance to quality in release management execution. The Release Manager defines and
enforces standards and processes for release management across all environments

Communications
Manager

The Communications Manager is accountable for MNsure IT system communications

affecting MNsure, MN.IT, DHS and their key stakeholder groups (vendors, carriers,
counties, navigators, and brokers). The Communications Manager is responsible for the
development of an integrated communication plan for project stakeholders and for
monitoring communications triggers such as updated release functionality, technical events,
and operational changes

Defect/Triage
Lead

The Defect/Triage Lead will be a member of the testing team and will track and manage all
defects, will lead defect and triage meetings, and will report on identified defects and their
status to the State and vendor partners

Other personnel gaps or needed positions were identified during the course of the governance assessment of the
MNsure organization, those considerations are identified in the remainder of Deloitte Deliverable #2: Program and
Project Management Assessment report.

- 30 -

Communication and Information Flow

Communications and Information Flow Overview

Communication is an integral part of the success of the MNsure IT system. Communications leverage familiar methods to
reinforce messaging and use multiple methods for each stakeholder group. Various communication methods are used
depending on the purpose of the message and its intended audience. Communications are used to either inform or engage
specific stakeholders. Selecting the appropriate method to target the right stakeholders is key to the successful execution o f the
communication at hand
It was observed that communication silos exist within MNsure, MN.IT, DHS and their key stakeholder groups - vendors,
counties, navigators, and brokers. Meetings are being conducted and communications are being distributed within the
individual silos. An integrated communication plan for project stakeholders has not been developed. In addition,
communication ownership and triggers such as technical events, operational changes, and policy modifications have not been
defined. Vendor communications have not been formalized and vendors currently do not interact with end users of the
application. As part of information flow bidirectional communication occurs with feedback being actively solicited
Summary Observations from the assessment, and additional detailed observations, impacts, and considerations for the
Communication and Information Flow area are presented on following slides

- 32 -

Communications and Information Flow Summary Observations

Communication and Information Flow
Component

Summary Observation

Integrated project communication plan with key communications events as well as the
target audience, timing, delivery mechanism, key messages, and responsible parties

Not present
An integrated communications plan does not
exist, communication occurs in silos

Stakeholder matrix implemented for project communications that identifies and

categorizes stakeholders and key areas for communication or focus

Partially present
Matrixes exist, however communication
categorization and focus is not included

Project templates, triggers, timing, and channels for communications

Not present
Templates, triggers, and timing, are not
standardized and integrated with the project

Project communication creation, approval, distribution, and processes that formalize

communications processes

Not present
A formalized process is not documented for
communication creation, approval, and
distribution processes

Project communication feedback mechanisms that obtain bi-directional feedback

Partially Present
Bidirectional feedback mechanisms have not
been fully and consistently implemented to
measure stakeholder engagement

Multiple forums and channels for project communications

Partially present
Communication forums take place within
individual stakeholder groups. Additional
communication forums have not been
implemented for project communications such as:
Newsletters
Collaboration Groups
Town halls

- 33 -

Communications and Information Flow Detailed Observations

ID

Observation

Impact

Considerations

19

A consolidated plan and strategy for

stakeholder communications including
vendors, health plans, counties,
navigators, brokers and internal
stakeholders does not exist

Communications with internal and

external stakeholders are fragmented
and not formalized resulting in
stakeholders being updated on an adhoc basis that could result in
inconsistent messages

Develop and manage to an integrated

communication plan for stakeholders that
details: types of communications, target
audience, timing, delivery mechanism,
messages, triggers, and responsible
parties to standardize and formalize
communications

20

Communication channels are not

managed across DHS, MNsure, and
MN.IT; there are individual owners
responsible for communications that
relate to the MNsure IT system, including
distributing related communications

Due to the lack of defined ownership

between business and technology
groups non-standard communications
are sent which may lead to inconsistent
stakeholder communication by both the
business and technology groups about
project-related decisions creating
confusion about operational
procedures, schedule, policies and
technology

Identify a Communications Manager that

is part of the MN.IT MNsure PMO and is
responsible for coordinating
communications related to the IT System
across the project and is responsible for
making sure that communications are
aligned and planned for with key system
milestones

21

There is a lack of standardization in

communications triggers, templates, and
processes for both business and MN.IT;
communications, are not defined or
standardized for audience, templates, and
triggers

Details such as release status and

scope, release schedule, release
functionality, and downtime may not
reach the right stakeholders at the right
time in the right format, leading to
misunderstandings and confusion and
may limit ability to serve the customer

MN.IT can define a set of triggers,

templates, and processes for
communications as well as their audience,
focus can occur on the following technical
communications:
Release plan
Release calendar
Release notes
System outages
Testing status

- 34 -

Communications and Information Flow Detailed Observations (cont.)

ID

Observation

Impact

22

Internal stakeholders receive inconsistent

communications from various
stakeholders such as business and
technology groups and communications
feedback is not solicited

As a result of inconsistent
communications, confusion and
miscommunication may occur and
ultimately stakeholders could become
disengaged

Solicit feedback and develop forums for

internal stakeholder communications such
as town halls and newsletters to promote
open and transparent communications,
town halls occur quarterly and newsletters
are distributed monthly and engage
stakeholders to provide feedback
channels

23

MNsure and MN.IT communications with

vendors are not structured and formalized
and vendors have limited involvement
with user groups of the application

Vendors can receive informal

contradictory guidance from MNsure
and MN.IT which could lead to
inaccurate priorities, rework, and
confusion. Vendors do not receive
feedback from end users of the
application leading to missed
opportunities to improve the system

MN.IT assumes the leadership role over

communications with IT vendors, MN.IT
works with MNsure and DHS operations
staff to help set priorities and the overall
plan and create focus groups that provide
user feedback to the vendors

24

Meetings with stakeholders including

health plans, navigators, brokers, and
counties are scheduled but not
coordinated in terms of communication
content and messaging

Due to the lack of coordination around

stakeholder communications for health
plans, navigators, brokers, and
counties each group may have receive
different messaging with different
content at different times

Incorporate health plans, navigators,

brokers, and counties into the overall
MNsure communication strategy and
develop an integrated communications
calendar and detail communication
triggers to synchronize communications
for these stakeholders to maintain a
defined communication schedule, so that
all stakeholders receive timely
coordinated messages

- 35 -

Considerations

Communications and Information Flow Detailed Observations (cont.)

ID

Observation

Impact

25

Health plan meetings occur weekly to

discuss business and technology
processes but are not aligned with
MNsure communications and are tactical
in nature

Health plan meetings are not

integrated into an overall
communication plan or strategy which
could lead to missed opportunities to
improve communications with Health
Plans

Incorporate health plans into the overall

MNsure communication strategy and
develop a communications calendar and
detail communication triggers to for timely
and specific communications to relevant
stakeholders

26

Forums for county communications exist

to share business policy and system
information, however communication
gaps exist in terms of sharing policy,
operational, and technology information

Counties are one of the largest group

of the MNsure system users and often
deal with some of the most complex
family situations It is critical that
communications for policy, operational,
and technology are targeted, concise,
and timely to prevent inaccurate
information

Incorporate county information needs into

the overall communication strategy and
detail triggers for policy, operational, and
technology updates. Also consider
implementing additional county
communications strategies such as:
Testimonials
Fact Sheets
Job Aids
Frequently Asked Questions (FAQs)
Newsletters
Blogs or Collaboration Groups

27

The primary means of navigator

communications occur through weekly email communications

Navigator communications are not

timely and this is causing frustration
amongst this stakeholder group

Incorporate navigators plans into the

overall MNsure communication strategy
and develop a communications calendar
and detail communication triggers for
timely and specific communications

- 36 -

Considerations

Status Report

Status Report Overview

The project status report presents information on the activities of the MNsure IT system project including an overall project status,
an executive summary, and updates from vendors on scope, resources, schedule, and quality. The status report utilizes
dashboards to provide succinct, clear information for executives and managers

The project status report relies on close coordination between vendors and State resources to represent the project status. The
status report serves as an opportunity to communicate clearly across the project about activities and possible issues or risks that
may be present, and reduces the need for clarification or re-explanation of project status during the course of project activities. It
provides governance groups with appropriate information to allow the groups to make decisions that fulfill their responsibilities
Summary Observations from the assessment, and additional detailed observations, impacts, and considerations for the Status
Reporting area are presented on following slides

- 38 -

Status Report Summary Observations

Change Control
Component

Summary Observation

Overall project status report including weekly project progress and

performance

Present
A status report is produced weekly, however not all
key project health metrics are included

Executive Summary Section in status report

Not present
An overall executive summary that provides a high
level overview of the status is not present

Summarized items requiring leadership attention

Not present
A summary of executive items is not included in status
report

Upcoming milestones detailed in report include future releases, policy or

business operations updates

Partially present
Report describes some upcoming activities but does
not fully detail project interdependencies

Updates from vendors called out in specific sections

Partially present
Vendors provide only brief updates in the status report

Red, yellow, green status for scope

Present

Red, yellow, green status for resources

Not present
Not included in status report, a view of resources is not
present

Red, yellow, green status for schedule

Present
Not present
Not included in status report, a view of quality is not
present

Red, yellow, green status for quality

- 39 -

Status Report Summary Observations (Cont.)

Change Control
Component

Summary Observation

Project metrics included in status report

Partially present
Key metrics for managing the project are missing such
as variances and completion percentages

Project assessed using dashboards

Partially present
Insights are provided at a summary level, but detailed
dashboards do not exist for trends, change requests,
risks, issues

Distributed appropriately to stakeholders

Partially present
Currently distributed to project leadership but not all
stakeholders

- 40 -

Status Report Detailed Observations

ID

Observation

Impact

Considerations

28

The project is currently producing project

status reports, however gaps exist in
terms of consolidating information from
many stakeholders including MNsure,
DHS, and vendors

Overall status may not be reported

accurately due to the lack of integration
between MN.IT, MNsure, DHS, and
vendors

MN.IT MNsure PMO is responsible for

creating an integrated project status report
that includes status from stakeholders
including MN.IT, MNsure, DHS, and
vendors

29

Executive level project dashboards do not

currently exist for managing the MNsure
project at an executive level or displaying
impactful information to an executive
audience

Executives do not receive consolidated

dashboard views for the project making
it difficult to understand the full project
status including budget, scope, and
schedule

Develop and implement a project wide

dashboard that will display overall status
and provide metrics for change requests,
risks, and issues

30

Limited metrics reporting is included in the

project status report

Limited metrics do not provide

sufficient information to decision
makers for the purposes of managing
the project

The MN.IT MNsure PMO is responsible

for including additional metrics that
indicate the overall health of the project
and alert stakeholders to variances in
metrics as appropriate
Key metrics include:
Financial health variance
Requirements volatility
UAT test case first pass rate
Execution issues

- 41 -

Status Report Proposed Structure

The MN.IT MNsure PMO is responsible for consolidating information for the weekly status report. The Project Director reviews the
status report prior to distribution of the status report. The MNsure status report will be sent to a varied audience of stake holders that
includes agency executives, project leadership and management, and vendors

MNsure1
(Board of Directors)

MN.IT1

DHS1

Report Distributed

Vendors

MN.IT MNsure
PMO

Project
Director

MN.IT
Technical
Leads

MNsure/DHS
Business

- 42 -

Risk and Issue Management

Risk and Issue Management Overview

Risk and issue management are similar processes that enable the Project Director and MN.IT MNsure PMO to
monitor identified risks and issues during the course of the project. Risks and Issues may be proposed at any time
during the project and once confirmed, they are added in JIRA, are managed or resolved as appropriate, and are
included in the weekly status report

The assessment of risk and issue management included evaluating existing risk and issue management processes
and tools to provide assessment results, go-forward considerations, and an approach of how risks and issues can
be communicated across the project. The assessment was conducted across the elements of governance, process,
tools, and metrics for the entire issue and risk life cycle ranging from issue and risk reporting, tracking, assignment,
ownership, prioritization, resolution, and closure
A risk is defined as an event that has not occurred that will, if it does occur, impact the project schedule, scope,
budget, or quality. Risks need to be managed in terms of impact and probability. Mitigation strategies need to be
defined for all risks. These will be tracked and published in the weekly status report and escalated if not resolved
timely to reduce the likelihood that they become issues
An issue is defined as an event that has occurred that will impact the project schedule, scope, budget, or quality.
Unresolved Critical and High priority issues will be reported in the Weekly Status Report; medium issues greater
than 1 week past due will also be reported
The MN.IT MNsure PMO will conduct a weekly risks and issues meeting to proactively manage MNsure IT system
issues and risks
Summary Observations from the assessment, and additional detailed observations, impacts, and considerations for
the Risk and Issue Management area are presented on following slides

- 44 -

Risk and Issue Management Summary Observations

Risk and Issue Management

Component

Summary Observation

Risk/issue plan in project management plan

Partially present
Issues/risk management is present but plan is
over a year out of date

Risk log present and currently maintained

Partially present
Risk log is present but out of date

Issue log present and currently maintained

Present

Risk status present in risk log

Partially present
Risk status is present but risk log is over a year
out of date

Closed risk documented in risk log

Partially present
Risk status categorized but risk log is over a
out of date

Risk types categorized (i.e., Cost, Functional, Quality, Organization,

Performance, Project Management, Resource, Schedule, Scope, Technical,
General)

Not present
Risks in the log are not categorized by type

Prioritization of risks and issues

Partially present
Risk prioritization framework in place but risk
log is out of date

Issue types categorized (i.e., cost, functional, quality, resource)

Partially present
Some issues are categorized but others are not
categorized

- 45 -

Risk and Issue Management Detailed Observations

ID

Observation

Impact

Considerations

31

There is a lack of a formal risk/issue

escalation process

Leadership is challenged to identify

risk/issues across the project and
holistically identify threats to the project

32

Risk and issue logs are not standardized

or used across the MNsure governance
structure

Project status cannot be clearly

monitored without a central location to
track the progress or resolutions of
tasks, this presents risks to project
schedule, costs, and scope

Develop and manage risk and issue in

JIRA that will give each item a reference
number, owner, due date, and priority

33

Risks and issues lack owners and priority,

documentation of a process for escalating
issues and risks is limited

Due to limited detail for risk and

issues, decision-making can be
prolonged leading to additional cycles
to refine information

Implement risk and issue through the

MN.IT MNsure PMO to allow for scoring
(Probability * Impact) of risks and
document a process for risk and issue
management

34

Risk and Issue logs do not contain

needed information to fully track risks and
issues as the arise on the project

Risks and issue logs are incomplete

and project leaders cannot fully use
them in making project decisions

Develop risk and issue management in

JIRA that track item owners, priority,
owner, date creation, and criteria for
closure

- 46 -

Implement a formal risk/issue escalation

process, this would limit a reactionary
and inconsistent approach to mitigating
risks and issues

Risk and Issue Management Proposed Structure

The MN.IT MNsure PMO is responsible for managing the risk and issue processes. The PMT is responsible for risk and issue
resolution and escalation

Project
Management
Team

IT vendors

MN.IT MNsure
PMO

Project
Director

MN.IT
Technical Staff

MNsure/DHS
Business

- 47 -

Change Control

Change Control Process Overview

The change control process manages all changes requested during the MNsure IT system project. This includes technical
changes to application functionality, requested changes to schedule, and changes to scope. Change control is an integral part of
the project governance as it allows for changes to be proposed, approved and implemented through the appropriate governing
groups with responsibilities to manage change. Effective change control advises stakeholders and project team members of the
schedule for implementation of proposed changes

Summary Observations from the assessment, and additional detailed observations, impacts, and considerations for the Change
Control area are presented on following slides

- 49 -

Change Control Process Summary Observations

Change Control
Component

Summary Observation

Change control log present and being used by project team members

Partially present
Log is present but not used consistently at the CCB

Impact analysis performed on change requests

Partially present
Impacts discussed at the CCB but level of analysis
is inconsistent

Change control request template used for all change requests

Partially present
Request template in place, however it is
inconsistently used at the CCB

CCB operating on the project

Present
Not present
Change requests/orders not in status report

Change requests/orders in project status report

- 50 -

Change Control Process Detailed Observations

ID

Observation

Impact

Considerations

35

Projects are being proposed to

governing bodies by various State project
members and vendors to determine a
group of activities that should be
conducted as a project and prioritized

Projects are being used to manage

activities that should be driven by, and
prioritized, by an integrated project work
plan and these projects present
conflicting priorities and consume
resources to develop, discuss, and
determine validity

Project activities should be driven by an

integrated project work plan that is used to
determine and prioritize activities. A
change control process should be used to
manage requests to deviate from the
project plan (which is based on a baseline
set of requirements and approved design)

36

Change requests are made to vendors

by various project team members without
going through a formal change control
process prior to the work being
conducted

The State is being presented with

invoices for change orders from
vendors and the State is unable to
determine why a change was requested
or how the work was authorized to be
completed. Vendors are receiving
conflicting direction on activities and are
unclear on scope of activities

Project activities should be driven by an

integrated project work plan that is used to
determine and prioritize activities. A
change control process should be used to
manage requests to deviate from the
project plan (which is based on a baseline
set of requirements and approved design)

37

Decision prioritization of project change

requests cannot be determined because
governing groups are not provided with
sufficient information such as impact
analysis; including resource
requirements and dependencies to other
activities

Priorities for change requests are

undetermined or conflicting and the
organization cannot provide effective
direction to State staff and vendors

Provide governing groups with

appropriate information to make decisions
regarding change requests to allow them
to determine priorities including: a
detailed work plan and an impact analysis
for requested changes to the overall
project plan

38

Change request logs are not

standardized or used across the project

Project status cannot be clearly

monitored without a central location to
track the progress or resolutions of
tasks, this presents risks to project
schedule, costs, and scope

The Project Director should oversee the

MN.IT MNsure PMO to develop and
manage change request in JIRA that will
give each item a reference number,
status, justification, and impact summary

- 51 -

Change Control Process Proposed Structure

The MN.IT MNsure PMO is responsible for coordinating change requests for submission to the CCB. The MNsure Project Director
then reports the proposed changes in the form of change requests to the CCB for their decision to approve or deny the change.
Following the CCB action, the approved change orders are reported to the PMT for consolidation into the overall release plan as
needed

Project
Management
Team

MN.IT MNsure
PMO

Project
Director

Change
Control Board

Change Requests

Release Team

Business SMEs

IT vendors

- 52 -

Technology staff

Change Control Process Proposed Change Control Board Framework

Role for the MNsure Project
The Change Control Board evaluates proposed changes to the MNsure IT system. Changes can be proposed by MNsure, IT vendors, other state
agencies, and stakeholders using a MNsure IT system change request form. The Change Control Board evaluates, prioritizes, and approves or
denies requested changes for the MNsure IT system project. If approved, change requests become change orders and are passed to the PMT for
implementation into future releases

Key Responsibilities

Members Roles and Responsibilities

MN.IT: Chairs the Change Control Board and is responsible for
managing the activities during the CCB meetings
Project Director: Responsible for presenting change requests to
the CCB
DHS: DHS is a member and represents the interests of other DHS
programs that are affected by changes to the MNsure project
MNsure: MNsure is a member and represents the business and
operations impact of changes as they relate to the MNsure IT
system
Other Stakeholders: IT vendor project managers may be asked to
attend at the request of the chair to provide input to the change
request being discussed; other project members may be asked to
attend to provide input to the Board

Evaluate change requests

Prioritize change requests and change orders
Approve or deny change requests

Key Decisions

Key Relationships

Approve or deny change requests

Project Management Team: Once change requests are approved

by the CCB, the project management team works with the release
manager, and technology and business stakeholders to evaluate the
implementation of the change
Release Management Team: Provides input to inform the board's
discussion regarding the feasibility of the change requests
Vendors: IT vendors provide input to the CCB and the PMT
regarding the impacts of the change request

- 53 -

Meeting Cadence
One meeting per week

Defect Management

Defect Management Overview

Defect Management addresses all aspects of the defect life cycle from effective defect reporting and logging, ongoing review,
triage and prioritization, assignment to the appropriate owners for resolution, testing and validation, and certification to promote
the defect fixes to the production environment

Defect Management is closely integrated with testing and release management, and effective defect management contributes to
the quality of the system
Our defect management assessment spanned the review of existing defect management processes and tools to provide
assessment results and go-forward considerations, and the review of the current set of defects to support a re -prioritization to
align with State objectives. The assessment was conducted across the dimensions of governance, process, tools, and metrics fo r
the entire defect life cycle ranging from defect reporting, tracking, triage, assignment and ownership, prioritization, resol ution,
retest, and closure
Summary Observations from the assessment, and additional detailed observations, impacts, and considerations for the Defect
Management area are presented on following slides

- 55 -

Defect Management Summary Observations

Defect Management

Component

Summary Observation

Centralized owner or lead for defect management

Not present
Overall Defect Manager and defect management team does
not exist

Centralized ownership of defect management tools

Not present
The State does not maintain or have ownership of the
central defect repository (JIRA)

Consistent capture and recording of all reported defects accurately in the

defect management tool (JIRA)

Not present
Reported defects are not captured accurately in JIRA,
resulting in a far lower number of total open defects in JIRA

One system of record for production defects

Not present
Initial JIRA reports showed only 60-162 total open defects.
Upon further follow-up and detailed analysis of JIRA, 399
total open defects were identified, including duplicates.

Right complement of a defect management team

Partially present
Structured, coordinated Involvement of MN.IT, MNsure
business entities, and vendor teams in defect triage,
prioritization, and resolution is missing

Centralized access to defect management tools

Partially present
Key State personnel do not have the right access setup to
close resolved defects in JIRA

Coordinated defect handling from multiple defect channels and sources

Partially present
Defects reported from the field help desk, and various
contact centers are lost in transition and do not always make
it to JIRA

Established guidelines for the defect management life cycle

Partially present
Documented and established guidelines for the defect life
cycle are not fully in place

- 56 -

Defect Management Summary Observations (cont.)

Defect Management

Component

Summary Observation

Centralized defect triage

Not present
Regular and structured defect triage meetings and process not in
place

Coordinated defect prioritization, ownership, resolution, closure

Not present
Coordinated defect prioritization meetings, and tracking to resolution
and closure not in place

Pre-defined, timely defect resolution timeframes

Not present
Defects from the go-live timeframe are still open/unresolved in JIRA
without clarity as to the expected resolution timeframe

Coordinated, cross-vendor defect resolution with limited churn and

iterations

Not present
Established meetings and processes for reviewing and confirming
defect fix cross-vendor impacts prior to resolution not in place. Can
lead to cross-module issues getting uncovered for the first time in
integrated test/UAT leading to multiple iterations and rework

Adequate defect resolution details

Not present
Lack of clarity in JIRA when a defect is closed as to what was fixed,
and how the problem was addressed. Root cause details are missing.
Often, defects are closed prematurely in the early part of the SLDC
without UAT validation and approval

Robust defect management tool that supports defined defect

processes

Partially present
JIRA is not configured and setup to support defect management
processes needed on a project of this scale and complexity

Summary and detailed defect dashboard and metrics

Not present
Detailed defect dashboards and metrics are currently not in place and
not being distributed to management or executive teams. A clear,
concise current state of defects not depicted in JIRA

- 57 -

Defect Management Detailed Observations

ID

Observation

Impact

Considerations

39

Capture of reported defects is not

occurring consistently in the defect
management tool (JIRA)

Results in far fewer defects being

tracked and reported for the project,
thereby not providing an accurate
picture of system quality

Establish, communicate, and enforce

clear guidelines and setup resources to
enable all reported defects to be captured
in JIRA

40

Total number of open defects not reported

consistently, and reported numbers very
low (range from 60-399 total open defects
for the entire project)

In the absence of an accurate system

of record for defects, a clear picture of
system quality cannot be procured, and
focused, prioritized plans for resolution
cannot be put in place

Establish a centralized owner for JIRA

and enforce process for defect reporting,
capture, and management across the life
cycle to create a clear defect picture for
the project

41

Lack of a centralized owner for defect

management across the entire defect life
cycle

May lead to a lack of clarity and limited

understanding as to the current state of
defects and can result in outcomes of
the defect management process not
meeting expectations

Designate a Defect Manager from MN.IT

who is responsible and accountable for
defect management for UAT and
production. Define clear roles and
responsibilities and owners for each step
of the defect life cycle. Define clear
ownership for defect resolution

42

Access to JIRA is limited to a few

individuals, access is not aligned with the
duties of the individual, and licensing
issues have been observed, preventing
JIRA from scaling up for MNsure

This impacts the reporting of defects

and closure of reported defects, and
may impact the overall quality of the
system and detracts focus from the
real set of defects

With the proposed MN.IT ownership of

JIRA, configure JIRA to meet project
needs and align access groups and
controls with project roles and
responsibilities. Once the JIRA access
structure is established, review license
needs and upgrade as needed

- 58 -

Defect Management Detailed Observations (cont.)

ID

Observation

Impact

43

Ownership and maintenance of the defect

management tool (JIRA) is not with the
State

A single, consolidated view of the

universe of defects is lacking, thereby
potentially limiting the ability to use
defects as a reflection of system quality

Consider that MN.IT take responsibility

and accountability for the defect
management tool (JIRA), including setup,
access, usage, and maintenance, to
effectively leverage the tool for defect
management

44

Access to JIRA is limited to a few

individuals, access is not aligned with the
duties of the individual, and licensing
issues have been observed, preventing
JIRA from scaling up for MNsure

This may impact the reporting of

defects and closure of reported
defects, and impacts the overall quality
of the system and detracts focus from
the real set of defects

With the proposed MN.IT ownership of

JIRA, configure JIRA to meet project
needs and align access groups and
controls with project roles and
responsibilities. Once the JIRA access
structure is established, review license
needs and upgrade as needed

45

Centralized, coordinated defect resolution

process does not exist across vendors

Can result in multiple iterations of

testing and churn to successfully
resolve a defect to closure, and can
cause rework and additional usage of
resources such as people and time

Established a centralized prioritization and

resolution plan following defect triage and
expand testing in the lower environments
to reduce churn, achieve more successful
defect resolution with fewer iterations, and
prevent the scenarios of having to rush
partially tested code to the production
system

- 59 -

Considerations

Defect Management Detailed Observations (cont.)

ID

Observation

Impact

Considerations

46

Limited resources (knowledge base and

dedicated time) to triage and route
defects for resolution

Lack of defect triage can lead to issues

lingering in production longer than they
should and as a result, can be an
impact to system quality and end user
access to functionality

Setup a dedicated triage team structure

(MN.IT with support from business SMEs
and vendor developers) for timely triage
support

47

A single, consistent system of record for

all defects is missing. JIRA has been
setup and is being used, but not
consistently and effectively

Lack of consistent usage of JIRA

results in defects being reported and
tracked via email or not being reported
at all, which can lead to issues
lingering in production longer than they
should and impacting the quality of the
system

Establish, document, communicate, and

implement clear defect reporting, tracking,
and resolution guidelines and roles and
responsibilities so that defect processes
are being consistently followed across all
users and entities (MNsure, MN.IT, and
DHS). Clarify specifically how JIRA will be
used for recording the right content and
for driving defects to closure

48

Established guidelines for defect

reporting, tracking, and resolution do not
exist. Reported defects are missing
necessary pieces of data such as
severity, priority, and associated business
function which makes the triage and
prioritization a challenge. Definitions of
defect attributes and values, for example,
defect severity, environment defect was
identified in, are unspecified or setup as
optional in JIRA

May lead to fewer than actual defects

being reported. Insufficient information
may lead to issues with replicating the
defect and causing it to be deemed
invalid (when it is not). Defects are not
logged at the correct severity level,
impacting the ability to prioritize and fix
critical issues. Defects are closed
prematurely outside the testing/SME
group with limited clarity on the
resolution and root cause

Implement a full end-to-end defect

lifecycle, including guidelines for reporting
and detailed defect logging (including the
severity level, detailed descriptions,
screenshots, etc.), tracking, and
resolution, with detailed processes, roles
and responsibilities for each stage of the
lifecycle. Analyze and revise definitions
and data values for defect fields in JIRA

- 60 -

Defect Management Detailed Observations (cont.)

ID

Observation

Impact

49

Defect triage and prioritization are missing

and there is lack of clarity during triage on
items that function as designed vs. real
defects

A clear representation of system

quality is lacking as defects are not
being accurately reported and defect
queues are not being monitored,
triaged, and prioritized. This may
ultimately impact the quality of the
system

Designate a MN.IT Defect Manager

accountable for production defects
Conduct a defect clean-up effort in
concert with vendor teams, business, and
MN.IT to bring the current set of defects
up-to-date. Establish a team of defect
personnel from MN.IT with business
SMEs in an advisory role to monitor defect
queues on an ongoing basis. Establish
recurring defect triage meetings with key
stakeholders (MN.IT, business, vendors)
to review defect status reports, key
findings, and triage outcomes. Have triage
outcomes include impact analyses to drive
prioritization of defect work load to the
vendor teams and developers and drive
defects to resolution

50

Defect triage and defined resolution

timeframes are missing

In the absence of pre-defined

timeframes, there is limited
accountability from the responsible
parties to turnaround defect triaging
and resolution within expected
timeframes. This causes defects to
linger in production and impact system
quality

Amend the contract to include official, predefined timeframes for defect triaging and
resolution based on defect severity and
impact. Clarify scope and expectations
around ownership and accountability for
each step during the timeframe

- 61 -

Considerations

Defect Management Detailed Observations (cont.)

ID

Observation

Impact

Considerations

51

Inadequate resources (masked

production data, environment setup, etc.)
to triage and route defects for resolution

Valid defects may get canceled if

unable to be replicated and the
corresponding issues may continue to
linger in production and impact system
quality and end user access to
functionality

Dedicate a team (MN.IT, business, vendor

developers) to have bi-directional
communication and escalation in the
event of insufficient defect data. Take
corrective measures to provide a
production-like environment with masked
production data that is refreshed at
regular intervals, to facilitate successful
defect replication and to augment reported
defects with additional information for
developers to resolve. Provide access to
canceling and closing defects to a select
group of individuals

52

Issues reported via the field help desks,

contact centers, escalation centers and
other sources do not migrate effectively to
the central source of defects (JIRA)

Can cause a breakdown of critical

information flow and may result in
delaying the resolution of critical
defects and in causing ambiguity and
uncertainty to the reporting party
around the status and resolution of
reported issues

Identify and define possible sources of

defects. Identify clear roles and
responsibilities for each defect source.
Provide the required tools, skills, training,
documented process, and dedicated
resources to the defect source centers.
Establish a mechanism to allow for bidirectional communication and escalation
between the source centers and central
defect team on the project

53

When defects are resolved, there is lack

of clarity on what the root cause was or
how the defect was resolved

Limits insight into the perceived quality

of the system and the volume of
potentially duplicate issues

Leverage the defect management tool

(JIRA) effectively to enforce that key data
be entered as part of the defect lifecycle
process

- 62 -

Defect Management Detailed Observations (cont.)

ID

Observation

Impact

Considerations

54

Although JIRA is being used as the defect

management tool, JIRA has not been
adequately setup, configured, and
leveraged to an extent that could make
defect management more effective

Defect reporting, tracking, and

prioritization are impacted as a result of
the limitations imposed by the current
setup and usage of JIRA

Setup and configure JIRA for the needs of

the MNsure project. Identify critical defect
fields to be mandatory and create training
guides for defect reporting so that defects
are reported consistently. Dedicate
focused resources and time to keep JIRA
up-to-date and conduct ongoing review
and triage to further defect prioritization
and resolution. Establish and
communicate clear guidelines on
managing the defect life cycle in JIRA

55

Status dashboard and metrics for defect

management are not being created,
maintained, and distributed

Can lead to limited transparency and

visibility around the status of defects
and can result in lack of clarity or an
impact to perceived system quality.
Can also hinder focus on the real
issues and the ability to prioritize and
resolve them to closure

Conduct a clean-up of the current state of

defects in JIRA. Define a team for daily
review, triage, clean-up, assignment, and
prioritization of defects. Define and
publish a detailed defect status report that
includes data such as defect status,
severity, priority, and impacted business
function. Define the frequency, content,
and audience for an executive summary
dashboard of defect results. Identify the
stakeholders who will receive dashboard
and drive outcomes and resolution

- 63 -

Defect Management Proposed Structure

The MN.IT Defect Manager is responsible for defect clean-up and prioritization, defect assignment, tracking resolution, and closure.
The Defect Manager manages multiple teams that support the various defect management activities. The MN.IT Defect Manager
reports up to the overall Test Manager who is ultimately accountable for defect management, and who in turn, reports up to th e
MNsure Project Director

MN.IT MNsure
PMO

Project
Director

MN.IT Overall
Test Manager

MNsure/DHS
Business

IT vendors

MN.IT Defect
Manager

MN.IT Defect
Triage Team

- 64 -

MN.IT Defect
Management
Team

MN.IT Defect
Tools and
Metrics Team

Defect Management Proposed Defect Management Team Framework

Role for the MNsure Project
Addresses all aspects and activities of defect management from establishing and implementing processes throughout the defect lifecycle,
enforcing SLAs for tracking, and resolution of defects, and manage defect triage and prioritization. The team is led by a Defect Manager who is
responsible for this and who reports up to the overall Test Manager who is ultimately accountable for Defect Management
.

Key Responsibilities

Members Roles and Representation

MN.IT: Provides the Defect Manager who is responsible for owning
and managing the process and a triage team experienced and
knowledgeable in MNsure. Is responsible for maintaining the defect
tool (JIRA)
MNsure and DHS: Provides business analysts and SMEs for
subject matter clarifications and representation, sign-off, and
approval at the triage and prioritization meetings
Vendors: Provide development leads as vendor product SMEs,
support for defect triage, estimation for prioritization of defects

Monitor defect reporting and defect processes and their

adherence to established processes
Monitor defect queues
Drive defect triage
Drive prioritization
Monitor defect SLAs
Track defects to resolution
Defect status reporting
Stakeholder communication

Key Decisions

Key Relationships
Project Management Team: Understand the status and progress
of the current state of defects and the resolution plan relative to the
overall release schedule
MNsure and DHS: Active representation across business areas for
review and approval of work prioritization
Vendors: Provide triage and produce SME support, provide
estimations to factor into the prioritization and defect resolution plan
for information about the status and future of MNsure

- 65 -

Defects triage outcomes are confirmed

Defect prioritization is established
Escalation checkpoints are established, triggered, and
escalated issues are tracked to resolution
Resolution plans, including estimates, are confirmed and
communicated officially to vendor groups

Defect Management Defect Prioritization

The below aspects have been taken into consideration for defect prioritization for the MNsure IT system project
Scope:
It was not possible to get a consistent result of the total universe of non -closed defects in JIRA
Multiple data requests sent for a report of non-closed JIRA defects yielded inconsistent results, ranging from a total of 60
to 162 defects
An independent assessment of the universe of total non-closed* defects in JIRA on 5/8/2014, was 399 defects, which is
the basis of this prioritization

Below is an existing breakdown of the 399 non-closed defects by Priority and Severity
Existing Breakdown of 399 nonclosed defects by Priority

Existing Breakdown of 399 nonclosed defects by Severity

Defects by Severity

Defects by Priority
Blocker

Critical

11

1.5 - Major blocking

42
2 - Major no workaround

63

3 - Major w workaround

44

Minor

192

Trivial

196

4 - Minor w workaround
5 - Cosmetic

Deferred

32

Major

78

15 5

1 - Total Failure

14 5

14

80

Pending PM
assignment

Major
Pending Assignment

A summary of the existing breakdown by Severity is:

11% have no value assigned for Severity
5% are categorized as SEV 1, 1.5
71% are categorized as SEV 2, 3, Major
13%% are categorized as SEV 4, 5

A summary of the existing breakdown by Priority is:

48% have no value assigned for Priority (of which
70% are major severity defects)
39% are categorized as blocker/ critical/ major
13% are categorized as minor/trivial/deferred

* Non-closed includes all statuses except closed, from all projects in JIRA (MNHIX*, SCM Team, Short Term Projects, Security Dom ain)

- 66 -

Defect Management Defect Prioritization (cont.)

Re-Prioritized Resolution Criteria:
The existing universe of 399 defects have been analyzed and grouped into the below three priorities to address for
resolution:
Priority 1 based on critical functionality (as defined in the Key Function Matrix (KFM) in the functional assessment
in deliverable 3)
Priority 2 based on functionality (outside of critical functionality addressed in priority 1). Defects still pending
triage and open more than 90 days are also included in this priority
Priority 3 based on functionality deferred or not in near-term scope, or internal, isolated, technical errors for
vendor-specific modules that may not be reproducible or are open for more than 90 days
Of the 399, non-closed defects, 52% are currently assigned
a priority

Breakdown of 399 non-closed defects by proposed

Resolution Priority

Defects breakdown by Priority for 207

defects (where Priority is populated)

Defects by Resolution Priority

53

Blocker, Critical,
Major

15

37

Priority 1

Minor, Trivial

Priority 2
123

223

155
Deferred

A summary of the breakdown before re-prioritization is:

75% are categorized as top priority
18% are categorized as low priority
7% are categorized as deferred

Priority 3

A summary of the breakdown after re-prioritization is:

Priority 1:
Only 43% (of the previously categorized 207 defects) are at Priority 1
55% of the total 399 are at Priority 1
Priority 2:
39% (of the previously categorized 207 defects) are at Priority 2
30% of the total 399 are at Priority 2
Priority 3:
9% (of the previously categorized 207 defects) are at Priority 2
15% of the total 399 are categorized as Priority 3

- 67 -

Defect Management Defect Prioritization (cont.)

Key takeaways:
30% of the non-closed defects are outstanding from 2013
A large percentage of defects (75%) was tagged as top priority in the existing non-closed defect universe. This leads to
dilution of the concept of top priority and makes it challenging to arrive at a realistic, achievable, defect resolution pl an
Going forward, existing definitions of Severity and Priority should be re-evaluated to refine the definitions and usage of
these fields during defect reporting, logging, triage, and prioritization
Attached are the prioritization results:
Below is an embedded excel file that should be reviewed in its entirety for the detailed results of the defect prioritization
effort
Given the volume of defect content, this document is being included in electronic format only

- 68 -

Test Management

Test Management Overview

Testing is an integral part of the Software Development Life Cycle (SDLC) because it validates the ability of components and the
system to meet business requirements. Testing verifies that the system works as designed and drives the identification and
management of defects in software quality towards resolution. Testing advises stakeholders, clients, and project team members
as to the software quality
For a system implementation to be effective, quality must be built in from the beginning and across the entire SDLC ranging f rom
unit test during development to User Acceptance Test (UAT) and post-deployment validation in production. An organized, well
documented, and structured testing process creates transparency and accountability for quality at each step of the SDLC
The testing assessment spanned current testing processes and tools across the testing phases to provide assessment results and
go-forward considerations. The assessment was conducted across the dimensions of governance, process, tools, and metrics for
the testing phases of unit test, integration test, system test, user acceptance test and production smoke test, and across the
testing types of performance test, automation test, security testing, and ADA testing
Deloittes observations, impacts, and considerations for the Test Management area are presented on the following slides

- 70 -

Test Management Summary Observations

Test Management

Component

Summary Observation

Test team by phase, where the team is well-defined with roles and
responsibilities, including a Test Manager, Testers, Business SMEs,
and Development/Product Support

Not present
Overall Test Manager, Test Leads for each phase and the right
complement of test resources not in place

Thorough testing in each phase prior to UAT and production smoke

test

Partially present
Testing occurs directly in production and for the first time in UAT or
production for complex functionality and components such as
batch jobs and notices

Adequate testing training that ramps up the testing staff on critical

business functions

Partially present
Training occurs on an as-needed basis or not at all, and often,
business SMEs pick up testing due to limited functional knowledge
outside that group

Well-defined test strategy and approach

Partially present
Does not exist for some phases (such as unit test or integration
test). May exist for phases such as UAT but is not documented.
Often created ad hoc and as-needed and not maintained or
tracked against

Detailed test plan outlining key components of a test phase

Partially present
Not documented and may exist informally; often created just in
time but not maintained or tracked against

Clear, achievable test schedule maintained and updated to factor in

dependencies and delays

Not present
Pre-defined schedule does not exist. Delays in earlier test phases
or deployment delays to UAT not factored in to adjust the UAT
schedule, causing impacts to the available time for testing in UAT

Documented test scenarios and test cases

Partially present
Not updated for ongoing functionality. High-level and usable by a
small group only; cannot be leveraged effectively by IT Testers and
other stakeholders

- 71 -

Test Management Summary Observations (cont.)

Test Management

Component

Summary Observation

Documented test case traceability matrix

Partially present
Only one point-in-time, outdated version. Not maintained
for ongoing changes in requirements and test cases

Clear, specific, well-documented, pre-approved entrance and exit criteria

Not present
Acceptance and certification of functionality is done on a
qualitative basis by business SMEs and documented
entrance and exit criteria not present

Well-defined test data

Partially present
Insufficient, often invalidated by multiple testers using the
same data set, and created manually for the most part

Re-usable and repeatable test data creation and automation testing

Partially present
Limited means to effectively create large volumes of data

Robust test environment to support end-to-end testing

Not present
Does not exist for UAT; interfaces, batches, notices cannot
be tested in the UAT environment

Formalized and documented smoke testing

Partially present
Occurs to a limited extent

State-owned and managed performance testing

Not present
Only 3 runs of performance test to-date, conducted by a
third party

Robust, repeatable regression testing

Partially present
All regression testing owned by vendor and done primarily
in system test. Limited to no regression test in UAT

Testing of components such as interfaces, batches, notices, and reports

Partially present
Limited, can test only in system test and not in UAT

- 72 -

Test Management Summary Observations (cont.)

Test Management

Component

Summary Observation

Testing tools usage for test case creation and maintenance, test
execution, Performance testing, Security Testing, ADA Testing

Partially present
Limited, de-centralized, not coordinated and fully leveraged

Testing Dashboard and Metrics

Not present
Executive and detailed dashboards for test metrics not present

- 73 -

Test Management Detailed Observations

ID

Observation

Impact

Considerations

56

State (and specifically MN.IT) supervision

of unit, integration, and system test
phases is limited. Each phase in the lower
environments is owned and managed by
a different stakeholder with a lack of
consistent processes across the board

Limited visibility and transparency to

testing in the lower environments may
lead to unclear entry and exit criteria
and may result in more defects
identified in later stages of the project,
which may result in more time, cost,
and resources expended for resolution
at that stage

Designate a MN.IT Test Manager who is

accountable for testing (including test
cases, results, defect management,
testing communications, stakeholder
involvement, entry and exit criteria) across
all test phases (unit, integration, system,
UAT, production). Develop plan to
coordinate testing in lower regions and do
not wait to UAT. Create a team of MN.IT
test leads, wherein the leads report up to
the Test Manager and each lead is
aligned with and accountable for one test
phase each (unit, system, integration,
UAT, production, regression). For
instance, for UAT, there will be a test team
and test lead that report up to the Test
Manager (more details to follow in the
chart). Make provisions in the contract to
allow vendor teams to share unit and
integration test details with the State

57

The User Acceptance Test (UAT) team is

lacking the full complement of the right
mix of resources, knowledge base, and
stakeholders for testing

Can impact the quality and

effectiveness of testing and overall
confidence in approving the release to
production. Can also limit the ability to
confirm if the release functionality
meets business requirements or not,
which has the likelihood of impacting
end user access to functionality

Designate a MN.IT Test Lead, who

reports to the overall MN.IT Test Manager,
and is accountable for UAT. Involve
stakeholders from MN.IT, DHS, MNsure,
and the vendor teams in UAT to augment
the knowledge base and provide
clarifications as to the build content.
Establish a team and stakeholder
structure with clear expectations around
roles and responsibilities

- 74 -

Test Management Detailed Observations (cont.)

ID

Observation

Impact

Considerations

58

UAT is conducted during a limited test

window and on an unpredictable schedule
with insufficient knowledge as to the
contents of the release, thereby resulting
in incomplete testing. The contents of the
release are not clearly documented via
release notes, and documentation around
MNsure functionality is limited or entirely
lacking in instances. Documentation is
also not kept updated to reflect updates to
functionality

Lack of documentations limits the

testing teams ability to write thorough
test cases targeted to test critical
functionality, which thereby limits
testing effectiveness. Limited testing
may lead to the release being
prematurely promoted to production,
causing delayed identification of
defects and regression items, and
increased time, cost, and resources to
resolve issues found in production

Establish a consistent schedule and plan

for releases to UAT, outlining the timeline,
expectations, and criteria for UAT kick-off.
Plan for adequate buffer in the schedule
to factor in unknowns. Outline clearly and
specifically the contents of releases to
UAT via release notes or other such
documentation. Proactively communicate
schedule changes to the UAT stakeholder
group. Create and maintain
documentation around MNsure
functionality via up-to-date requirements
and design documents

59

There are instances where testing of

complex functionality occurs directly and
for the first time in production

Lack of testing of specific functionality

prior to production poses a risk of
regression, where existing functionality
is impaired, or the intended new
functionality deployed to production
does not work. This can result in
impacting access to production and in
severe circumstances, even impact
production availability or uptime

Setup adequate resources (environment

setup, data, people, time in the schedule)
to initiate business user testing early on
and in advance of UAT and more
comprehensive testing during UAT to
avoid the situation of testing in production

- 75 -

Test Management Detailed Observations (cont.)

ID

Observation

Impact

Considerations

60

The UAT phase has incomplete and

inconsistent testing processes,
specifically around test case scope and
creation, test execution, reporting and
review of results, defect identification,
tracking, and resolution, established entry
and exit criteria, and stakeholder
representation and communications

Diminishes the effectiveness and intent

of the UAT phase and can lead to
delayed identification of defects at a
later stage or directly in production,
resulting in increased time, cost, and
resources for defect resolution and
increasing the risk of impacting end
user experience and access to critical
functionality

Establish and implement process for the

areas outlined below:
Test cases: scope, creation, review,
traceability, sign-off, and maintenance
to reflect new functionality
Test execution: data creation,
execution, tracking and reporting of
results
Defect management: Identification,
reporting, tracking, communication, and
resolution of defects
Established entry and exit criteria
Stakeholder involvement in UAT and
timely communication of decisions and
outcomes

61

UAT is limited in its effectiveness as a

result of environment constraints such as
the inability to test end-to-end scenarios,
components such as interfaces, notices,
reports, and batches, and time based
scenarios that need time advancement

This results in testing some

functionality for the first time in
production and identifying and
resolving issues at that point, which
may result in expending more time,
cost, and resources, and delaying the
access of planned functionality to the
end user

Prioritize the setup of a UAT environment

to allow for the testing of critical
components such as interfaces, notices,
reports, and batches. Build focused test
teams knowledgeable in testing each
component including stakeholders from
MN.IT, MNsure, DHS, and the vendors.
Prioritize the addition of system
functionality to advance the time clock

- 76 -

Test Management Detailed Observations (cont.)

ID

Observation

Impact

Considerations

62

UAT is limited in its effectiveness as a

result of data constraints such as limited
test data, lack of a means to automate
data creation, and lack of masked
production data to replicate and retest
production issues

Limited testing leads to more issues

identified in production, resulting in
expending more time, cost, and
resources for resolution. Lack of
masked production data in a secondary
environment limits the ability to
replicate and resolve critical production
defects that may continue to linger in
production longer than they should and
impact the end users access to
functionality

63

Detailed security testing has been

conducted, however, code corrective
actions suggested to some of the vendor
groups have not been prioritized and
implemented to date

If identified code issues and gaps are

resolved timely, then the effectiveness
of security testing can improve.
Depending on the type of gaps
outstanding, those may result in
security non-compliance and render
the product vulnerable to security
threats

Prioritize the remediation of security gaps

with the vendors. Identify the list of all
pending gaps by vendor and create a
resolution plan in concert with the Security
team and MN.IT

64

ADA testing is still ongoing; the State is

working with a third party vendor to
assess any gaps in accessibility and
disability compliance of the product and
ADA testing

Unless the current plan with the third

party vendor is followed closely, any
potential gaps in accessibility and
disability compliance may not be
remediated in a timely manner

Suggest active monitoring, tracking, and

reporting of status against the plan, timely
review of the assessment, and
prioritization of resolution for any identified
gaps

- 77 -

Identify and allocate test resources to the

UAT team for supporting data
management (creation and automation).
Setup a secondary environment with
masked production data, or alternatively,
refresh this data periodically into UAT,
and provide access to this data to
vendors, testers, and the business users,
to allow for production issues to be
replicated and resolved

Test Management Detailed Observations (cont.)

ID

Observation

Impact

Considerations

65

The performance test efforts undertaken

by the State with SOASTA have
uncovered performance issues and gaps,
many of which are yet to be resolved.
These issues range from site capacity
limitations, HTTP errors once the capacity
is reached, lower than expected response
times, throughput, bandwidth, and server
stability, and connection reset and other
errors

System performance may directly

translate to end user experience and
access, and the users ability to
effectively use the MNsure website.
Resolution of lingering performance
issues can result in improving end user
access and the number of successful
enrollments

Identify a MN.IT owner for performance

testing, through its life cycle from testing
to issue resolution and fix migration to
production. Identify and designate a
performance team within MN.IT to track
and monitor progress with each vendor
via the issue resolution plan. Identify
critical performance attributes and
establish clear requirements for each
attribute. Work with SOASTA to
understand the current state against these
attributes. Prioritize and create a
resolution plan with the vendors for the
performance issues and gaps identified to
date and new gaps against the
established baseline. Rerun performance
tests with SOASTA at periodic intervals
monitor progress against the baseline

- 78 -

Test Management Detailed Observations (cont.)

ID

Observation

Impact

Considerations

66

Testing tools currently used for test

execution (MS Excel) and defect
management (JIRA) can be setup and
leveraged more effectively. Testing tools
are currently not integrated or available to
all stakeholders

Effective usage of tools may enable

better tracking of test case traceability
to requirements, test case results, and
defect management. This may result in
more transparency, accountability, and
accurate reporting of the outcomes of
testing

In the near-term, analyze and leverage

existing tools effectively. This entails
activities such as setup, providing user
access to stakeholders, creating and
communication training guides for correct
usage, ongoing tracking and monitoring of
data, ongoing review of the results and
tracking to expected outcomes. In the
long-term, assess integrated test and
defect management tools that provide
strong out-of-the-box capabilities that can
be leveraged on a project of this scale and
size

67

Status dashboard and metrics for test

management are not being created,
maintained, and distributed

May limit the transparency and visibility

around the status of testing which
could limit the ability to drive to
successful outcomes and hinder the
full effectiveness of the testing process

Define and publish on a weekly basis a

detailed test status report that outlines the
scope of testing, traceability to
requirements, test execution results, test
case first pass rate, defect density,
resolution plan, and plan as to additional
test cycles, if any. Define the frequency,
content, and audience for an executive
summary dashboard of test results.
Identify the stakeholders who will receive
dashboard and drive outcomes and
resolution

- 79 -

User Acceptance Test (UAT) Management Proposed Structure

The MN.IT Test Lead is responsible for User Acceptance Test (UAT) and manages the Test Team and testing involvement with the
business entities and vendor groups. As referenced in ID 56 on slide 74, the structure below can be used for testing beyond UAT.
The MN.IT UAT Test Lead reports up to the MN.IT Overall Test Manager whose responsibility extends beyond UAT. The Overall Test
Manager is ultimately accountable for UAT. The Test Manager reports to the overall MNsure Project Director

MN.IT MNsure
PMO

Project
Director

MN.IT Overall
Test Manager

MNsure/DHS
Business

IT vendors

MN.IT UAT
Test Lead

MN.IT Test
Team

- 80 -

Test Management Team Proposed User Acceptance Testing (UAT) Framework

Role for the MNsure Project
Addresses all aspects and activities of UAT from test case and test data management, test execution, test status reporting and tracking, defect
reporting and tracking, to regression testing and certifying code readiness for production. The team is led by a UAT Lead who is responsible for
this phase and who reports up to the overall Test Manager who is ultimately accountable for UAT

Members Roles and Representation

Key Responsibilities

MN.IT: Provides the Test Manager who is accountable for UAT, a

UAT Lead who is responsible for day to day activities in UAT, and
testers experienced in testing processes and tools. Provides UAT
environment support and maintenance
MNsure and DHS: Provide SMEs for subject matter clarifications
and Q&A; and review/sign-off of test cases. Provide business
representation for UAT sign-off/approval.
Vendors: Provide development leads for Q&A and triage of defects
or issues identified during UAT

UAT Planning and Management

Test case management
Test data management
Test execution
UAT Environment management
UAT Status Reporting and Tracking
Defect Reporting and Tracking
Discussing/approving entry and exit criteria
Stakeholder communication
Gate/Approval of code migration to production

Key Decisions

Key Relationships

Confirm that entry criteria are met to start UAT

Approve the scope and content of test cases
Review and approve test case results
Agree on defect reporting guidelines, severity, and priority of
defects identified in UAT
Approve functionality conformance to requirements
Approve and certify code readiness for production

Project Management Team: Understand the status and progress

of UAT relative to the release schedule
MNsure and DHS: Understand the UAT plan in advance to
anticipate and plan resource needs and representation across all
key business areas
Vendors: Need to provide triage support and Q&A for issues
identified during UAT

- 81 -

Test Management Test Plan Outline

The following slide illustrates a representative test plan outline, with key components to be included in the plan, for User
Acceptance Test (UAT)
The creation and effective implementation of plans similar to this for other test phases unit test, integration test, system test and
other test types such as performance test, ADA test, and regression test are likely to result in structure and coordination for those
test phases and types

- 82 -

Test Management UAT Plan Outline

ID

Activity

Description

Purpose and Scope

The purpose provides an introduction to the test plan and outlines the intent and components of the plan. The scope highlights the high level functional requirements or functional areas that the test plan applies to.

Test Objectives

This section will outline the motivating factors and expected outcomes of testing, the aspects that are in scope, and an overview of
planned tests with whats included and whats not

Test Strategy

The Test Strategy establishes the foundation for all testing activities. It covers testing policies and processes to support the vario us test
levels and cycles. The Test Strategy will provide flexible, consistent delivery of testing services to drive improved quality , lower cost and
increase speed of delivery across the system.

Test Approach

The Test Approach is created after the Test Strategy has been approved. It outlines the scope of the overall testing effort, the test levels
required for the project, the test team organization, the estimated effort needed to plan and execute, the issue resolution p rocess and the
roles/responsibilities of the team involved. The Test Approach is the predecessor to the detailed Test Plan.

Detailed Test Plan

The Test Plan includes: the scope of the testing effort, roles and responsibilities of all team members providing support, th e schedule and
time frame for scenario development and testing, and a detailed overview of all activities involved in the system testing pro cess. The Test
Plan will identify the standards and metrics against which test activities are planned and measured.

Test Scenario, Test Cases, and

Test Case Traceability Matrix

Includes detailed definition of the test scenarios, review, and approval, and traceability of the test cases to requirements

Entry and Exit Criteria

The Test Entry Criteria help determine if the execution of a particular Test Plan can begin. All criteria within the Testing Approach must
be met or documented. Exceptions must be mutually agreed upon before testing can begin. The Test Exit Criteria will be used t o
determine if the execution of the Test Plan is complete and intended objectives are met. The criteria must be clearly documented upfront.

Test Data Requirements

Outlines all aspects of test data management, including the types of data, how frequently data should be refreshed, mechanisms to
create and use data, any automated tools for creating data, and the resources and ownership of data management

Test Environmental Needs

Includes the environment name and technical details, for the source and target systems as well as any tools used for testing.
Environment sizing and the intended number of testing iterations (assuming the target environment will be refreshed/cleared i n between
iterations) will be critical expectations to document. Access to the environment(s) should also be defined.

10

Staffing, Roles and

Responsibilities, Training Needs

This section outlines the required resources to address the test effort, main roles and responsibilities of these resources, along with
expected knowledge base and skill sets. The section also discusses how to approach training for the testing roles on the proj ect.

11

Test Schedule

This section will include the key schedule milestones, the test schedule for detailed planning and iterations (execution cycles), number of
iterations, characteristics of each iteration (for example: size of load, timeframe, data variations), and the expected timeframe for each.
Depending on the solution, it may be advisable to begin with a subset of production data that represents the basic or most common
business scenarios, and then perform iterations on more focused scenarios individually.

12

Testing Dashboard and Metrics

Reports should be defined to be created regularly to track, manage, and communicate the progress and status of testing. These reports
include summary and detailed information of test scripts executed and defects discovered during testing. The reports are gene rated
based on the data elements in the Test Management Tool, which provides for customization of the attributes as needed.

13

Testing Risks, Dependencies,

Assumptions, and Constraints

This section will identify potential risks, mitigation and contingency for each risk, and its likelihood. Any assumptions or depe ndencies
likely to impact the test plan, test executions, or outcomes of testing should also be outlined here, with an escalation and mitigation plan.

- 83 -

Release Management

Release Management Overview

Release management activities include planning releases, scheduling releases, monitoring releases status, overseeing vendor
resources, aligning releases to business expectations, and ensuring release quality
Currently release management for the MNsure project is present however opportunities exist to improve release management

Release Management is closely integrated with testing and development and determines that code is deployed in the right
environment at the right time. The release manager coordinates release activities with change management, testing
management, and defect management to align activities across the project
The components that were assessed for release management include release plans, calendars, roles and responsibilities,
prioritizations, release estimates, deployment standards and tools including release management checklists
Summary Observations from the assessment, and additional detailed observations, impacts, and considerations for the
Release Management area are presented on following slides

- 85 -

Release Management Summary Observations

Release Management
Component

Summary Observation

Release Plan that details the software release to all environments, identifies release
strategy, logistics, tasks, recovery and disaster plans, rollback plans, and pre and
post-implementation activities

Partially present
Some details are contained in individual documents

Integrated Release Calendar that provides a view of all activities such as

development and testing and details release dependencies such as vendor product
dependencies

Partially present
A schedule has been developed, but it is missing
the integrated view

Compliance/standards champion present in the release management team with the

ability to understand the requirements associated with the standards and able to
verify that the standards are appropriately implemented and adhered to in the
application

Not present
Individuals are driving requirements to completion,
however the no cross-organization role has been
defined with expectations

Roles and Responsibilities of Release Management Organization that defines the

organization structure and the role, responsibility, and activities

Not present
Individuals are performing roles, however the roles
have not been defined with expectations

Release Notes that list all items delivered within a particular release for both
business and technical audiences

Partially present
Technical details are included based upon vendor
input, however the business focus of release notes
is limited

Prioritization Matrix that identifies importance of defects, enhancements, and is used

to develop budgets for releases

Not present
Prioritization occurs through informal processes
and no priorities are documented associated to
defects and requirements

Release Checklist a deployment tool that encourages the deployment process is

followed and may have environment specific features

Partially present
Checklists are being used for migrations, however
information sharing between environments is
limited, processes are not documented, and
checklists are not used to drive continuous
improvement

- 86 -

Release Management Detailed Observations

ID

Observation

Impact

68

A release management plan along with

associated roadmap does not exist

Makes it difficult to manage the overall

release management process including
release planning and estimation, release
governance including prioritization of the
requirements can lead to difficulty in
planning and executing releases and
have appropriate requirements met within
the allocated budget

MN.IT should develop a release

management plan along with a roadmap,
the release management plan includes
release planning, release governance,
process documentation, and
documentation standards

69

There is a lack of an overall release

manager across all environments, there
is no one on point for maintaining an
overall release calendar and no single
point of contact to drive deployments to
each environment and encourage
expected processes are being followed
consistently for each environment

Results in quality and deployment issues

such as missed deployment windows,
rework, incomplete regression testing,
and missed requirements. This also
results in different approaches being
followed in each environment which can
lead to confusion and inconsistent
processes

Define the role of release manager and

provide the release manager the authority
to lead release management end-to-end
to promote quality and improvement in
release management execution. Develop
consistent standards and processes for
release management across all
environments

70

Due to multiple entities involved in

release management, there is a lack of
clarity around roles and responsibilities
and a consolidated view thereof

Can lead to confusion as to who is

responsible and results in quality issues.
Can also lead to missed deployment
windows and rework along with budget
being spent on unsuccessful
deployments

Define specific and clear roles and

responsibilities to improve the structure for
release management

71

An overall approach or strategy

associated to driving requirements
relative to standards is lacking.
Individual business owners are
identified to drive requirements and
implementation of functionality specific
to their products

Prevents a holistic view of how guidelines

and standards are met and can lead to
missed requirements

MN.IT should define and incorporate a

role for a cross-organization
compliance/standards champion into the
release management team. That person
should have the ability to understand the
requirements associated with the
standards and be able to verify that the
standards are appropriately implemented
and adhered to in the application

- 87 -

Considerations

Release Management Detailed Observations (cont.)

ID

Observation

Impact

Considerations

72

Deployment processes have not

been documented and are only
partially implemented

Results in inefficient deployment

processes being executed and
environment discrepancies delaying the
deployment of releases; this can lead to
deployment dates being missed and
lead to resources working on the same
deployment multiple times, thereby
wasting deployment resources and
possibly impacting the schedule of future
releases

Develop a list of deployment processes

including deployment checklists for each
vendor and environment, implement
environment standards and documentation
standards such as standardized release notes
and standardized change controls, and find
opportunities to streamline deployment through
automated tools such as ClearQuest or other
MN.IT tools thereby reducing the resources
needed for deployments

73

Defect fixes, new code, and product

upgrades are not actively managed
and prioritized by the State

Lower priority items may be fixed prior to

higher priority items and budget may be
spent on fixing items that may not be a
priority. More complex, higher priority
items remain unresolved, impacting
availability of functionality and overall
product quality

Implement an estimation and prioritization

process associated to defects that uses
standardized tools such as JIRA and
ClearQuest so that high priority defects can be
estimated and scheduled for release. Ensure
collaborative process is established between
Release, Defect, and Test Managers

74

Mapping the dependencies

between the various vendors in
terms of software versions needed
in order to meet the release
schedule has not occurred

Leads to unsupported combinations of

vendor packages, thereby increasing
risk and possibly requiring additional
testing or not meeting functional
requirements for the end user

The State should map the dependencies

between the various vendors in terms of
software versions needed in order to meet the
release schedule and determine if there are
unsupported combinations of vendor
packages and determine associated
mitigations

75

Release testing by the State is

primarily done in the User
Acceptance Testing Environment

Leads to identifying defects reactively

delaying releases and requiring
additional budget to resolve defects

The State should test prior to the User

Acceptance Testing Environment and be
responsible for regression testing. This helps
confirm timeliness and quality around
deployment and testing, and prevent defects
from being identified in the User Acceptance
Environment Testing for the first time

- 88 -

Release Management Detailed Observations (cont.)

ID

Observation

Impact

76

Full requirements traceability does not

exist due to the original requirements from
Maximus not being utilized; 2700
requirements were documented at varying
levels of detail

Due to this lack of rigor, requirements

for implementation have been missed
leading to additional spending to
remediate gaps downstream

Conduct a fit gap analysis of the current

application factoring in any assumptions
and gaps around underlying technologies
and pre-existing functionality, determine
the associated gaps and develop a plan to
address the gaps

77

The current approach to documenting

requirements is not standardized, at times
a blank whiteboard is used versus a fit
gap analysis for the vendor applications

Leads to designing processes that do

not coincide with functionality of the
vendor applications and may result in
wasted coding effort or having to
rework the requirements. Assumptions
are also made around what exists outof-the-box and what functionality needs
to be built

The State should document the

requirements gathering process taking
into consideration the underlying
technologies and pre-existing functionality

78

Vendors have expressed concern about

the lack of business ownership of
requirements and the overall release
management process including
deployment management and support of
business processes such as prioritization

Results in conflicting priorities and

rework due to confusion about the
requirements and their priority, this can
lead to missed requirements or work
being done on lower priority
requirements requiring additional
budget to address the higher priority
requirements

The State should develop a matrix and

implement a process that indicates who is
responsible for owning business
requirements and setting priorities

- 89 -

Considerations

Release Management Detailed Observations (cont.)

ID

Observation

Impact

Considerations

79

Release notes are focused on the

software (vendor specific) and do not
effectively highlight the business changes
that are being deployed for the MNsure
application

Stakeholders do not have exposure to

all the change requests that have been
implemented thereby making it difficult
to understand what is delivered in each
release and communicate changes
outward and making it more difficult to
test and validate the deployment
effectively

Release notes should be improved to

serve both the needs of the business and
to provide a consolidated view of the
release. This allows the business to better
structure their UAT and execute
communication plans for users of the
application, and enable the technical
teams to better understand all the
components that are being deployed and
any inter-dependencies between
components

80

Tactical planning has occurred for release

management but an integrated calendar
view is lacking

Makes it challenging to understand

delivery schedules and dependencies
associated to releases, makes it more
difficult to plan for deployments and
testing, and may be more challenging
to identify code and version conflicts of
the various vendor packages

The State should develop an integrated

calendar view of future releases along
with all dependencies

81

No prioritization process associated to

requirements exists, and there is a lack of
formalized process associated to
business requirements. Release
schedules are developed, but priorities
change and then schedules are adjusted

Results in confusion about delivery and

can lead to missed requirements,
requiring additional budget to address
missed requirements

The State should develop a prioritization

matrix associated to requirements and
formalize requirements definition
processes and release schedules so that
high priority items are addressed

- 90 -

Release Management Proposed Structure

The MN.IT Release Manager reports to the Project Director and is responsible for managing vendor deployments as well as the
MN.IT Deployment Team, the MN.IT Release Manager provides status to MNsure and DHS business stakeholders and to the PMO,
the MN.IT Release Manager is responsible for obtaining deployment approval in each environment and coordinating environment
changes with MN.IT Infrastructure Team

MN.IT MNsure
PMO

Project
Director

MNsure/DHS
Business

MN.IT Release
Manager

IT vendors
MN.IT
Infrastructure

- 91 -

MN.IT
Deployment
Team

Release Management Proposed Release Management Team Framework

Role for the MNsure Project
Release Management addresses all aspects of deployment and release management including the development of the release management
strategy and plan, the governance of business requirements, prioritization, and definition of the roles and responsibilities for the deployment and
release teams

Key Responsibilities

Members Roles and Representation

MN.IT: MN.IT provides the Release Manager who executes the
Release Management Plan and develops the Integrated Release
Calendar
MNsure: Provides business SMEs by functional areas and who are
responsible for approving the Prioritization Matrix and the Release
Roadmap
DHS: DHS to represent the interests of other State programs that
are affected by Release Management
Vendors: Project Managers from each of the IT Vendors are
recommended to attend the meeting and present their requested
changes as appropriate and discuss any dependencies

Develop Release Management Plan

Complete Prioritization Matrix
Create the deployment and release checklist
Establish consistent deployment and release processes in
each environment
Manage Integrated Release Calendar
Develop Release Roadmap
Implement Documentation Standards
Execute production and UAT deployments

Key Decisions

Key Relationships

Counties/ Providers/Brokers/Navigators: Several groups work

directly with MNsure customers and have a need to know the plan
and business impacts of deployments
MNsure and DHS Stakeholders: Need to understand the
deployment and overall release schedule to allocate resources for
testing and support
Vendors: Need to understand the implications of deployments

Approve product upgrade deployments

Approve the content for defect fixes and code upgrades
Certify environment readiness before start of a deployment
Certify and approve completed deployments and release for
testing

Meeting Cadence
As needed per workplan

- 92 -

Release Management Plan Outline

ID

Activity

Description

Purpose and Scope

The purpose provides an introduction to the release plan and outlines the intent and components of the plan. The scope highlights the
high-level functional requirements that are required to be implemented as part of release management

Current State Inventory

As part of creating a release management plan, an initial step is to conduct a current state inventory of release management processes,
tools, and stakeholders

Release Strategy

This section provides an overview of the strategy for future MNsure releases. The strategy includes understanding of any concurrent
project deployments to be included within the release, identification of any components or sub-systems that are impacted, and the
nature of impact. For MNsure, the release strategy incorporates the overall orchestration of resources, tasks, and environments
required to perform a successful release

Release Logistics

This section documents the logistics for this release in terms of technology infrastructure, network, application, third party software and
resource needs. The logistics included here provide an overview of the process and components needed to orchestrate a successful
release into production or non-production environments

Release Estimation

This section defines the estimating actions that need to be completed for a successful production or non-production release

Release Classification

Releases will be managed by the Release Manager and grouped into Release Types such as Major, Minor, and Emergency. Individual
process steps may vary by release type

Roll Back Planning

It may become necessary to revert to the pre-release state, if possible. Detailed steps should be developed for Rollback. These are
taken in the event that a contingency occurs during or after release that cannot be appropriately mitigated

Release Go-No Go Criteria

The purpose of the Release Go-No Go Criteria Checklist is to evaluate the readiness of going live with the new system. The criteria
should be used to help aid the decision of whether or not to move a release into the production environment

Release Notes

The Software Release Notes is the quality record that lists the items delivered within a particular release. The document includes
general information about the release, compatible products in the release, upgrades from previous releases, new features introduced in
the release and known limitations, bugs, and workarounds

10

Prioritization Process

As part of the prioritization process, stakeholders need to prioritize defects, enhancements, and develop overall budgets for releases.
Prioritization correlates with release classification. The overall process for prioritization integrates with the release management
process and the overall governance structure

11

Roles and Responsibilities

This section identifies the roles for performing the release and deployment activities and describes the responsibilities of identified roles

12

Release Processes

All release management processes will be documented in each environment as part of developing the release management plan

- 93 -

Appendix A:
Project Management
Processes

Proposed Project Status Reporting Process

The MN.IT MNsure PMO will be in charge of triggering the status report data request process on a weekly basis. Under this structure,
they are responsible for compiling, synthesizing, and developing the weekly project status reports which, following the Project
Directors approval, are distributed. Status report distribution list includes State executives and leaders, governance groups, vendor
partners, and external stakeholders

MN.IT
MNsure
PMO

Project
Status

1. Submit Data
Request

Action
Items

Test
Management

Change
Requests

Release
Management

Vendors

2. Compile Data
Request

3. Update JIRA with

necessary data

MN.IT

2. Compile Data
Request

3. Update JIRA with

necessary data

DHS
MNsure

2. Compile Data
Request

3. Update JIRA with

necessary data

Decisions

Risks

Legend
Task

Inputs /
Outputs

Step

Supporting
Outputs

Procedure

- 95 -

Decision

Issues

Work Plan

4. Distribute
Project Status

Project Status Reporting: Executive Summary <dd-mmm-yyyy dd-mmm-yyyy>

Project Status Summary
Overall

Project
Status

Scope

Items Needing Leadership Attention

Resources Schedule

Quality

Budget

Request
ID

Priority/
Sev erity

Description

Target
Resolution Date

Project
Status
Sum m ary

Upcoming Deliverable and Key Milestone Status

Deliverable / Milestone Nam e

Progress

Baseline

Planned/Actual

Finish Date

Finish Date

Status

Com m ents

C
G

Y
R
NS

Deliv erable Status and Milestone Summary Legend

NS

Not started

Proj ect Trends

Trending Up (Improving)

Completed

- 96 -

On track

Flat Trend (Steady)

<1 week behind schedule

>1 week behind schedule

Trending Down (Declining)

Proposed Risk Management Process

Project Management Team assesses the severity of the risk and manages the risk once a mitigation strategy is determined

Project
Team
Members
MN.IT
MNsure
PMO

1. Identify Risk

Update JIRA

2. Validate JIRA
Entry

5. Develop Risk
Response

Project
Management
Team

Yes
No
4. Is Risk Severity
High?

3. Analyze Risk

6. Monitor Risk

No

Risk
Owner

7. Is the
Risk
Real?

Legend
Task

Inputs /
Outputs

Step

Supporting
Outputs

Procedure

- 97 -

Decision

8.
Manage
Risk

Yes

9. Close Risk

Proposed Issue Management Process

The objective of this process is to manage the issues identified in the project, which includes identifying, prioritizing, assigning,
monitoring, and closing issues through all project phases. Issues are managed on an ongoing basis and reviewed on a monthly basis.
JIRA is the tool used for issue management

MN.IT
MNsure
PMO

Update JIRA

1. Record and
validate details
about issue in
JIRA

Project
Management
Plan

4. Track and follow

up on issue

2. Assign
proposed issue

5. Change
Request?

3. Performs issue
management

Yes

Project
Management
Team

Change
Control
Board

5. Manages
Change
Requests

Legend
Task

Inputs /
Outputs

Step

Supporting
Outputs

Procedure

- 98 -

Decision

No

6. Closes issue

Proposed Change Requests Process

The objective of this process is to identify, manage, and facilitate change control decisions on change requests to client contract terms
and/or project deliverables that have been signed off and placed under change control. Change requests are managed on an ongoing
basis and the change control board will meet on a weekly basis. JIRA is the tool used for change control

MN.IT
MNsure
PMO

2. Review
Change Request
Form

3. Review Impact
Analysis

Change
Control
Board

Change
Requestor

Project
Management
Plan

4. CCB Review
and Approval?

Yes

No

1. Complete
Change Request
Form

PMO
Communications
Manager

5. Communicate
Changes

Legend
Task

Inputs /
Outputs

Update JIRA

Step

Supporting
Outputs

Procedure

- 99 -

Decision

Proposed Change Request Template

The Change Request template will be used by stakeholders to initiate a Change Request. The template will be submitted to the MN.IT
MNsure PMO and will be used to record, track, and manage change requests throughout the life of the project. The PMO will keep
JIRA up to date based on any Change Request forms received. Once the Change Control Board makes a decision on a specific
Change Request the PMO will update JIRA to reflect it

- 100 -

Proposed Defect Management Process

The objective of the defect management process is to enable timely communication of defects through appropriate channels in an
effort to quickly triage and resolve issues as they are detected throughout the project lifecycle. Defect management occurs on an
ongoing basis. JIRA is the tool used for defect management

MN.IT Defect
Manager

2. Drive defect
triage

MN.IT Defect
Team

3. Conduct
defect
triage

1. Intake
Defects

Business
SMEs

4.
Participate
in defect
triage

Vendor
Teams

5. Drive
defect
prioritization

8. Create
defect
resolution and
retest plans

9. Retest defects
and track defects
to resolution and
closure

6.
Participate in
defect
prioritization

11. Validate and

approve defect
closure

7. Provide input
to the defect
plans and
resolution details

Legend
Task

Inputs /
Outputs

Step

Supporting
Outputs

12. Publish
executive and
detailed defect
dashboards

Procedure

- 101 -

Decision

10. Support defect

management and
Q&A

Proposed User Acceptance Test (UAT) Process

The objective of the UAT process is to develop the User Acceptance Test (UAT) Plan to validate that the system meets business
requirements. UAT is managed on an ongoing basis and MS Excel is used as the primary tool for test case creation, execution, and
maintenance

3. Approve the
Test Strategy,
Approach, and
Plan,

MN.IT Test
Leadership

MN.IT Test
Team

Business
SMEs

1. Create Test
Strategy,
Approach, and
Plan,

10. Publish test

results and build
acceptance

Provide
notification as
to a new build

4. Create the test

scenarios and
cases

6. Create
test data
7. Execute
test cases

2. Review and provide

input on the test
strategy, approach,
and inputs

5. Review and
provide feedback on
the test scenarios
and cases

8. Support test
execution

Vendor
Teams
Legend
Task

Inputs /
Outputs

Step

Supporting
Outputs

Procedure

- 102 -

Decision

9. Review
and
Approve
test results

Proposed Release Management Process Flow

The objective of this process is to manage the migration of software changes developed and deployed in the form of packages
released to the production system. The process is managed on an ongoing basis. ClearQuest and JIRA are the tools used for release
management.

MN.IT

Release
Plan

1. Plan
Release

PMT

Vendors

2.
Develop
Release
Schedule

3. Develop
Deployment
Plan

No

4. Approve
Release?

Yes

Release
Schedule

PMO
Communications
Manager

7. Communicate
Release

Legend
Task

6. Deploy
Release

Inputs /
Outputs

Step

Supporting
Outputs

Procedure

- 103 -

Decision

8. Close
Release

Appendix B:
Roles and Responsibilities

Roles and Responsibilities Status reporting, vendor management, work plan, and
requirements RACI Matrix

R
A
C
I

IT Vendors

C
C
C
C
C
A
A
I
R

C
C
C
C
C
C
C
I
A

C
C
C
C
C
C
C
C
I

C
C
C
C
C
I
C
A
C

MN.IT MNsure PMO

MNsure

Compile and Verify Raw Metrics Data

Populate Project Status Reports
Analyze Metrics and Complete Report
Distribute project status report to appropriate internal and external stakeholders
Status reporting
Vendor management
Work plan management
Work plan - updates and maintenance
Develop Requirements

DHS

MNsure Project Activities

MN.IT

The RACI matrix identifies the governing groups, agencies and stakeholders involved in governance that are responsible,
accountable, consulted and informed

A, R
A, R
A, R
A, R
A, R
R
R
R
I

Legend
Who is Responsible? The person or entity assigned to do the work
Who is Accountable? The person or entity that makes the final decision
Who is Consulted?
The person or entity assigned that must be consulted before a decision or action is taken
Who is Informed?
The person or entity that must be informed that a decision or action has been taken

- 105 -

Roles and Responsibilities Risk Management RACI Matrix

Identify and analyze risk

Determine risk severity
Develop risk response for high level risk
Monitor risk
Manage risk
Close risk
Incorporate risk information into weekly status report

R
A
C
I

C
A
A
A
A
A
A

A
C
C
C
C
C
C

MN.IT MNsure PMO

Project Team

MNsure Project Activities

Project Management
Team

The RACI matrix identifies the governing groups, agencies and stakeholders involved in governance that are responsible,
accountable, consulted and informed

R
R
R
R
R
R
R

Legend
Who is Responsible? The person or entity assigned to do the work
Who is Accountable? The person or entity that makes the final decision
Who is Consulted?
The person or entity assigned that must be consulted before a decision or action is taken
Who is Informed?
The person or entity that must be informed that a decision or action has been taken

- 106 -

Roles and Responsibilities Issue Management RACI Matrix

R
A
C
I

CCB

Record and validate issue

Assign and prioritize issue
Perform issue management
Track and follow-up on issue
Determine if issue requires a change request
Close issue
Document closed issue
Incorporate issue information into weekly status report

MN.IT MNsure PMO

MNsure Project Activities

Project Management
Team

The RACI matrix identifies the governing groups, agencies and stakeholders involved in governance that are responsible,
accountable, consulted and informed

A
A
A
R
A
A
A
A

R
R
R
A
R
R
R
R

C
C
C
C
C
C
C
C

Legend
Who is Responsible? The person or entity assigned to do the work
Who is Accountable? The person or entity that makes the final decision
Who is Consulted?
The person or entity assigned that must be consulted before a decision or action is taken
Who is Informed?
The person or entity that must be informed that a decision or action has been taken

- 107 -

Roles and Responsibilities Change control RACI Matrix

R
A
C
I

CCB

PMO Communications
Manager

Complete change request form

Review change request and perform impact analysis
Accept or reject change request
Draft change request communication
Approve and send change request communication
Maintain change requests in JIRA
Incorporate change request information into weekly status report

MN.IT MNsure PMO

MNsure Project Activities

Change Requestor

The RACI matrix identifies the governing groups, agencies and stakeholders involved in governance that are responsible,
accountable, consulted and informed

R
C
C
C
C
C
A

A
R
R
R
I
R
R

C
A
A
C
A
A
C

I
I
I
A
R
I
I

Legend
Who is Responsible? The person or entity assigned to do the work
Who is Accountable? The person or entity that makes the final decision
Who is Consulted?
The person or entity assigned that must be consulted before a decision or action is taken
Who is Informed?
The person or entity that must be informed that a decision or action has been taken

- 108 -

Roles and Responsibilities Defect management RACI Matrix

R
A
C
I

MNsure Project Activities

MN.IT Defect Manager

MN.IT Defect Team

Business SMEs

IT Vendors

The RACI matrix identifies the governing groups, agencies and stakeholders involved in governance that are responsible,
accountable, consulted and informed.

Drive defect triage calls

Conduct defect triage
Participate in defect triage
Prioritize defect
Provide input to the defect plans and resolution details
Create defect resolution and retest plans
Retest defects and track defects to resolution and closure
Support defect management and Q&A
Validate and approve defect closure
Publish executive and detailed defect dashboards
Incorporate defect information into weekly status report

A
R
R
A
C
A
R
C
C
A
R

R
A
A
R
R
R
A
R
I
R
A

I
I
C
I
I
I
I
I
A
I
I

C
C
C
C
A
C
C
A
R
C
C

Legend
Who is Responsible? The person or entity assigned to do the work
Who is Accountable? The person or entity that makes the final decision
Who is Consulted?
The person or entity assigned that must be consulted before a decision or action is taken
Who is Informed?
The person or entity that must be informed that a decision or action has been taken

- 109 -

Roles and Responsibilities Testing management RACI Matrix

R
A
C
I

Business SMEs

IT Vendors

Write and execute UAT Scenario

Oversee UAT process
Test reports for prioritization and decision making
Perform load and performance testing
Produce UAT reporting and dashboards
Create Test Strategy, Approach, and Plan
Review and provide input on the test strategy, approach, and inputs
Approve the Test Strategy, Approach, and Plan
Create the test scenarios and cases
Approve the test scenarios and cases
Create test data
Execute test cases
Support Q&A for test execution
Review and approve test results
Publish test results and build acceptance

MN.IT Test Team

MNsure Project Activities

MN.IT Test Leadership

The RACI matrix identifies the governing groups, agencies and stakeholders involved in governance that are responsible,
accountable, consulted and informed.

A
R
A
A
R
R
C
A
R
C
R
R
C
C
A

C
A
R
C
A
A
R
C
A
R
A
A
R
R
R

I
I
I
I
I
I
A
R
I
A
I
I
I
A
I

R
C
I
R
C
C
C
C
C
C
C
C
A
C
C

Legend
Who is Responsible? The person or entity assigned to do the work
Who is Accountable? The person or entity that makes the final decision
Who is Consulted?
The person or entity assigned that must be consulted before a decision or action is taken
Who is Informed?
The person or entity that must be informed that a decision or action has been taken

- 110 -

Roles and Responsibilities Release management RACI Matrix

R
A
C
I

MNsure Project Activities

MN.IT

Project Management
Team

IT Vendors

PMO Communication
Team

The RACI matrix identifies the governing groups, agencies and stakeholders involved in governance that are responsible,
accountable, consulted and informed

Plan release
Develop release schedule and deployment plan
Review and approve release
Deploy release
Develop release notes
Draft release communications
Approve and send release communications
Close release
Document closed release in release management plan
Incorporate release information into weekly status report

R
R
R
A
R
C
C
I
R
R

A
A
A
R
A
R
A
R
A
A

C
C
C
C
C
C
C
C
C
C

I
I
I
I
I
A
R
A
I
I

Legend
Who is Responsible? The person or entity assigned to do the work
Who is Accountable? The person or entity that makes the final decision
Who is Consulted?
The person or entity assigned that must be consulted before a decision or action is taken
Who is Informed?
The person or entity that must be informed that a decision or action has been taken

- 111 -

Appendix C:
Interviews Conducted

Interviews Conducted
No.

Organization

Interview Subject

Interview Date

MN.IT

Testing

May 1, 2014

MN.IT

Governance

May 2, 2014

MNsure

Deployment Process

May 2, 2014

MNsure

Governance

May 5, 2014

MNsure

Governance

May 6, 2014

MN.IT

Testing

May 6, 2014

MN.IT

Federal Hub Discussion

May 7, 2014

MN.IT

Testing

May 7, 2014

DHS

Governance

May 7, 2014

10

DHS

Governance

May 7, 2014

11

MNsure

Governance

May 7, 2014

12

MN.IT, IV&V

Testing

May 7, 2014

13

DHS

Conversion Strategy

May 7, 2014

14

EngagePoint

Release Management

May 7, 2014

15

MN.IT

Governance

May 7, 2014

16

DHS

Governance

May 7, 2014

17

MNsure

Governance

May 7, 2014

- 113 -

Interviews Conducted (cont.)

No.

Organization

Interview Subject

Interview Date

18

MNsure

Carrier Communication

May 8, 2014

19

MN.IT

Testing

May 8, 2014

20

DHS

Governance

May 8, 2014

21

DHS

Governance

May 8, 2014

22

DHS

Governance

May 8, 2014

23

Connecture

Release Management

May 8, 2014

24

Connecture

Testing

May 9, 2014

25

MN.IT

Testing

May 9, 2014

26

Counties

Governance, communication

May 9, 2014

27

EngagePoint

Governance

May 12, 2014

28

IBM/Cram

Release Management

May 12, 2014

29

MN.IT

Testing

May 12, 2014

30

Board of Directors

Governance

May 12, 2014

31

DHS, MN.IT

Testing

May 12, 2014

32

Carriers

Governance

May 13, 2014

33

MNsure

Testing

May 13, 2014

- 114 -

Interviews Conducted (cont.)

No.

Organization

Interview Subject

Interview Date

34

EngagePoint

Testing

May 13, 2014

35

IBM/Cram

Governance, communication

May 13, 2014

36

DHS

Testing

May 13, 2014

37

PwC

Governance

May 13, 2014

38

DHS, MN.IT, MNsure

Testing

May 13, 2014

39

EngagePoint

Testing

May 13, 2014

40

Navigators

Governance

May 13, 2014

41

IV&V

Governance

May 14, 2014

42

DHS

Testing

May 14, 2014

43

MN.IT

Testing

May 14, 2014

44

DHS, MN.IT

Governance

May 15, 2014

45

Connecture

Governance

May 15, 2014

46

MN.IT

Governance

May 20, 2014

47

MN.IT

Governance

May 21, 2014

48

MNsure

Brokers

June 3, 2014

- 115 -

Disclaimer

This document may contain Confidential Information and is intended strictly for MNsure s internal use and
not for any third party. As such, Deloitte is not, by means of any resulting publication of this document,
rendering professional advice or services to any third party. Any resulting publication should not be used
by any third party as a basis for any decision or action that may affect its business. Third parties should
consult a qualified professional advisor before making any decision or taking any action that may affect its
business.
Deloitte shall not be responsible for any loss sustained by any third party who relies on any resulting
publication of this document.

About Deloitte
Deloitte refers to one or more of Deloitte Touche Tohmatsu Limited, a UK private company limited by
guarantee, and its network of member firms, each of which is a legally separate and independent entity.
Please see www.deloitte.com/about for a detailed description of the legal structure of Deloitte Touche
Tohmatsu Limited and its member firms. Please see www.deloitte.com/us/about for a detailed description of
the legal structure of Deloitte LLP and its subsidiaries. Certain services may not be available to attest clients
under the rules and regulations of public accounting.

- 116 -