Upgrading Oracle Enterprise

Manager: Why and How

Aaron Rimel
Team Lead

October 2015

Innovating at the intersection of Health Care

and IT
Worlds largest
standalone health care
IT company
$2.9 billion in revenues
21,000 associates+ worldwide
14,000 client facilities in 30+ countries
4,900 physician practices
590 ambulatory facilities
3,500 extended care facilities
150 employer sites
1,790 retail pharmacies

R&D commitment and

clinical culture
$3.8 billion of cumulative R&D
3,000+ person software development org
1,800 clinicians
More than 140 health IT patents

#4

Top 100 Healthiest

Workplaces in America

#22

Most innovative
companies 2014

2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.

Managed services
386 contracts
CernerWorks
52,403 Systems
1,034 Network Circuits
90 PB Disk Storage

Managed
OMS
33 Clients
DBA
4 Clients

Disaster
Recovery

Remote
Hosted

33 Clients

356 Clients

248,576 Contracted Users / 247,856 Peak Users

245,851 Beds = 27 % of total beds in United States
36,971 Facilities

ITWorks

Cloud

SaaS

15 Clients

Cerner Wellness
359 Databases

PowerWorks
2,718 Clients
20,215 Concurrent Users

Patient Portal
943 (290 Acute, 684 PWx)
Healthe Record RHIO / HIE
13 Sites

CommunityWorks
72 Clients

Healthe Transaction/Exchange
3 TPA Clients
Skybox
180 Clients
2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.

EM @ CERNER

Oracle Management Server version 12.1.0.4.4

Three EM installations, Production/Sandbox/Development
Repository version:11.2.0.3 3.5tb in size
EM Agents: 1500 Agents, 97% RHEL, 3% other
Targets 25k
Tableau, Vertica
AWR Warehouse

4
Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.

OEM Monitoring at Cerner

60M Raw Metrics

Daily
33,000
Incidents
18,000 Incidents
2,300
Alarms
540 Tickets

2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.

Corrective Action statistics

2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner.

EM 13.1.0.0 Way to go !

We love the ease of installation. =)

Gold agent image, ora check plug-in
Single point solution for Agent Lifecycle
Flash is gone (almost)

7
Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.

EM 13.1.0.0 : Upgrade Vs Install ?

13.1 Upgrade or Fresh Install?

Large repository, extended upgrade time

Creating a new multi EM installation
Repository clean-up, upgraded from 10.1.x,10.2.x,11.1,12.1.x
Methodical upgrade, allows for a client focued upgrade, making
changes to client systems when it is right for them.

8
Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.

How we Manage Agents with Same Configuration

We were way behind the Oracle Patch recommendations.
Prior to Golden Agent Images, upgrading/patching agents would

have taken a FTE. We almost never patched agents.

EM now has a patch/psu rotation. We can now plan and execute our
EM agent upgrades staying current with PSUs or applying a one-off
to fix an immediate problem.
Upgrading agents prior to GAI, took nearly a year to accomplish due
to the vast number of agents that exist.
The implementation of Gold Agent Images has altered our deployment
schedule to be in-line with PSU.
9
Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.

Solution: Gold Agent Lifecycle Feature

Requirements (One time activity)
Software Library Configured
Source Agent to Create Gold Image Fresh deploy or use existing agent
Step1: Apply patches, configuration changes to the Source Agent
Step2:Test the Source Agent Check if agent is up, uploading
Step3: Create Gold Image
Step4: Use Gold Image to deploy, update, upgrade Agents (Recurring)
Step5: Apply any new patches, Config changes to Source Agent
Step6: Create another version of Gold Image
10
Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.

Solution: Gold Agent Lifecycle Feature

Activity with Agent Gold Image

Password Required

Root.sh execution

Fresh Deploy

Password required and supports SSH

keys, PBRUN, Sudo, all EM supported
authentications

Yes

Update (Patching) agent with

newer version of Gold Image

Password required

Root.sh depends on what is the type of update

a)If the patch need sbin changes then it
requires root.sh execution.
b)For config property changes, root.sh is not
required

Upgrade agent using Gold image

No password required, uses existing

OMS-Agent communication

We have updated /etc/sudeors to allow for our OS user to run commands

specific to the agent, ie root.sh, allroot.sh to make the upgrade even
easier by removing the need for root access to the machines.

11
Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.

 oem ALL=(root,oracle) NOPASSWD:/bin/su - oracle*,

/usr/bin/id,
/u01/oracle/product/oemagent/agent13c/*/agentdeployroot.sh, /
u01/oracle/product/oemagent/agent13c/*/*/agentdeployroot.sh,
/u01/oracle/product/oemagent/agent13c/*/sbin/nmosudo *,
/u01/oracle/product/oemagent/agent13c/sbin/nmosudo *,
/u01/oracle/product/oemagent/agent13c/*/root.sh

12
Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.

Changes to Agent Directory Structure

Base Directory Structure of

Fresh Agent Install Using a
Gold Image

<agent_base_directory>
|_____core
|_____<agent_software_version>
|_____plugins
|_____agent_inst
|_____sbin
|_____plugins.txt
|_____plugins.txt.status
|_____agentimage.properties

Base Directory Structure of

Upgraded Agent Using a Gold
Image

<agent_base_directory>
|
_____GoldImage_<gold_image_nam
e>
|_____core
|
_____<agent_software_version>
|_____sbin
|_____plugins
|_____agent_inst

Base Directory Structure Agent is

upgraded or updated using a gold
image multiple times

<agent_base_directory>
|_____GoldImage_<gold_image_name_1>
|_____core
|
_____<agent_software_version>
|_____sbin
|_____plugins
|_____GoldImage_<gold_image_name_2>
|_____GoldImage_<gold_image_name_3>
.
. |_____agent_inst

Tip: Sudoers file should have $oracle_home set and agent user profile with right oracle home. Don't
hard core the agent base dir/sbin in the sudoers file.
13
Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.

Why Out of Place Agent Updates ?

Simple rollback process - If Gold Agent Update fails then your
old Agent will come up automatically
Simplified old Agent home cleanup
Reduce downtime
Multiple Agents with multiple patches in a single downtime
While updating old Agent is brought down later in the process

14
Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.

Solution: Gold Agent Lifecycle Feature

Dashboard, real time tracking,
maintenance via clicks

Agents yet to be updated

with Gold Image

Evaluated as compliance
against the latest image

15
Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.

Solution: Gold Agent Lifecycle Feature

Chart showing
subscribers versions

Drifters Detects manual

Patching and drifts from
Current Gold Image

Displays all artifacts

included in the version of
Gold Image
Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.

16

Solution: Gold Agent Lifecycle Feature

7
1
2

1.Gold Image can be Imported/Exported

from a Test to Prod EM
2.Agents can be updated with Gold Image
using Push or Pull
3.Define Batch size, success criteria and
frequency which will reduce workload on
OMS and help you plan your updates
3

4.Updates can be planned during

maintenance and comfortable time periods
5.Get email notification for
Success/Failed/Status for each Agent
update in a Batch

6.Ability reclaim space by cleaning out old

Agent Oracle Homes
5

7.Run just Prerequisites before actual

update operation

6
Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.

17

Solution: Gold Agent Lifecycle Feature

Export Gold Images across Test /Dev/Prod EM environments

2
3

18
Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.

Solution: Gold Agent Lifecycle Feature

Single view for all activities

19
Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.

Solution: Gold Agent EMCLI scripting option

Used EMCLI Gold Agent commands for updating 2500 Agents!

Worked like a charm
Repeatable and reliable process
Use cases executed:
Upgraded Agents 12.1.0.3 to 12.1.0.4
Updated all Agents 12.1.0.4 to 12.1.0.4.1
Again updated Agents from 12.1.0.4.1 to 12.1.0.4.2

20
Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.